Download - High-resolution Hyperspectral Imaging via Matrix Factorization

Transcript
Page 1: High-resolution Hyperspectral Imaging via Matrix Factorization

High-resolution Hyperspectral Imaging via Matrix Factorization

Rei Kawakami1 John Wright2 Yu-Wing Tai3 Yasuyuki Matsushita2 Moshe Ben-Ezra2 Katsushi Ikeuchi3

1University of Tokyo, 2Microsoft Research Asia (MSRA),

3Korea Advanced Institute of Science and Technology (KAIST)

CVPR 11

Page 2: High-resolution Hyperspectral Imaging via Matrix Factorization

Giga-pixel Camera

M. Ezra et al.Giga-pixel Camera

@ Microsoft research

Large-format lens CCD

Page 3: High-resolution Hyperspectral Imaging via Matrix Factorization

Spectral cameras

LCTF filter35,000 $

Hyper-spectral camera55,000 $

Line spectral scanner25,000 $

• Expensive• Low resolution

Page 4: High-resolution Hyperspectral Imaging via Matrix Factorization

Approach

Low-reshyperspectral

high-resRGB

High-reshyperspectral image

Combine

Page 5: High-resolution Hyperspectral Imaging via Matrix Factorization

Problem formulation

W(Image width)

H(Image height)

S

Goal:

Given:

Page 6: High-resolution Hyperspectral Imaging via Matrix Factorization

A1: Limited number of materials

Sparse vector

For all pixel (i,j)

Sparse matrix

W (Image width)

H (Image height)

S

= …

00.40…

0.6

Page 7: High-resolution Hyperspectral Imaging via Matrix Factorization

Sampling of each camera

• Low-res camera • RGB camera

Spectrum

Wavelength

Intensity

Page 8: High-resolution Hyperspectral Imaging via Matrix Factorization

Sparse signal recovery

• •

Filter SignalObservation

t

tm

n

Page 9: High-resolution Hyperspectral Imaging via Matrix Factorization

Sparsity

Signal Basis Weights

Signal Basis Weights

0

S

S-sparse

Page 10: High-resolution Hyperspectral Imaging via Matrix Factorization

Sparse signal recovery

Observation

Need not to know which bases are important

Sparsity and Incoherence matters

Page 11: High-resolution Hyperspectral Imaging via Matrix Factorization

A2: Sparsity in high-res image

W (Image width)

H (Image height)

S

Sparse coefficients

Sparse vector

Page 12: High-resolution Hyperspectral Imaging via Matrix Factorization

Simulation experiments

Page 13: High-resolution Hyperspectral Imaging via Matrix Factorization

460 nm 550 nm 620 nm 460 nm 550 nm 620 nm

Page 14: High-resolution Hyperspectral Imaging via Matrix Factorization

430 nm 490 nm 550 nm 610 nm 670 nm

Page 15: High-resolution Hyperspectral Imaging via Matrix Factorization

Error images of Global PCA with back-projection

Error images of local window with back-projection

Error images of RGB clustering with back-projection

Page 16: High-resolution Hyperspectral Imaging via Matrix Factorization

Estimated430 nm

Page 17: High-resolution Hyperspectral Imaging via Matrix Factorization

Groundtruth

Page 18: High-resolution Hyperspectral Imaging via Matrix Factorization

RGBimage

Page 19: High-resolution Hyperspectral Imaging via Matrix Factorization

Errorimage

Page 20: High-resolution Hyperspectral Imaging via Matrix Factorization
Page 21: High-resolution Hyperspectral Imaging via Matrix Factorization
Page 22: High-resolution Hyperspectral Imaging via Matrix Factorization

HS camera

Filter

CMOSLensAperture

Focus

Translational stage

Page 23: High-resolution Hyperspectral Imaging via Matrix Factorization

Real data experiment

Input RGB Input (550nm) Input (620nm)Estimated (550nm) Estimated (620nm)

Page 24: High-resolution Hyperspectral Imaging via Matrix Factorization

Summary

• Method to reconstruct high-resolution hyperspectral image from – Low-res hyperspectral camera– High-res RGB camera

• Spatial sparsity of hyperspectral input– Search for a factorization of the input into

• basis • set of maximally sparse coefficients.