Computational Single-photon Imaging · Computational Single-photon Imaging! Gordon Wetzstein!...

Computational Single-photon Imaging!

Gordon Wetzstein!Stanford University!

ISSW 2/28/2018!

www.computationalimaging.org!

Muybridge’s Multi-Camera Array at Stanford!

Computational Imaging!

Computational Displays!

Computational Cameras!

computation! optics &!electronics!

human visual system!

optics! computation!sensing!




optics! computation!sensing!



Computational Displays

computation optics &electronics

human visual system


HDR Imaging [Debevec, Nayar, …]! Super-resolution [Baker, …]! Light Fields [Levoy, …]!





Computational Displays

computation optics &electronics

human visual system

HDR Display [Seetzen, …]! Super-resolution [Hirsch, Heide, …]! Light Fields [Wetzstein, …]!

Super-resolution [Baker, …]! Light Fields [Levoy, …]!HDR Imaging [Debevec, Nayar, …]!






Computational Light Transport!

physical !world!light!

light!

Computational Light Transport!



LIDAR (light detection and ranging)!Velodyne VLS-128!

3D Imaging for Autonomous Vehicles!

3D Imaging for Smartphones!

iPhone X!

HTC Vive!

HTC Vive Lighthouse!

3D Imaging for VR/AR!

Direct Time-of-Flight 3D Imaging!

Challenges!

1.  Light efficiency!

2.  High-speed time stamping!

Solution: time-correlated single photon counting (TCSPC)!

hist. 1!

picosecond laser!

SPAD sensor!SPAD sensor

*!8-bit timestamp!of photon event! Time-to-Digital!

Converter (TDC)!

hist. 1

scene!

picosecond laser

Single-photon Avalanche Diode (SPAD)!

y (2

56 p

ixel

s)!

t (256 bins)!spatiotemporal image!

...!

hist. 1!hist. 2!

hist. 256!

picosecond laser!

SPAD sensor!SPAD sensor

*!8-bit timestamp!of photon event! Time-to-Digital!

Converter (TDC)!

hist. 1hist. 2

hist. 256scene!

picosecond laser

Single-photon Avalanche Diode Array!

glowing fiber!optic cable!

picosecond!laser!

regular image!

“transient” image!

SPAD output! regular image!x (320 pixels)!

y (2

56 p

ixel

s)!

y (2

56 p

ixel

s)!

t (256 bins)!

scan

line!

LinoSPAD Scanning Procedure!

x (320 pixels)!

y (2

56 p

ixel

s)!


regular image!

x (320 pixels)!

y (2

56 p

ixel

s)!

x (320 pixels)!

y (2

56 p

ixel

s)!


regular image!transient image!

Reconstructing Transient Images!log likelihood of image formation !

model with Poisson noise !prior on 3D spatio-temporal data, !e.g. 3DTV, NLM, BM3D, ...!

nonnegativity constraints!

Splitting-based ADMM Solver:!

O’Toole et al., CVPR 2017!

3D deconvolution, closed-form solution!

Poisson denoising, closed-form solution!

projection, closed-form solution!

proximal operator of prior, closed-form solution!sum of terms, closed-form solution!

Transient Imaging Results!

regular image!regular image!



transient image!regular image!



regular image!regular image!



transient image!regular image!


Pushing the Limits of Transient Imaging!

Acquisition at 25 Hz with 64x80 resolution!

transient image!(RAW)!

transient image!(processed)!

scene under laser light!!

Lindell et al., ICCP 2018!

scene under laser light!!

transient video (processed)!Lindell et al., ICCP 2018!

Applications of Transient Imaging!

•  depth estimation!•  direct-global illumination separation!

•  light transport analysis!•  fundamentally new imaging modality that could enable new capabilities

for image processing & computer vision algorithms … ongoing work!

•  enables Non-line-of-sight (NLOS) Imaging!

Non-line-of-sight Imaging!

Raskar et al. 2012!

Non-line-of-sight Imaging!Non-line-of-sight Imaging

Raskar et al. 2012!

sensor!

light source!occluder!

histogram!

timestamp (nanoseconds)!

phot

ons

dete

cted

!

direct!reflection!

indirect!reflection!

direct!reflection!

indirect!reflection!

sensor!


phot

ons

dete

cted

!

t r =ct

2

histogram!


sensor!


phot

ons

dete

cted

! histogram!


sensor!


imaging area!

x

0

y0ph

oton

s de

tect

ed! histogram!


sensor!


imaging area!

x

0

y0x

0y0

t

3D measurements!

x

0

y0x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

geometric term!radiometric term!3D measurements! hidden 3D volume!

x

0

y0x

0y0

t

3D measurements!

x

0

y0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

3D measurements! geometric term!radiometric term! hidden 3D volume!

) =

ZZZ

⌦

ZZZ

⌦

ZZZ1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

geometric termradiometric term hidden 3D volume

x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

x

y0x

0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

3D measurements! hidden 3D volume!

dx dy dz

geometric term!radiometric term!

⌧(x0, y

0, t) =

ZZZ

⌦

ZZZ

⌦

ZZZ1

r

2l r

2� (rl + r � tc)

3D measurements geometric termradiometric term

x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

x

0

y0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

3D measurements!

⌧(x0, y

0, t) =

ZZZ

⌦

ZZZ

⌦

ZZZ

3D measurements

rlr

geometric term!radiometric term! hidden 3D volume!

· ⇢(x, y, z) dx dy dz

hidden 3D volume

x

0

y0x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

x

0

y0x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

A⌧ ⇢= ⇥

NLOS image formation mode:!

⌧ = A⇢measurements! unknown volume!

n3 ⇥ 1n3 ⇥ 1

transport matrix!n3 ⇥ n3

B on [Vel ]!

Flops: !O(n5)

O(n3)Memory: !

Iterative Inversion [Gupta 12, Wu 12, Heide 13]!

Flops: !

Memory: ! O(n5)

O(n5) per iter.!

PROBLEM: ! A extremely large in practice !•! for n=100, A has 1 trillion elements!•! for n=1000, sparse A needs 9 petabyte of memory!

B on [Vel ]

Flops: O(n5)

O(n3)Memory:

Challenges of NLOS Imaging!

1.  Light efficiency, high-speed time stamping!

2.  Efficient Scanning!

3.  Large-scale inverse problem!

4.  Accurate (and invertible) NLOS light transport model!

Confocal Non-line-of-sight Imaging!


1.! Light efficiency, high-speed time stamping ! SPADs!

2.! Efficient Scanning!

3.! Large-scale inverse problem!

4.! Accurate (and invertible) NLOS light transport model!

2. Efficient Scanning

3. Large-scale inverse problem

4. Accurate (and invertible) NLOS light transport model

•! compatible with LIDAR systems!•! low-cost fabrication; silicon & CMOS!

•! increasing availability of detectors!


1.! Light efficiency, high-speed time stamping!





4. Accurate (and invertible) NLOS light transport model

1. Light efficiency, high-speed time stamping

Confocal SPAD-based Scanning Setup!

O’Toole et al., Nature 2018!

Single Photon Avalanche Detector!


Focusing Lens!


Short-Pulsed Laser Illumination (50ps FWHM)!


Beam-Splitter in Coaxial Alignment!


Scanning Galvo Mirror System!


Illumination Path!


Detection Path!






4.! Accurate (and invertible) NLOS light transport model!4. Accurate (and invertible) NLOS light transport model



Transient Light Transport Analysis!

z!zzx!x

2D space!

x0, z0( )


z!zzx!x

t!t

3D space-time!

! x " x0( )2 + y " y0( )2 + z " z0( )2 " tc( )surface of hypercone!

x0, z0( )

r!


z!zzx!x

t!t

3D space-time! Minkowski’s light cone!


z!zzx!x

t!t

shift-invariant convolution with light cone!


z!zzx!x

t!t

spatio-temporal information transfer!


z!zzx!x

t!t

intersection with light cone!


z!x!

t!t

zzx!x

t!t

laterally shift invariant!

axia

lly N

OT

shift

inva

riant!

intersection with light cone! convolution – NOT shift invariant!


z!x!

t!t

zzx!x

t!t

2D SPAD measurements!

x

measured along x!measured along x


z!zzx!x

t!t

3D SPAD measurements!

x

0y0

t

3D measurements!

x

0

y0xx

0y00

t

3D measurements

x

0

y0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

Our approach!!express image formation model as a !3D convolution, by:!

1.! confocalizing measurements!2.! performing a change of variables!

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

2l r

2� (rl + r � tc) · ⇢(x, y, z) dx dy dz

rlr

Our a ch!!express image formation model as a !3D convolution, by:!


Our a ch

express image formation model as a 3D convolution, by:

2. performing a change of variables

⌧(x0, y

0, t) =

ZZZ

⌦

ZZZ

⌦

ZZZ· ⇢(x, y, z) dx dy dz

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0

r

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

4� (2r � tc) · ⇢(x, y, z) dx dy dz



Our a ch



⌧(x0, y

0, t) =

ZZZ

⌦

ZZZ

⌦

ZZZ1

r

4· ⇢(x, y, z) dx dy dz

r =ct

2

hypercone with factor 2!

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0

r

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

4� (2r � tc) · ⇢(x, y, z) dx dy dz



Our a ch



⌧(x0, y

0, t) =

ZZZ

⌦

ZZZ

⌦

ZZZ� (2r � tc) · ⇢(x, y, z) dx dy dz

r =ct

2

= 2ct

!"#

$%&4

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

4� (2r � tc) · ⇢(x, y, z) dx dy dz

O h!express image formation model as a !3D convolution, by:!


O h


1. confocalizing measurements

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

4� (2r � tc) · ⇢(x, y, z) dx dy dz

⌧(x0, y

0, t) =

ZZZ

⌦

1

r

4� (2r � tc) · ⇢(x, y, z) dx dy dz

2ct

!"#

$%&4

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0O h!express image formation model as a !3D convolution, by:!


O h



v

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdydu

v

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdydu

x

0y0

3D measurements!

x

0

y0

3D measurements

x

0

y0

⌧ ⇢= ⇤a

O h!express image formation model as a !3D convolution, by:!


O h



v

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdydu

v

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdyduv

3/2⌧(x0

, y

0,

2

c

pv) =

ZZZ

⌦

1

2pu

�

�(x0 � x)2 + (y0 � y)2 + u� v

�· ⇢(x, y,

pu)dxdydu

Backpropagation [Velten 12, Buttafava 15]!

Flops: !O(n5)

O(n3)Memory: !


Flops: !

Memory: ! O(n5)

O(n5) per iter.!

NLOS image formation mode:!


n3 ⇥ 1n3 ⇥ 1transport matrix!

n3 ⇥ n3

measurements! unknown volume!blur kernel!

Confocal NLOS image formation mode:!

⌧ = a ⇤ ⇢n⇥ n⇥ n n⇥ n⇥ nn⇥ n⇥ n

B p p gation [Velten 12, But a 15]!

Flops: !O(n5)

O(n3)Memory: !


Flops: !

Memory: ! O(n5)

O(n5) per iter.!

NLOS im de:!



n3 ⇥ n3

B p p gation [Velten 12, But a 15]

Flops: O(n5)

O(n3)Memory:

Iterative Inversion [Gupta 12, Wu 12, Heide 13]

Flops:

Memory: O(n5)

O(n5) per iter.

NLOS im de:

⌧ = A⇢measurements unknown volume

n3 ⇥ 1n3 ⇥ 1transport matrix

n3 ⇥ n3

measurements! unknown volume!blur kernel!

Confocal NLOS image formation mode:!

3D Deconvolution with Light Cone Transform![O’Toole et al. 2018, Nature (to appear)]!

Flops: !

O(n3)Memory: !

O(n3log(n))

⌧ = a ⇤ ⇢n⇥ n⇥ n n⇥ n⇥ nn⇥ n⇥ n

NLOS im de:!



n3 ⇥ n3

B Velten 12, But a 15]!

Flops: !

Memory: !

Ite Inversion [Gupta 12, Wu 12, Heide 13]!

Flops: !

Memory: !

per iter.!

O(n5)

O(n3)

O(n5)

O(n5)

NLOS im de:

⌧ = A⇢measurements unknown volume

n3 ⇥ 1n3 ⇥ 1transport matrix

n3 ⇥ n3

B Velten 12, But a 15]

Flops:

Memory:

Ite Inversion [Gupta 12, Wu 12, Heide 13]

Flops:

Memory:

per iter.

O(n5)

O(n3)

O(n5)

O(n5)

x

0

y0

t

measurements!

measurements!

Step 1: resample!and attenuate!along -axis!

x

0

y0

t

x

0

y0

t

v

Step 2: 3D !convolution!

x

0

y0

t

*!

t

x

0

y0

convolution kernel!

inverse filter!

x

y

v u

measurements!

1: resample!and attenuate!along -axis!

x

0

yy0

t

measurements

1: resampleand attenuatealong -axistalong -axist

t

2: 3D !convolution!

inverse filter!

1: resample!and attenuate!along -axis!

x

0

y0

t

x

0

y0

recovered volume!

x

y

z

z

x

y

v u

convolution kernel!measurements!

*!2: 3D

convolution

inverse filter

1: resampleand attenuatealong -axistalong -axist

x

0

yy0

t

x

0

yy0

v

convolution kernelmeasurements

*

Step 3: resample!and attenuate!along -axis!

laser spot! confocal NLOS system!

hidden object!

occluder!

1.25 meters!

laser spot!

x

0

y0

Exposure time (per sample): 1 sec!Spatial resolution: 64x64!

Retroreflective: Yes!

Experiments!

Exposure time (per sample): 0.1 sec!Spatial resolution: 64x64!

Retroreflective: Yes!

Experiments!

Exposure time (per sample): 1 sec!Spatial resolution: 64x64!

Retroreflective: No!

Experiments!

occluder!

object!wall!

prototype!

wall!

x

zx

y

occluder

objectwall

prototype

x

yz

Towards Real-time NLOS Imaging!

Resolution Limits of NLOS Imaging!

… slides omitted for confidentiality …!

Stanford Computational Imaging Lab!

Time-of-Flight Imaging!Image Optimization!

Light Field Cameras! Computational Microscopy!

Virtual & Augmented Reality!


Gordon Wetzstein!Computational Imaging Lab!Stanford University!

www.computationalimaging.org!

Matt O’Toole! David Lindell! Felix Heide!

Thanks to:!•! Edoardo Charbon, EPFL!•! Samuel Burri, EPFL!•! Pierre-Yves Cattin, Fastree3D!

Computational Single-photon Imaging · Computational Single-photon Imaging! Gordon Wetzstein!...

Documents

Transcript of Computational Single-photon Imaging · Computational Single-photon Imaging! Gordon Wetzstein!...