Post on 04-Jun-2020
Computational Single-photon Imaging!
Gordon Wetzstein!Stanford University!
ISSW 2/28/2018!
www.computationalimaging.org!
Muybridge’s Multi-Camera Array at Stanford!
Computational Imaging!
Computational Displays!
Computational Cameras!
computation! optics &!electronics!
human visual system!
optics! computation!sensing!
Computational Displays!
computation! optics &!electronics!
human visual system!
optics! computation!sensing!
Computational Imaging!
Computational Cameras!
Computational Displays
computation optics &electronics
human visual system
Computational Displays!
HDR Imaging [Debevec, Nayar, …]! Super-resolution [Baker, …]! Light Fields [Levoy, …]!
computation! optics &!electronics!
human visual system!
Computational Imaging!
Computational Cameras!
Computational Displays
computation optics &electronics
human visual system
HDR Display [Seetzen, …]! Super-resolution [Hirsch, Heide, …]! Light Fields [Wetzstein, …]!
Super-resolution [Baker, …]! Light Fields [Levoy, …]!HDR Imaging [Debevec, Nayar, …]!
Computational Imaging!
Computational Displays!
Computational Cameras!
Computational Displays!
Computational Cameras!
Computational Light Transport!
physical !world!light!
light!
Computational Light Transport!
Computational Displays!
Computational Cameras!
LIDAR (light detection and ranging)!Velodyne VLS-128!
3D Imaging for Autonomous Vehicles!
3D Imaging for Smartphones!
iPhone X!
HTC Vive!
HTC Vive Lighthouse!
3D Imaging for VR/AR!
Direct Time-of-Flight 3D Imaging!
Direct Time-of-Flight 3D Imaging!
Challenges!
1. Light efficiency!
2. High-speed time stamping!
Solution: time-correlated single photon counting (TCSPC)!
hist. 1!
picosecond laser!
SPAD sensor!SPAD sensor
*!8-bit timestamp!of photon event! Time-to-Digital!
Converter (TDC)!
hist. 1
scene!
picosecond laser
Single-photon Avalanche Diode (SPAD)!
y (2
56 p
ixel
s)!
t (256 bins)!spatiotemporal image!
...!
hist. 1!hist. 2!
hist. 256!
picosecond laser!
SPAD sensor!SPAD sensor
*!8-bit timestamp!of photon event! Time-to-Digital!
Converter (TDC)!
hist. 1hist. 2
hist. 256scene!
picosecond laser
Single-photon Avalanche Diode Array!
glowing fiber!optic cable!
picosecond!laser!
regular image!
“transient” image!
SPAD output! regular image!x (320 pixels)!
y (2
56 p
ixel
s)!
y (2
56 p
ixel
s)!
t (256 bins)!
scan
line!
LinoSPAD Scanning Procedure!
x (320 pixels)!
y (2
56 p
ixel
s)!
LinoSPAD Scanning Procedure!
regular image!
x (320 pixels)!
y (2
56 p
ixel
s)!
x (320 pixels)!
y (2
56 p
ixel
s)!
LinoSPAD Scanning Procedure!
regular image!transient image!
Reconstructing Transient Images!log likelihood of image formation !
model with Poisson noise !prior on 3D spatio-temporal data, !e.g. 3DTV, NLM, BM3D, ...!
nonnegativity constraints!
Splitting-based ADMM Solver:!
O’Toole et al., CVPR 2017!
3D deconvolution, closed-form solution!
Poisson denoising, closed-form solution!
projection, closed-form solution!
proximal operator of prior, closed-form solution!sum of terms, closed-form solution!
Transient Imaging Results!
regular image!regular image!
O’Toole et al., CVPR 2017!
Transient Imaging Results!
transient image!regular image!
O’Toole et al., CVPR 2017!
Transient Imaging Results!
regular image!regular image!
O’Toole et al., CVPR 2017!
Transient Imaging Results!
transient image!regular image!
O’Toole et al., CVPR 2017!
Pushing the Limits of Transient Imaging!
Acquisition at 25 Hz with 64x80 resolution!
transient image!(RAW)!
transient image!(processed)!
scene under laser light!!
Lindell et al., ICCP 2018!
scene under laser light!!
transient video (processed)!Lindell et al., ICCP 2018!
Applications of Transient Imaging!
• depth estimation!• direct-global illumination separation!
• light transport analysis!• fundamentally new imaging modality that could enable new capabilities
for image processing & computer vision algorithms … ongoing work!
• enables Non-line-of-sight (NLOS) Imaging!
Non-line-of-sight Imaging!
Raskar et al. 2012!
Non-line-of-sight Imaging!Non-line-of-sight Imaging
Raskar et al. 2012!
sensor!
light source!occluder!
histogram!
timestamp (nanoseconds)!
phot
ons
dete
cted
!
direct!reflection!
indirect!reflection!
direct!reflection!
indirect!reflection!
sensor!
light source!occluder!
phot
ons
dete
cted
!
t r =ct
2
histogram!
timestamp (nanoseconds)!
sensor!
light source!occluder!
phot
ons
dete
cted
! histogram!
timestamp (nanoseconds)!
sensor!
light source!occluder!
imaging area!
x
0
y0ph
oton
s de
tect
ed! histogram!
timestamp (nanoseconds)!
sensor!
light source!occluder!
imaging area!
x
0
y0x
0y0
t
3D measurements!
x
0
y0x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
geometric term!radiometric term!3D measurements! hidden 3D volume!
x
0
y0x
0y0
t
3D measurements!
x
0
y0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
3D measurements! geometric term!radiometric term! hidden 3D volume!
) =
ZZZ
⌦
ZZZ
⌦
ZZZ1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
geometric termradiometric term hidden 3D volume
x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
x
y0x
0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
3D measurements! hidden 3D volume!
dx dy dz
geometric term!radiometric term!
⌧(x0, y
0, t) =
ZZZ
⌦
ZZZ
⌦
ZZZ1
r
2l r
2� (rl + r � tc)
3D measurements geometric termradiometric term
x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
x
0
y0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
3D measurements!
⌧(x0, y
0, t) =
ZZZ
⌦
ZZZ
⌦
ZZZ
3D measurements
rlr
geometric term!radiometric term! hidden 3D volume!
· ⇢(x, y, z) dx dy dz
hidden 3D volume
x
0
y0x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
x
0
y0x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
A⌧ ⇢= ⇥
NLOS image formation mode:!
⌧ = A⇢measurements! unknown volume!
n3 ⇥ 1n3 ⇥ 1
transport matrix!n3 ⇥ n3
B on [Vel ]!
Flops: !O(n5)
O(n3)Memory: !
Iterative Inversion [Gupta 12, Wu 12, Heide 13]!
Flops: !
Memory: ! O(n5)
O(n5) per iter.!
PROBLEM: ! A extremely large in practice !•! for n=100, A has 1 trillion elements!•! for n=1000, sparse A needs 9 petabyte of memory!
B on [Vel ]
Flops: O(n5)
O(n3)Memory:
Challenges of NLOS Imaging!
1. Light efficiency, high-speed time stamping!
2. Efficient Scanning!
3. Large-scale inverse problem!
4. Accurate (and invertible) NLOS light transport model!
Confocal Non-line-of-sight Imaging!
Challenges of NLOS Imaging!
1.! Light efficiency, high-speed time stamping ! SPADs!
2.! Efficient Scanning!
3.! Large-scale inverse problem!
4.! Accurate (and invertible) NLOS light transport model!
2. Efficient Scanning
3. Large-scale inverse problem
4. Accurate (and invertible) NLOS light transport model
•! compatible with LIDAR systems!•! low-cost fabrication; silicon & CMOS!
•! increasing availability of detectors!
Challenges of NLOS Imaging!
1.! Light efficiency, high-speed time stamping!
2.! Efficient Scanning!
3.! Large-scale inverse problem!
4.! Accurate (and invertible) NLOS light transport model!
3. Large-scale inverse problem
4. Accurate (and invertible) NLOS light transport model
1. Light efficiency, high-speed time stamping
Confocal SPAD-based Scanning Setup!
O’Toole et al., Nature 2018!
Single Photon Avalanche Detector!
O’Toole et al., Nature 2018!
Focusing Lens!
O’Toole et al., Nature 2018!
Short-Pulsed Laser Illumination (50ps FWHM)!
O’Toole et al., Nature 2018!
Beam-Splitter in Coaxial Alignment!
O’Toole et al., Nature 2018!
Scanning Galvo Mirror System!
O’Toole et al., Nature 2018!
Illumination Path!
O’Toole et al., Nature 2018!
Detection Path!
O’Toole et al., Nature 2018!
Challenges of NLOS Imaging!
1.! Light efficiency, high-speed time stamping!
2.! Efficient Scanning!
3.! Large-scale inverse problem!
4.! Accurate (and invertible) NLOS light transport model!4. Accurate (and invertible) NLOS light transport model
1. Light efficiency, high-speed time stamping
2. Efficient Scanning
Transient Light Transport Analysis!
z!zzx!x
2D space!
x0, z0( )
Transient Light Transport Analysis!
z!zzx!x
t!t
3D space-time!
! x " x0( )2 + y " y0( )2 + z " z0( )2 " tc( )surface of hypercone!
x0, z0( )
r!
Transient Light Transport Analysis!
z!zzx!x
t!t
3D space-time! Minkowski’s light cone!
Transient Light Transport Analysis!
z!zzx!x
t!t
shift-invariant convolution with light cone!
Transient Light Transport Analysis!
z!zzx!x
t!t
spatio-temporal information transfer!
Transient Light Transport Analysis!
z!zzx!x
t!t
intersection with light cone!
Transient Light Transport Analysis!
z!x!
t!t
zzx!x
t!t
laterally shift invariant!
axia
lly N
OT
shift
inva
riant!
intersection with light cone! convolution – NOT shift invariant!
Transient Light Transport Analysis!
z!x!
t!t
zzx!x
t!t
2D SPAD measurements!
x
measured along x!measured along x
Transient Light Transport Analysis!
z!zzx!x
t!t
3D SPAD measurements!
x
0y0
t
3D measurements!
x
0
y0xx
0y00
t
3D measurements
x
0
y0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
Our approach!!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
2l r
2� (rl + r � tc) · ⇢(x, y, z) dx dy dz
rlr
Our a ch!!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
Our a ch
express image formation model as a 3D convolution, by:
2. performing a change of variables
⌧(x0, y
0, t) =
ZZZ
⌦
ZZZ
⌦
ZZZ· ⇢(x, y, z) dx dy dz
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0
r
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
4� (2r � tc) · ⇢(x, y, z) dx dy dz
Our a ch!!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
Our a ch
express image formation model as a 3D convolution, by:
2. performing a change of variables
⌧(x0, y
0, t) =
ZZZ
⌦
ZZZ
⌦
ZZZ1
r
4· ⇢(x, y, z) dx dy dz
r =ct
2
hypercone with factor 2!
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0
r
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
4� (2r � tc) · ⇢(x, y, z) dx dy dz
Our a ch!!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
Our a ch
express image formation model as a 3D convolution, by:
2. performing a change of variables
⌧(x0, y
0, t) =
ZZZ
⌦
ZZZ
⌦
ZZZ� (2r � tc) · ⇢(x, y, z) dx dy dz
r =ct
2
= 2ct
!"#
$%&4
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
4� (2r � tc) · ⇢(x, y, z) dx dy dz
O h!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
O h
express image formation model as a 3D convolution, by:
1. confocalizing measurements
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
4� (2r � tc) · ⇢(x, y, z) dx dy dz
⌧(x0, y
0, t) =
ZZZ
⌦
1
r
4� (2r � tc) · ⇢(x, y, z) dx dy dz
2ct
!"#
$%&4
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0O h!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
O h
express image formation model as a 3D convolution, by:
1. confocalizing measurements
v
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdydu
v
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdydu
x
0y0
3D measurements!
x
0
y0
3D measurements
x
0
y0
⌧ ⇢= ⇤a
O h!express image formation model as a !3D convolution, by:!
1.! confocalizing measurements!2.! performing a change of variables!
O h
express image formation model as a 3D convolution, by:
1. confocalizing measurements
v
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdydu
v
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdyduv
3/2⌧(x0
, y
0,
2
c
pv) =
ZZZ
⌦
1
2pu
�
�(x0 � x)2 + (y0 � y)2 + u� v
�· ⇢(x, y,
pu)dxdydu
Backpropagation [Velten 12, Buttafava 15]!
Flops: !O(n5)
O(n3)Memory: !
Iterative Inversion [Gupta 12, Wu 12, Heide 13]!
Flops: !
Memory: ! O(n5)
O(n5) per iter.!
NLOS image formation mode:!
⌧ = A⇢measurements! unknown volume!
n3 ⇥ 1n3 ⇥ 1transport matrix!
n3 ⇥ n3
measurements! unknown volume!blur kernel!
Confocal NLOS image formation mode:!
⌧ = a ⇤ ⇢n⇥ n⇥ n n⇥ n⇥ nn⇥ n⇥ n
B p p gation [Velten 12, But a 15]!
Flops: !O(n5)
O(n3)Memory: !
Iterative Inversion [Gupta 12, Wu 12, Heide 13]!
Flops: !
Memory: ! O(n5)
O(n5) per iter.!
NLOS im de:!
⌧ = A⇢measurements! unknown volume!
n3 ⇥ 1n3 ⇥ 1transport matrix!
n3 ⇥ n3
B p p gation [Velten 12, But a 15]
Flops: O(n5)
O(n3)Memory:
Iterative Inversion [Gupta 12, Wu 12, Heide 13]
Flops:
Memory: O(n5)
O(n5) per iter.
NLOS im de:
⌧ = A⇢measurements unknown volume
n3 ⇥ 1n3 ⇥ 1transport matrix
n3 ⇥ n3
measurements! unknown volume!blur kernel!
Confocal NLOS image formation mode:!
3D Deconvolution with Light Cone Transform![O’Toole et al. 2018, Nature (to appear)]!
Flops: !
O(n3)Memory: !
O(n3log(n))
⌧ = a ⇤ ⇢n⇥ n⇥ n n⇥ n⇥ nn⇥ n⇥ n
NLOS im de:!
⌧ = A⇢measurements! unknown volume!
n3 ⇥ 1n3 ⇥ 1transport matrix!
n3 ⇥ n3
B Velten 12, But a 15]!
Flops: !
Memory: !
Ite Inversion [Gupta 12, Wu 12, Heide 13]!
Flops: !
Memory: !
per iter.!
O(n5)
O(n3)
O(n5)
O(n5)
NLOS im de:
⌧ = A⇢measurements unknown volume
n3 ⇥ 1n3 ⇥ 1transport matrix
n3 ⇥ n3
B Velten 12, But a 15]
Flops:
Memory:
Ite Inversion [Gupta 12, Wu 12, Heide 13]
Flops:
Memory:
per iter.
O(n5)
O(n3)
O(n5)
O(n5)
x
0
y0
t
measurements!
measurements!
Step 1: resample!and attenuate!along -axis!
x
0
y0
t
x
0
y0
t
v
Step 2: 3D !convolution!
x
0
y0
t
*!
t
x
0
y0
convolution kernel!
inverse filter!
x
y
v u
measurements!
1: resample!and attenuate!along -axis!
x
0
yy0
t
measurements
1: resampleand attenuatealong -axistalong -axist
t
2: 3D !convolution!
inverse filter!
1: resample!and attenuate!along -axis!
x
0
y0
t
x
0
y0
recovered volume!
x
y
z
z
x
y
v u
convolution kernel!measurements!
*!2: 3D
convolution
inverse filter
1: resampleand attenuatealong -axistalong -axist
x
0
yy0
t
x
0
yy0
v
convolution kernelmeasurements
*
Step 3: resample!and attenuate!along -axis!
laser spot! confocal NLOS system!
hidden object!
occluder!
1.25 meters!
laser spot!
x
0
y0
Exposure time (per sample): 1 sec!Spatial resolution: 64x64!
Retroreflective: Yes!
Experiments!
Exposure time (per sample): 0.1 sec!Spatial resolution: 64x64!
Retroreflective: Yes!
Experiments!
Exposure time (per sample): 0.1 sec!Spatial resolution: 64x64!
Retroreflective: Yes!
Experiments!
Exposure time (per sample): 1 sec!Spatial resolution: 64x64!
Retroreflective: No!
Experiments!
occluder!
object!wall!
prototype!
wall!
x
zx
y
occluder
objectwall
prototype
x
yz
Towards Real-time NLOS Imaging!
Resolution Limits of NLOS Imaging!
Challenges of NLOS Imaging!
1.! Light efficiency, high-speed time stamping!
2.! Efficient Scanning!
3.! Large-scale inverse problem!
4.! Accurate (and invertible) NLOS light transport model!
1. Light efficiency, high-speed time stamping
2. Efficient Scanning
3. Large-scale inverse problem
… slides omitted for confidentiality …!
Stanford Computational Imaging Lab!
Time-of-Flight Imaging!Image Optimization!
Light Field Cameras! Computational Microscopy!
Virtual & Augmented Reality!
Computational Displays!
Gordon Wetzstein!Computational Imaging Lab!Stanford University!
www.computationalimaging.org!
Matt O’Toole! David Lindell! Felix Heide!
Thanks to:!•! Edoardo Charbon, EPFL!•! Samuel Burri, EPFL!•! Pierre-Yves Cattin, Fastree3D!