Depth and Transient Imaging with Compressive SPAD Array ... · Depth and Transient Imaging with...
1
Depth and Transient Imaging with Compressive SPAD Array Cameras Qilin Sun Xiong Dun Yifan Peng Wolfgang Heidrich KAUST KAUST KAUST UBC&KAUST Existing time-of-flight depth imaging and transient imaging systems are limited either in terms of spatial&temporal resolution or are prohibitively bulky&expensive. Microlens Array SPAD Random Pattern (a) Picosecond laser (e) Reimaging lens (b) Imaging lens (c) DMD (d) TIR Prism (f) SPAD array = arg min 1 2 − + () (a) Recovered data (b) Raw data SPAD Reimaging Lens DLP4500 Imaging lens Holder DMLA Die SPAD Sensor START END DMD Pattern Ready? Capturing Done? Control Signal Patten Data Yes Yes Data Stream Control&Trigger Synchronizing SPAD Picosecond Laser min , ; , () − Where ∈ ××× is the 4D data sharpened in the temporal domain after modulation, ∈ ×× is the 3D signal under evaluation, is an operator that maps the random patterns to individual pixels at each layer, and is 3D TV regularizer. By jointly designing optics, mechanics, electronics, and computation, we overcome the spatial resolution(64×32) limit of Single Photon Avalanche Diode(SPAD) arrays by compressive sensing(image resolution up to 800×400) and realize a temporal resolution of ~20 picoseconds via a physical temporal PSF model. Where is the raw sensor data We present the sharpened sensor data for each pixel as a sequence of Gaussians . As the picosecond laser pulse is approximately Gaussian and has a FWHM ~80ps, the target Gaussian pulse is shown and denoted as who has fixed . Therefore, we can estimate the depth through solving a least square problem. Our SPAD array is working in TCSPC mode and the measurements from each SPAD pixel could be reconstructed independently with tiling artifacts addressed. Light Concentrated by a lens 0ps 40ps 80ps 120ps 160ps 200ps 240ps 280ps 320ps Scene 0ps 80ps 400ps 480ps 560ps 160ps 240ps 640ps 320ps Scene Transient Scene Background and Our Solution Models and Optimization Prototype Working Flow Resolution Analysis Results ~8 × + 50 40 30 20 10 0 Depth/mm Marked Pixels profile 50 40 30 20 10 0 Depth/mm depth intensity raw 0 20 40 60 Depth/mm depth intensity raw 0 100 200 300 400 500 600 700 800 Compressive Depth Imaging 10mm 10mm KAUST * 0 1 2 3 Time/ns 200 400 0 Photon Counts 0 1 2 3 Time/ns 50 100 0 Photon Counts 0 1 2 3 0.5 1 0 Time/ns Time/ns 0 1 2 3 0.5 1 0 Low Pass Filter -1 = Target Gaussian Pulse RC Response Model at High Frequency FWHM≈80ps * ~1ns () Background noise &Dark counts 200 400 0 Photon Counts 200 400 0 Photon Counts Fitting Result (t) 0 1 2 Time/ns 1 2 Time/ns 0 3 Background noise &Dark counts Removed Optical Parameters in Experiment E/F Mounts TIR Prism DMD Bandpass Filter Laser DMLA Auxiliary Optics WaveLength 655nm Focal Length 1.035mm Imaging Canon 85mm Lens Average Power ~1mW Structure 2 period 2 4 Phase Level Reimaging Inverted 0.9X Edmund Double Side Telecentric FWHM ~80ps Effiency(/20) 52.87% DMD TI DLP4500 Repetition Rate 50MHz Fabrication Lithography(0.7m) +RIE etching DMD optical system Shown Below 1 2 1 1 2 − 1 2 2 = 17.43° SPAD Array Settings Integration Time Harware Binning Gate Width Shift per Cycle Photons Received 52s 1280 830ps 20ps ~40-60/pixel Raw Reconstructed
Transcript of Depth and Transient Imaging with Compressive SPAD Array ... · Depth and Transient Imaging with...
IEEE 2017 Conference on
Computer Vision and Pattern
Recognition
Depth and Transient Imaging with Compressive SPAD Array CamerasQilin Sun Xiong Dun Yifan Peng Wolfgang Heidrich
KAUST KAUST KAUSTUBC&KAUST
Existing time-of-flight depth imaging and transient imaging systems are
limited either in terms of spatial&temporal resolution or are prohibitively
bulky&expensive.
Microlens Array
SPAD
Random Pattern
(a) Picosecond laser
(e) Reimaging
lens
(b) Imaging lens
(c) DMD
(d) TIR Prism
(f) SPAD array
𝑿 = arg min𝑿
1
2𝚿 𝑿 − 𝒀 𝟐
𝟐 +
𝑖
𝝀𝑖 𝑫𝑖(𝑿)
(a) Recovered data
(b) Raw data
SPADReimaging Lens
DLP4500
Imaging lens
HolderDMLA
Die SPAD
Sensor
START
END
DMD
Pattern
Ready?
Capturing
Done?
Control Signal
Patten
Data
Yes
Yes
Data
Stream
Control&Trigger
SynchronizingSPAD
Picosecond Laser
min𝐴,𝜇
𝑮 𝑡; 𝐴, 𝜇 𝚷(𝑡) − 𝒀𝟐
𝟐
Where ∈ 𝑅𝐾×𝑇×𝑛×𝑚 is the 4D data sharpened in the temporal domain
after modulation, 𝑿 ∈ 𝑅𝑇×𝑁×𝑀 is the 3D signal under evaluation, 𝚿 is an
operator that maps the random patterns to individual pixels at each layer,
and 𝑫 is 3D TV regularizer.
By jointly designing optics, mechanics, electronics, and computation, we
overcome the spatial resolution(64×32) limit of Single Photon Avalanche
Diode(SPAD) arrays by compressive sensing(image resolution up to
800×400) and realize a temporal resolution of ~20 picoseconds via a
physical temporal PSF model.
Where 𝒀 is the raw sensor data We present the sharpened sensor data 𝒀𝑖for each pixel 𝑖 as a sequence of Gaussians 𝑮.
As the picosecond laser pulse is approximately Gaussian and has a
FWHM ~80ps, the target Gaussian pulse is shown and denoted as 𝐆 who
has fixed 𝜎. Therefore, we can estimate the depth 𝜇 through solving a
least square problem.
Our SPAD array is working in TCSPC mode and the measurements from
each SPAD pixel could be reconstructed independently with tiling artifacts
addressed.
Light Concentrated by a lens
0ps
40ps
80ps
120ps
160ps
200ps
240ps
280ps
320ps
Scene
0ps
80ps
400ps
480ps
560ps160ps
240ps 640ps
320ps Scene
Transient Scene
Background and Our Solution
Models and Optimization
Prototype Working Flow
Resolution Analysis
Results
~8×+
50 40 30 20 10 0
Depth/mmM
ark
ed
Pix
els
profile
50 40 30 20 10 0
Depth/mm
depth intensity raw
0
20
40
60
Depth/mm
depth
intensity
raw
0
100
200
300
400
500
600
700
800
Compressive Depth Imaging
10mm
10mm
KAUST
*
0 1 2 3 Time/ns
200
400
0
Photo
n C
ounts
0 1 2 3 Time/ns
50
100
0
Photo
n C
ounts
0 1 2 3
0.5
1
0Time/ns Time/ns0 1 2 3
0.5
1
0
Low Pass
Filter
-1 =Target Gaussian Pulse
RC Response Model
at High Frequency
FWHM≈80ps
*~1ns
𝚷(𝑡)
Background noise
&Dark counts
200
400
0
Photo
n C
ounts
200
400
0
Photo
n C
ounts
Fitting Result
𝐆(t)
0 1 2 Time/ns 1 2 Time/ns03
Background noise
&Dark counts Removed
Optical Parameters in Experiment
E/F Mounts
TIR Prism
DMD
Bandpass Filter
Laser DMLA Auxiliary Optics
WaveLength 655nm Focal Length 1.035mm Imaging Canon 85mm Lens
Average Power ~1mW Structure2𝜋 period
24 Phase LevelReimaging
Inverted 0.9X EdmundDouble Side Telecentric
FWHM ~80ps Effiency(𝑓/20) 52.87% DMD TI DLP4500
Repetition Rate 50MHz FabricationLithography(0.7𝜇m)
+RIE etchingDMD optical
systemShown Below
𝛼1
𝛼2 𝜃𝑖1𝜃𝑜1
𝛼2 − 𝛼1
𝛾
𝜃𝑖2
𝜃𝑜2
𝛾 = 17.43°
SPAD Array Settings
Integration Time
HarwareBinning
Gate Width
Shiftper Cycle
PhotonsReceived
52𝜇s 1280 830ps 20ps ~40-60/pixel
Raw Reconstructed
