Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse...
Transcript of Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse...
![Page 1: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/1.jpg)
Inverse Light Transport (and next Separation of Global and
Direct Illumination)
CS434
Daniel G. Aliaga Department of Computer Science
Purdue University
![Page 2: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/2.jpg)
Inverse Light Transport
• Light Transport
– Model transfer of light from “source” (e.g., light/projector) to “destination” (e.g., eye/camera) modulated by scene
• Inverse Light Transport
– Given a photograph of an unknown scene, compute (or decompose) the light into the needed source(s)
light scene camera
light scene camera
![Page 3: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/3.jpg)
![Page 4: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/4.jpg)
Topics
• A Theory of Inverse Light Transport
– Seitz et al., ICCV 2005
• Radiometric Compensation and Inverse Light Transport
– Wetzstein et al., PG 2007
– Work at Purdue
![Page 5: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/5.jpg)
Topics
• A Theory of Inverse Light Transport
– Seitz et al., ICCV 2005
• Radiometric Compensation and Inverse Light Transport
– Wetzstein et al., PG 2007
– Work at Purdue
![Page 6: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/6.jpg)
Theory of Inverse Light Transport
• Given a photo, decompose it into a sum of n-bounce images
• Each bounce image records the light that bounces ‘n’ times before reaching the camera
• Formulated for Lambertian scenes
![Page 7: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/7.jpg)
Theory of Inverse Light Transport
nIIIII ...321
I1 = direct illumination image
Ii = indirect illumination image, for i ≥ 2
…by removing the Ii‘s the photographs are converted into a form more amenable to existing graphics/vision processing algorithms
![Page 8: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/8.jpg)
Formulation
• Outward light field Lout from x to point y is
• Recall Rendering Equation (or synthetic light-transport equation):
),(),(),(,...3,21
yxLyxLyxL outoutout
s
dxxxIxxxxxxxgxxI '')'','()'',',()',()',()',(
![Page 9: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/9.jpg)
Formulation
• Outward light field Lout from x to point y is
• Rewrite Rendering Equation as
),(),(),(,...3,21
yxLyxLyxL outoutout
'
1'),'(),,'(),(),(
x
outoutout dxxxLyxxAyxLyxL
),,'( yxxA is the proportion of irradiance from x’ to x that gets transported to y
![Page 10: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/10.jpg)
Formulation
'
1'),'(),,'(),(),(
x
outoutout dxxxLyxxAyxLyxL
j
outoutout jLjiAiLiL ][],[][][1
outoutout ALLL 1
?outL
for small facets i,j
![Page 11: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/11.jpg)
Formulation
'
1'),'(),,'(),(),(
x
outoutout dxxxLyxxAyxLyxL
j
outoutout jLjiAiLiL ][],[][][1
outoutout ALLL 1
11)( outout LAIL
for small facets i,j
(well-known) maps a light field containing only direct light to a light field having indirect light…
![Page 12: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/12.jpg)
Cancellation Operator
AIC 1
or
which means C1 “cancels the interreflections” in Lout
11)( outout LAIL
outout LCL 11
111)( outout LCL
![Page 13: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/13.jpg)
Cancellation Operator
So what is all the light “except for the direct illumination”?
outout LCL 11
outout LCL 1
So now the previous “first bounce” indirect light is effectively now the direct illumination component
What is the Lout due to the second bounce of light?
)( 11outout LCLC
![Page 14: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/14.jpg)
Cancellation Operator
So in general,
111 )( nn CICC
outnn
out LCL
where Loutn defines the light field due to the
n-th bounce of light, and
n
noutout LL
![Page 15: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/15.jpg)
Computing C1
For Lambertian scenes, it turns out
111 TTC
where T is a light-impulse response matrix similar to that used for dual photography of a diffuse scene (i.e., it is a diagonal matrix)
where T1 is a matrix of the reciprocals of the diagonal elements of T-1
![Page 16: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/16.jpg)
Examples
![Page 17: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/17.jpg)
Examples
![Page 18: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/18.jpg)
Topics
• A Theory of Inverse Light Transport
– Seitz et al., ICCV 2005
• Radiometric Compensation and Inverse Light Transport
– Wetzstein et al., PG 2007
– Work at Purdue (Aliaga et al. TOG 2012)
![Page 19: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/19.jpg)
Radiometric Compensation and Inverse Light Transport
• Single projector case (Wetzstein et al. 2007)
– Theoretically simple, just invert the light transport:
– Computation can be expense
• Bimber spatially decomposes T and uses GPU
• Success of spatial decomposition is scene dependent
TPC
CTP 1
01 CTP
![Page 20: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/20.jpg)
Example
![Page 21: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/21.jpg)
Example
![Page 22: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/22.jpg)
Radiometric Compensation and Inverse Light Transport
• Multiple projector case
– More complicated…
– Need to constrain solution and is computationally much more challenging
![Page 23: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/23.jpg)
Use ILT to alter appearance • Alter the appearance of the object’s surface
![Page 24: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/24.jpg)
Single Projector Appearance Editing
![Page 25: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/25.jpg)
Multi-Projector Appearance Editing • Partially overlapping projectors
![Page 26: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/26.jpg)
Multi-Projector Appearance Editing • Fully superimposed projectors
![Page 27: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/27.jpg)
Multi-Projector Appearance Editing • Use higher resolution camera to capture
projector pixel interaction
![Page 28: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/28.jpg)
Overlapping Projector Interaction
![Page 29: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/29.jpg)
Overlapping Projector Interaction
proj. 1
![Page 30: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/30.jpg)
Overlapping Projector Interaction
proj. 1 proj. 2
![Page 31: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/31.jpg)
Overlapping Projector Interaction
proj. 1 proj. 2 proj. 3
![Page 32: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/32.jpg)
Overlapping Projector Interaction
proj. 1 proj. 2 proj. 3
![Page 33: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/33.jpg)
Overlapping Projector Interaction • Model metapixels and their interaction within
and across projectors
proj. 1 proj. 2 proj. 3
contribution from 3 projector pixels
![Page 34: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/34.jpg)
Challenges
• Efficiently model proj-proj-cam pixel interactions
![Page 35: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/35.jpg)
Challenges
• Constrain solution to produce valid projection values
![Page 36: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/36.jpg)
Challenges
• Constrain solution to produce valid projection values
![Page 37: Inverse Light Transport (and next Separation of Global and ......Topics • A Theory of Inverse Light Transport –Seitz et al., ICCV 2005 • Radiometric Compensation and Inverse](https://reader036.fdocuments.net/reader036/viewer/2022081620/6119c8be4f6f2d014f6577c5/html5/thumbnails/37.jpg)
Examples