Inverse Global Illumination:Inverse Global Illumination:Recovering Reflectance Models of Real Recovering Reflectance Models of Real

Scenes from PhotographsScenes from Photographs

Inverse Global Illumination:Inverse Global Illumination:Recovering Reflectance Models of Real Recovering Reflectance Models of Real

Scenes from PhotographsScenes from Photographs

Computer Science Division

University of California at Berkeley

Computer Science Division

University of California at Berkeley

Yizhou Yu, Paul Debevec, Jitendra Malik & Tim Hawkins

Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering

• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire photographs

– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, (Light Field Rendering) etc.

• 1st Generation---- vary viewpoint but not lighting– Recover geometry ( explicit or implicit )

– Acquire photographs

– Facade, Plenoptic Modeling, View Morphing, Lumigraph, Layered Depth Images, (Light Field Rendering) etc.

Image-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and RenderingImage-based Modeling and Rendering

• Photographs are not Reflectance Maps !

• 2nd Generation---- vary viewpoint and lighting for non-diffuse scenes– Recover geometry

– Recover reflectance properties

– Render using light transport simulation

• Photographs are not Reflectance Maps !

• 2nd Generation---- vary viewpoint and lighting for non-diffuse scenes– Recover geometry

– Recover reflectance properties

– Render using light transport simulation

Illumination Radiance

Reflectance

Previous WorkPrevious WorkPrevious WorkPrevious Work

• BRDF Measurement in the Laboratory– [ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97]

• Isolated Objects under Direct Illumination– [ Sato, Wheeler & Ikeuchi 97 ]

• Isolated Objects under General Illumination– [ Yu & Malik 98], [ Debevec 98]

• BRDF Measurement in the Laboratory– [ Ward 92 ], [Dana, Ginneken, Nayar & Koenderink 97]

• Isolated Objects under Direct Illumination– [ Sato, Wheeler & Ikeuchi 97 ]

• Isolated Objects under General Illumination– [ Yu & Malik 98], [ Debevec 98]

The ProblemThe ProblemThe ProblemThe Problem

• General case of multiple objects under mutual illumination has not been studied.

Global IlluminationGlobal IlluminationGlobal IlluminationGlobal Illumination

Reflectance Properties

Radiance Images

Geometry Illumination

Inverse Global IlluminationInverse Global IlluminationInverse Global IlluminationInverse Global Illumination

Reflectance Properties

Radiance Images

Geometry Illumination

Input Radiance ImagesInput Radiance ImagesInput Radiance ImagesInput Radiance Images

[ Debevec & Malik 97]http://www.cs.berkeley.edu/~debevec/HDR

In Detail ... In Detail ... In Detail ... In Detail ...

Geometry and Camera PositionsGeometry and Camera PositionsGeometry and Camera PositionsGeometry and Camera Positions

Light SourcesLight SourcesLight SourcesLight Sources

Synthesized ImagesSynthesized ImagesSynthesized ImagesSynthesized Images

Original Lighting Novel Lighting

OutlineOutlineOutlineOutline

• Diffuse surfaces under mutual illumination

• Non-diffuse surfaces under direct illumination

• Non-diffuse surfaces under mutual illumination

• Diffuse surfaces under mutual illumination

• Non-diffuse surfaces under direct illumination

• Non-diffuse surfaces under mutual illumination

Lambertian Surfaces under Lambertian Surfaces under Mutual IlluminationMutual IlluminationLambertian Surfaces under Lambertian Surfaces under Mutual IlluminationMutual Illumination

j

ijjiii FBEB j

ijjiii FBEB

• Bi, Bj, Ei measured

• Form-factor Fij known

• Solve for diffuse albedo

• Bi, Bj, Ei measured

• Form-factor Fij known

• Solve for diffuse albedo i

iB

jBijF

Source

Target

Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]Parametric BRDF Model [ Ward 92 ]

Isotropic Kernel

Anisotropic Kernel

NHi

r

),(

Ksd

2

22

4

]/tan[exp

coscos

1),(

ri

K

yx

yx

ri

K

4

)]/sin/cos(tan[exp

coscos

1),(

22222

( 3 parameters)

( 5 parameters)

Non-diffuse Surfaces underNon-diffuse Surfaces underDirect IlluminationDirect IlluminationNon-diffuse Surfaces underNon-diffuse Surfaces underDirect IlluminationDirect Illumination

2

,,)),(( min arg iisi

i

di IKIL

sd

2

,,)),(( min arg iisi

i

di IKIL

sd

NH

iisid

i IKIL )),((

iisid

i IKIL )),((

P1P2

P1

P2

Non-diffuse Surfaces under Non-diffuse Surfaces under Mutual IlluminationMutual IlluminationNon-diffuse Surfaces under Non-diffuse Surfaces under Mutual IlluminationMutual Illumination

• LPiAj is not known. ( unlike diffuse case, where LPiAj = LCkAj )

• LPiAj is not known. ( unlike diffuse case, where LPiAj = LCkAj )

j j

APCAPsAPAPdPC jivjijijiivKLFLL

j jAPCAPsAPAPdPC jivjijijiiv

KLFLL Cv

Ck

Aj

Pi

LPiAj

LCkAj

LCvPi

Source

Target

Solution: iteratively estimate Solution: iteratively estimate specular component.specular component.Solution: iteratively estimate Solution: iteratively estimate specular component.specular component.

jikjkji APCACAP SLL jikjkji APCACAP SLL

• Initialize

• Repeat– Estimate BRDF parameters for each surface

– Update and

• Initialize

• Repeat– Estimate BRDF parameters for each surface

– Update and

0jik APCS 0jik APCS

jik APCS jik APCS jiAPL

Estimation of Specular Difference SEstimation of Specular Difference SEstimation of Specular Difference SEstimation of Specular Difference S

• Estimate specular component of by Monte Carlo ray-tracing using current guess of reflectance parameters.

• Similarly for

• Difference gives S

• Estimate specular component of by Monte Carlo ray-tracing using current guess of reflectance parameters.

• Similarly for

• Difference gives S Cv

Ck

Aj

Pi

LPiAj

LCkAj

LCvPi

LPiAj

LCkAj

Recovering Diffuse Albedo MapsRecovering Diffuse Albedo MapsRecovering Diffuse Albedo MapsRecovering Diffuse Albedo Maps

• Specular properties assumed uniform across each surface, but diffuse albedo allowed to vary.

•

•

• Specular properties assumed uniform across each surface, but diffuse albedo allowed to vary.

•

•

)()()( xLxLxL sd

)(/)()( xIxLx dd

ResultsResultsResultsResults

• A simulated cubical room• A simulated cubical room

Results for the Simulated CaseResults for the Simulated CaseResults for the Simulated CaseResults for the Simulated Case

Diffuse Albedo

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 60

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6

Specular Roughness

ResultsResultsResultsResults

• A real conference room• A real conference room

Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting Real vs. Synthetic for Original Lighting

Real

Synthetic

Diffuse Albedo Maps of Identical Diffuse Albedo Maps of Identical Posters in Different PositionsPosters in Different PositionsDiffuse Albedo Maps of Identical Diffuse Albedo Maps of Identical Posters in Different PositionsPosters in Different Positions

Poster A Poster B Poster C

Inverting Color BleedInverting Color BleedInverting Color BleedInverting Color Bleed

Input Photograph Output Albedo Map

Real vs. Synthetic for Novel LightingReal vs. Synthetic for Novel LightingReal vs. Synthetic for Novel LightingReal vs. Synthetic for Novel Lighting

Real

Synthetic

VideoVideoVideoVideo

AcknowledgmentsAcknowledgmentsAcknowledgmentsAcknowledgments

• Thanks to David Culler and the Berkeley NOW project, Tal Garfinkel, Gregory Ward Larson, Carlo Sequin.

• Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship.

• Thanks to David Culler and the Berkeley NOW project, Tal Garfinkel, Gregory Ward Larson, Carlo Sequin.

• Supported by ONR BMDO, the California MICRO program, Philips Corporation, Interval Research Corporation and Microsoft Graduate Fellowship.

ConclusionsConclusionsConclusionsConclusions

• A digital camera can undertake all the data acquisition tasks involved.

• Both specular and high resolution diffuse reflectance properties can be recovered from photographs.

• Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints.

• A digital camera can undertake all the data acquisition tasks involved.

• Both specular and high resolution diffuse reflectance properties can be recovered from photographs.

• Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints.