Accurate Accurate Image BasedImage Based

Relighting Relighting through through

OptimizationOptimization

Pieter PeersPhilip Dutré

Department of Computer ScienceK.U.Leuven, Belgium

What is Image Based Relighting?What is Image Based Relighting?

Basis ImagesBasis Images

Processing & Processing & RelightingRelighting

Relit ImageRelit Image

Previous WorkPrevious Work

• The Light Stage (Debevec et al. ’00)

Images: ACM SIGGRAPH ‘00

Previous WorkPrevious Work

• Environment Matting (Zonker et al. ’99, Chuang et al. ’00)

Images: ACM SIGGRAPH ’99

Previous workPrevious work

Diffuse Materials

Specular Materials

The LightStage

+ -

Environment Matting

- +

Sparse sampling of light source directionsSufficient angular sampling

Large span of incominglight directions

Bundled incominglight directions

• Matusik et al. EGWR’02

Three-step algorithmThree-step algorithm

Basis Image Basis Image RecordingRecording

Relit image = Captured reflectance + Light-mapRelit image = Captured reflectance + Light-map

1

3

2

Reflectance field Reflectance field calculationcalculation

Basis Image RecordingBasis Image Recording

Basis Image RecordingBasis Image Recording

...

Basis Image RecordingBasis Image Recording

HDR photographsHDR photographs

Practical SetupPractical Setup

• HDR photographs

• Monitor calibration

• 4x4 grid, 9 patterns per lightpatch

• ±600 basis images

Three-step algorithmThree-step algorithm

Basis Image Basis Image RecordingRecording

1

3

2

Relit image = Captured reflectance + Light-mapRelit image = Captured reflectance + Light-map

Reflectance field Reflectance field calculationcalculation

Support and reflection coefficientSupport and reflection coefficient

Support and reflectanceSupport and reflectance

…

PatternsPixel

× Rpix,patch

Reflection Coefficient

=

=

=

=

Resulting Value

……

→ Error1

→ Error2

→ Errorn-1

→ Errorn

Find and Rpix,patch that minimize: Errork

click

Support and reflectance: examplesSupport and reflectance: examples

top

left back right

Three-step algorithmThree-step algorithm

Basis Image Basis Image RecordingRecording

1

3

2

Relit image = Captured reflectance + Light-mapRelit image = Captured reflectance + Light-map

Reflectance field Reflectance field calculationcalculation

Re-lightingRe-lighting

For each pixel:

Pixel value = ∑ Filteri(LightMap) × Rpix,i

support i

ResultsResults

A

C

B

D

ResultsResults

A

C

B

D

ResultsResults

A

C

B

D

ResultsResults

A

C

B

D

ComparisonComparison

Stochastic ray tracing Relit from rendered basis images

ComparisonComparison

Stochastic ray tracing Relit from rendered basis images

ComparisonComparison

Stochastic ray tracing Relit from rendered basis images

ConclusionConclusion

• Accurate relighting:– Diffuse materials– Specular materials

• Combine strengths from:– The Light Stage– Environment matting

Future WorkFuture Work

• Alternative emitting devices

• Investigate other patterns & optimization algorithms

AcknowledgementsAcknowledgements

• Frank Suykens and Vincent Masselus

• Some environment maps from: http://www.debevec.org/Probes

Thank you…Thank you…

Questions?Questions?