15-463: Computational Photography donated by Paul Debevec...

75
Image-Based Lighting II 15-463: Computational Photography Alexei Efros, CMU, Fall 2006 …with a lot of slides donated by Paul Debevec © Clément Poline

Transcript of 15-463: Computational Photography donated by Paul Debevec...

Page 1: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Image-Based Lighting II

15-463: Computational PhotographyAlexei Efros, CMU, Fall 2006…with a lot of slides

donated by Paul Debevec

© Clément Poline

Page 2: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Illuminating a Small SceneIlluminating a Small Scene

Page 3: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 4: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

We can now illuminatesynthetic objects with real light.

How do we add synthetic objects to a real scene?

We can now illuminatesynthetic objects with real light.

How do we add synthetic objects to a real scene?

Page 5: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Real Scene ExampleReal Scene Example

Goal: place synthetic objects on tableGoal: place synthetic objects on table

Page 6: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Light Probe / Calibration GridLight Probe / Calibration Grid

Page 7: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

real scenereal scene

Modeling the SceneModeling the Scene

light-based modellight-based model

Page 8: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

The Light-Based Room ModelThe Light-Based Room Model

Page 9: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

real scenereal scene

Modeling the SceneModeling the Scene

synthetic objectssynthetic objects

light-based modellight-based model

local scenelocal scene

Page 10: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

The Lighting ComputationThe Lighting Computation

synthetic objects(known BRDF)

synthetic objects(known BRDF)

distant scene (light-based, unknown BRDF)distant scene (light-based, unknown BRDF)

local scene(estimated BRDF)

local scene(estimated BRDF)

Page 11: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Rendering into the SceneRendering into the Scene

Background PlateBackground Plate

Page 12: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Rendering into the SceneRendering into the Scene

Objects and Local Scene matched to SceneObjects and Local Scene matched to Scene

Page 13: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Differential RenderingDifferential Rendering

Local scene w/o objects, illuminated by modelLocal scene w/o objects, illuminated by model

Page 14: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Differential Rendering (2)Difference in local sceneDifferential Rendering (2)Difference in local scene

-- ==

Page 15: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Differential RenderingDifferential Rendering

Final ResultFinal Result

Page 16: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 17: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

IMAGE-BASED LIGHTING IN FIAT LUXPaul Debevec, Tim Hawkins, Westley Sarokin, H. P. Duiker, Christine Cheng, Tal Garfinkel, Jenny Huang

SIGGRAPH 99 Electronic Theater

Page 18: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 19: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

HDR Image SeriesHDR Image Series

2 sec2 sec 1/4 sec1/4 sec 1/30 sec1/30 sec

1/250 sec1/250 sec 1/2000 sec1/2000 sec 1/8000 sec1/8000 sec

Page 20: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Stp1 PanoramaStp1 Panorama

Page 21: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Assembled PanoramaAssembled Panorama

Page 22: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Light Probe ImagesLight Probe Images

Page 23: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Capturing a Spatially-Varying Lighting EnvironmentCapturing a Spatially-Varying Lighting Environment

Page 24: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

The MovieThe Movie

Page 25: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Simulating the Glare in the Human Eye

Simulating the Glare in the Human Eye

• Greg Spencer, Peter Shirley, Kurt Zimmerman, and Donald Greenberg. Physically-based glare effects for digital images. SIGGRAPH 95.

• Greg Spencer, Peter Shirley, Kurt Zimmerman, and Donald Greenberg. Physically-based glare effects for digital images. SIGGRAPH 95.

Page 26: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Scattering in the eye

What’s the scattering model?

Page 27: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

HDR Image

Page 28: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Gaussian Blur, LDR information Only

Page 29: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Gaussian Blur, Full HDR Information

Page 30: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Full HDR Disc Blur

Page 31: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Frame Postprocessing in Rendering with Natural LightFrame Postprocessing in Rendering with Natural Light

Page 32: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Real objects under new lighting

Page 33: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Rendering Light Probes as Light Sources

19991999

Page 34: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 35: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

A Lighting Reproduction Approach

Page 36: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Composited Results

Page 37: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Environment Map from Single Image?

Page 38: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Eye as Light Probe! (Nayar et al)

Page 39: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Cornea is an ellipsoid

Page 40: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Ellipsoid fitting

Page 41: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 42: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Putting it all together!

Page 43: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Reach for the sky

• How can we capture the whole sky as an environment map?

• What happens with the sun?

Page 44: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Direct HDR Capture of the Sun and Sky

Direct HDR Capture of the Sun and Sky

• Use Sigma 8mm fisheye lens and Canon EOS 1Ds to cover entire sky

• Use 3.0 ND filter on lens back to cover full range of light– Only 0.1% of light

gets through!

• Use Sigma 8mm fisheye lens and Canon EOS 1Ds to cover entire sky

• Use 3.0 ND filter on lens back to cover full range of light– Only 0.1% of light

gets through!Stumpfel, Jones, Wenger, Tchou, Hawkins, and Debevec. “Direct HDR Capture of the Sun and Sky”. To appear in Afrigraph 2004.Stumpfel, Jones, Wenger, Tchou, Hawkins, and Debevec. “Direct HDR Capture of the Sun and Sky”. To appear in Afrigraph 2004.

Page 45: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Extreme HDR Image SeriesExtreme HDR Image Series

1 secf/4

1 secf/4

1/4 secf/4

1/4 secf/4

1/30 secf/4

1/30 secf/4

1/8000 sec f/16

1/8000 sec f/16

1/30 secf/16

1/30 secf/16

1/250 secf/16

1/250 secf/16

1/1000 secf/16

1/1000 secf/16

Page 46: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

1 secf/4

1 secf/4

1/4 secf/4

1/4 secf/4

1/30 secf/4

1/30 secf/4

1/8000 sec f/16only image that does not

saturate!

1/8000 sec f/16only image that does not

saturate!

1/30 secf/16

1/30 secf/16

1/250 secf/16

1/250 secf/16

1/1000 secf/16

1/1000 secf/16

Extreme HDR Image Series- sun closeupExtreme HDR Image Series- sun closeup

Page 47: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Spectral Calibration – ND filters are NOT Necessarily Neutral!

Spectral Calibration – ND filters are NOT Necessarily Neutral!

Before correctionBefore correction

After correction based on MacBeth ColorChecker chart appearance

After correction based on MacBeth ColorChecker chart appearance

Page 48: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Two Complete days of HDR Lighting(see video)

Two Complete days of HDR Lighting(see video)

Feb 22, 2004 Feb 23, 2004

(day averages at 1 min. intervals)

Page 49: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Lit by sun and skyLit by sun and sky

Page 50: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

9 samples per pixel, 17 min.9 samples per pixel, 17 min.

Page 51: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

16 samples per pixel, 46 min.16 samples per pixel, 46 min.

Page 52: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

100 samples per pixel, 189 min.100 samples per pixel, 189 min.

Page 53: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

A sunlit sample pointA sunlit sample point

Page 54: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 55: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 56: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 57: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 58: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 59: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 60: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

A shadowed sample pointA shadowed sample point

Page 61: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 62: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 63: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 64: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 65: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 66: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 67: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

HDRI Sky ProbeHDRI Sky Probe

Page 68: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Clipped Sky + Sun SourceClipped Sky + Sun Source

Page 69: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Lit by sky only, 17 min.Lit by sky only, 17 min.

Page 70: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Lit by sun only, 21 min.Lit by sun only, 21 min.

Page 71: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

Lit by sun and sky, 25 min.Lit by sun and sky, 25 min.

Page 72: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 73: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

24 Samples per Pixel – 6h, 22min

Page 74: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.
Page 75: 15-463: Computational Photography donated by Paul Debevec …graphics.cs.cmu.edu/courses/15-463/2006_fall/www/... · 2007-01-14 · Paul Debevec, Tim Hawkins, Westley Sarokin, H.

24 Samples per Pixel – 6h, 22min