Multi Aperture Photography
-
Upload
makrofajj -
Category
Art & Photos
-
view
1.591 -
download
0
description
Transcript of Multi Aperture Photography
![Page 1: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/1.jpg)
Multi-Aperture Photography
Paul Green – MIT CSAILWenyang Sun – MERLWojciech Matusik – MERLFrédo Durand – MIT CSAIL
![Page 2: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/2.jpg)
Motivation
http://photographertips.net
Portrait
Landscape
Small Aperture
Large Aperture
Depth of Field Control
Shallow Depth of Field
Large Depth of Field
![Page 3: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/3.jpg)
plane of focus
Depth and Defocus Blur
sensor lens
defocus blur depends on distance from plane of focus
subject
rays from point in focus converge to single pixel
circle of confusio
n
![Page 4: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/4.jpg)
Defocus Blur & Aperture
lens plane of focus
defocus blur depends on aperture size
aperture
http://photographertips.net
sensor
subject
circle of confusio
n
![Page 5: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/5.jpg)
Goals
Aperture size is a critical parameter for photographers
■ post-exposure depth of field control
■ extrapolate shallow depth of field beyond physical aperture
![Page 6: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/6.jpg)
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ (Limited) refocusing
![Page 7: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/7.jpg)
Related WorkComputational Cameras
■ Plenoptic Cameras■ Adelson and Wang ‘92■ Ng et al ‘05■ Georgiev et al ‘06
■ Split-Aperture Camera■ Aggarwal and Ahuja ‘04
■ Optical Splitting Trees■ McGuire et al ‘07
■ Coded Aperture■ Levin et al ’07■ Veeraraghavan et al ’07
■ Wavefront Coding■ Dowski and Cathey ‘95
Depth from Defocus■ Pentland ‘87
Georgiev et al‘06
Aggarwal and Ahuja ‘04McGuire et al ‘07
Adelson and Wang ‘92
Levin et al ’07 Veeraraghavan et al ’07
![Page 8: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/8.jpg)
Plenoptic Cameras
Capture 4D LightField■ 2D Spatial (x,y)■ 2D Angular (u,v
Aperture)
Trade resolution for flexibility after capture■ Refocusing■ Depth of field control■ Improved Noise
Characteristics
Lens Aperture
u
v
Sensor (x,y)
Lenslet Array
Subject
Lens (u,v)
![Page 9: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/9.jpg)
1D vs 2D Aperture Sampling
u
v
Aperture
2D Grid Sampling http://photographertips.net
![Page 10: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/10.jpg)
4 Samples4 Samples
u
v
Aperture
2D Grid Sampling
1D vs. 2D Aperture Sampling
Aperture
1D “Ring” Sampling
45 Samples45 Samples
http://photographertips.net
![Page 11: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/11.jpg)
Optical Splitting Trees
General framework for sampling imaging parameters■ Beamsplitters ■ Multiple cameras
Large ApertureCamera
Small ApertureCamera
McGuire et al ‘07
Beamsplitter
Incoming light
![Page 12: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/12.jpg)
Goals
■ post-exposure depth of field control
■ extrapolate shallow depth of field
■ (limited) refocusing
■ 1d sampling■ no beamsplitters■ single sensor■ removable
![Page 13: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/13.jpg)
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ Refocusing
![Page 14: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/14.jpg)
Optical Design Principles
Aperture
3D sampling■ 2D spatial■ 1D aperture
size■ 1 image for
each “ring”
Sensor
http://photographertips.net
![Page 15: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/15.jpg)
Goal: Split aperture into 4 separate optical paths■ concentric tilted mirrors■ at aperture plane
Aperture Splitting
Tilted Mirrors
![Page 16: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/16.jpg)
Aperture Splitting
Incoming light
Sensor
MirrorsFocusing lenses
Tilted Mirrors
![Page 17: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/17.jpg)
Aperture Splitting
Photographic Lens
Aperture Plane
Relay system
Aperture splitting optics
New Aperture Plane
X
Ideally at aperture plane
, but not physically possible!Solution: Relay Optics to create virtual
aperture plane
![Page 18: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/18.jpg)
Optical Prototype
Mirror Close-up
main lens
relay optics
mirrors
tilted mirrors
lenses
SLR Camera
![Page 19: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/19.jpg)
Sample Data
Raw data from our camera
![Page 20: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/20.jpg)
Ideally would be ringsGaps are from
occlusion
Point Spread Function Occlusion
combinedinner ring 1 ring 2 outer
![Page 21: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/21.jpg)
Outline
Multi-Aperture Camera■ New camera design■ Capture multiple aperture
settings simultaneously
Applications■ Depth of field control■ Depth of field extrapolation■ Refocusing
![Page 22: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/22.jpg)
DOF Navigation
0I 2I
1I 3I
![Page 23: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/23.jpg)
Approximate defocus blur as convolution
DOF Extrapolation?
0I 1I 2I 3I
?EI
)(0 nn KII
)( nK - Circular aperture blurring kernel
Depends on depth and aperture size
What is at each pixel in ?
EEI
![Page 24: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/24.jpg)
Blu
r siz
e
Aperture Diameter
Largest physical aperture
DOF Extrapolation Roadmap
capture
estimate blur fit
model
extrapolate blur
IIEE
I1I2
I0
I3
![Page 25: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/25.jpg)
Blu
r siz
e
Aperture Diameter D
I1 I2
IE
I0
σ
I3
Largest physical aperture
Defocus Gradient
Defocus blur
o
sos
fd
fddfdG
)(
o
sos
fd
fddfd )(DG
odsd
σ D
G is slope of this line
Defocus Gradient Map
Defocus Gradient
focal length
aperture diameter
sensor distance
object distance
Blur proportional to aperture diameter
![Page 26: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/26.jpg)
Optimization
solve for discrete defocus gradient values G at each pixel
Data term
Graph Cuts with spatial regularization term
i
Ni iGKIIGD )()( 1
0
Defocus Gradient Map
Smallest Aperture Image
![Page 27: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/27.jpg)
Depth of Field Extrapolation
![Page 28: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/28.jpg)
Synthetic Refocusing
Modify gradient labels and re-synthesize image
gradient map
“refocused” map
extrapolated f/1.8
“refocused” synthetic f/1.8
![Page 29: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/29.jpg)
Synthetic Refocusing Video
![Page 30: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/30.jpg)
Depth Guided DeconvolutionDeconvolve (deblur) with kernel given
by defocus gradient map
Before After depth-guided deconvolution
Defocus gradient mapSmallest aperture image
![Page 31: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/31.jpg)
Discussion■ Occlusion
■ Could help depth discrimination (coded aperture)
■ Difficult alignment process■ Mostly because prototype
■ Refocusing limited by Depth of Field■ helped by depth-guided deconvolution
■ Texture required for accurate defocus gradient map■ Not critical for depth of field and refocus
![Page 32: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/32.jpg)
Summary■Multi-aperture camera■1D sampling of aperture■Removable
■Post-Exposure depth of field control
■Depth of field extrapolation■ Limited refocusing■Depth-guided deconvolution
![Page 33: Multi Aperture Photography](https://reader033.fdocuments.net/reader033/viewer/2022061509/55583693d8b42acb078b4891/html5/thumbnails/33.jpg)
ThanksPeople
■ John Barnwell■ Jonathan
Westhues■ SeBaek Oh■ Daniel Vlasic■ Eugene Hsu■ Tom Mertens■ Britton Bradley■ Jane Malcolm■ MIT Graphics
Group
Funding■ NSF CAREER award
0447561■ Ford Foundation
predoctoral Fellowship■ Microsoft Research New
Faculty Fellowship■ Sloan Fellowship