1 Linear View Synthesis Using a Dimensionality Gap Light Field Prior Anat Levin and Fredo Durand...
-
date post
15-Jan-2016 -
Category
Documents
-
view
220 -
download
0
Transcript of 1 Linear View Synthesis Using a Dimensionality Gap Light Field Prior Anat Levin and Fredo Durand...
1
Linear View Synthesis Using a Dimensionality Gap Light Field Prior
Anat Levin and Fredo Durand
Weizmann Institute of Science & MIT CSAIL
22Light fields
Light field: the set of rays emitted from a scene in all possible directions
33Light fields
Novel view rendering
(Animation by Marc Levoy)
44Light fields
Novel view rendering
(Animation by Marc Levoy)
55Light fields
Novel view rendering Synthetic refocusing
(Animation by Marc Levoy)
66
u
v
4D light field
The set of light rays hitting the camera aperture plane is 4D:
• Ray hitting point- 2D
• Ray orientation- 2D
(In general: a 7D plenoptic space, including time and wavelength dimensions)
77Light field acquisition schemes and priors
Very different approaches to light field acquisition and manipulations exist in the literature.
The inherent difference between them is a different prior model on the light field space
88Light field acquisition schemes and priors
• 4D:
The light field is smooth, but involves 4 degrees of freedom
-Capture: 4D data (e.g. camera array)
-Inference: linear
99
• 4D:
-Capture: 4D data (e.g. camera array)
-Inference: linear
• 2D:
For Lambertian scenes all rays emerging from one point have same color. If depth is known, only 2 degrees of freedom
-Capture: 2D data (e.g. stereo camera)
-Inference: non linear depth estimation
Light field acquisition schemes and priors
1010In this talk: 3D light field prior
• 4D:
-Capture: 4D data (e.g. camera array
-Inference: linear
• 2D:-Capture: 2D data (e.g. stereo camera)
-Inference: non linear depth estimation
• 3D:
Depth is a 1D variable, hence the union of images at any depth covers no more than a 3D subset. Show that in the frequency domain there is only a 3D manifold of non zero entries.
-Capture: 3D data (e.g. focal stack)
-Inference: linear
uvy
x
1111Outline
• Linear view synthesis from a focal stack sequence
• The 3D light field prior
• Frequency derivation of synthesis algorithm
• Other applications of the 3D prior
1212Linear view synthesis with 3D prior
Output: Novel viewpoints (4D data)
2D Images x 2D set of novel viewpoints
Linear image processing
Input: Focal stack (3D data)
1D set of 2D images focused at different depth
1313Linear view synthesis algorithm
Average shifted images
Shift focal stack images by disparity
of desired view
Depth invariant deconvolution
1 2 3
No depth estimation!
1414
Shift invariant convolution~ focus sweep camera
Average shifted images
Ideal pinhole image
Depth invariant blur kernel
Inspiration: The focus sweep camera Hausler 72, Nagahara et al. 08
Captures a single image, average over all focus depths during exposure, provides
EDOF image from a single view
1515Linear view synthesis results
Video animation here
1616Disclaimers
• Novel viewpoints limited to the aperture area
• Convolution model breaks at occlusion boundaries
• Assume scene is Lambertian- in practice holds within the narrow range of angles of the aperture
1717Outline
• Linear view synthesis from a focal stack sequence
• The 3D light field prior
• Frequency derivation of synthesis algorithm
• Other applications of the 3D prior
18
y
xu
v
u
vy
x
• The set of light rays hitting the lens is 4D (x,y,u,v)
4D light field
(?,?,u0,0)
19
y
xu
v
u
v
y
x
• The set of light rays hitting the lens is 4D (x,y,u,v)
4D light field
(?,?,0,v0)
20
y
xu
v
u
v
y
x
• The set of light rays hitting the lens is 4D (x,y,u,v)
4D light field
21
y
xu
v
u
vy
x
• The set of light rays hitting the lens is 4D (x,y,u,v)
4D light field
22
uv
• The set of light rays hitting the lens is 4D
• Study the 4D Fourier domain L( , , , )x y u v(x,y,u,v)
4D light field spectrum
4D Fourier Transform
y
xu
v
y
x
4D Fourier Transform
23
uv
L( , , , )x y u v• The set of light rays hitting the lens is 4D
• Study the 4D Fourier domain
(x,y,u,v)
L( ,0,?,?)
uv
0x
y
xu
v
y
x
4D light field spectrum
24
y
xu
v uv
Frequency content only along 1D segments
4D Fourier Transform
y
x
4D light field spectrum
4D light field spectrum
Scene
4D Light field spectrum
Energy portion away from focal segments
26The slicing theorem
2D focused images at varying depths
y
xu
v
4D Fourier Transform
2D Fourier Transform
uvy
x
y
27The dimensionality gap
y
xu
v uvuv
4D Fourier Transformnear
far
depth color coding
uv
x
Light field spectrum: 4DImage spectrum: 2DDepth: 1D → Dimensionality gap
(Ng 05, Levin et al. 09)
Only the 3D manifold corresponding to physical focusing distance is useful
3D
28
uvGaussian prior: assigns non zero variance only to 3D set of entries on the focal segments
• Gaussian=> inference simple and linear
• Focal stack directly samples the manifold with non zero variance
y
x
3D Gaussian light field prior
29Outline
• Linear view synthesis from a focal stack sequence
• The 3D light field prior
• Frequency derivation of synthesis algorithm
• Other applications of the 3D prior
30
y
x
uv
View synthesis in the frequency domain4D spectrum of constant depth
scene
Spectra of focal stack images
1
Average focal stack spectra Spectra of
correct depthSample density
Deconvolution
(frequency domain)
31Outline
• Linear view synthesis from a focal stack sequence
• The 3D light field prior
• Frequency derivation of synthesis algorithm
• Other applications of the 3D prior
32Prior to infer light field from partial samples
In many other light field acquisition schemes we capture only a partial information on the light field- limited resolution, aliasing and each.
However, we capture linear measurements
On the other hand, we have a Gaussian prior, and we know the light field actually occupies only a low dimensional manifold of the 4D space.
Use the prior to “invert the rank deficient projection” and interpolate the measurements to get a light field with higher resolution, less aliasing.
33Improved viewpoints sample
4D Light field acquisition systems sample a 2D set of view points
• Can we do with sparser sample and 3D Gaussian prior for interpolation?
• How many samples needed? What is the right spacing?
• Shall we distribute samples on a grid? Better arrangement?
Grid: Standard sampling pattern
Circle: Sampling pattern with improved reconstruction
using 3D prior
34Superesolution of plenoptic camera
measurements
Plenoptic camera measurements are aliased
Replicas off the focal segments are high frequencies which we can re-bin and restore high frequency information
35Superesolution of plenoptic camera
measurements
Bicubic interpolation
Our result: applies for all depths simultaneously, no depth estimation
Lumsdaine and Georgiev: applies for a single known depth
36Summary
• Light field acquisition and synthesis strongly depends on light field prior
Existing priors:
• Linear view synthesis from the focal stack
• Other applications of 3D prior:
- viewpoints sample pattern
- depth invariant superesolution of plenoptic camera data
4D prior: capture- 4D data (e.g. camera array), inference- linear
2D prior: capture- 2D data (e.g. stereo), inference- non linear
Our new prior:
3D prior: capture- 3D data (e.g. focal stuck), inference linear