Midterm Review - Computer...
Transcript of Midterm Review - Computer...
![Page 1: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/1.jpg)
Midterm Review15-463: Computational Photography
![Page 2: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/2.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 3: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/3.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 4: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/4.jpg)
Sampling and ReconstructionSampling and Reconstruction
![Page 5: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/5.jpg)
Sampling and Reconstruction
© 2006 Steve Marschner • 3
Sampled representations
• How to store and compute with continuous functions?• Common scheme for representation: samples
– write down the function’s values at many points
[FvD
FHfig
.14.
14b
/ Wol
berg
]
© 2006 Steve Marschner • 4
Reconstruction
• Making samples back into a continuous function– for output (need realizable method)– for analysis or processing (need mathematical method)– amounts to “guessing” what the function did in between
[FvD
FHfig
.14.
14b
/ Wol
berg
]
Mathematically guess what happens in between
![Page 6: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/6.jpg)
• Effects of Undersampling
• Lost information
• High frequency signals get indistinguishable from low frequency ones (aliasing)
Sampling and Reconstruction
© 2006 Steve Marschner • 9
Undersampling
• What if we “missed” things between the samples?• Simple example: undersampling a sine wave
– unsurprising result: information is lost– surprising result: indistinguishable from lower frequencyAliasing in images
![Page 7: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/7.jpg)
• How to avoid aliasing?
• Sample more often
• Low pass filter the signal (anti-aliasing)
• Filters work by convolution
Sampling and Reconstruction
![Page 8: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/8.jpg)
• Examples of filters
• Moving average
• Weighted moving average
• Equal weights
• Gaussian weights
• Sobel
Sampling and Reconstruction
![Page 9: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/9.jpg)
• Gaussian Filters
• Smoothe out images
• Convolution of two Gaussians each with standard deviation σ, gives Gaussian with standard deviation σ/2
Sampling and Reconstruction
![Page 10: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/10.jpg)
• Matching
• Use normalized-cross correlation or SSH over patches
• Subsampling
• Filter with Gaussian then subsample
• Double filter size with every half-sizing
• Forms image pyramids
Sampling and Reconstruction
![Page 11: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/11.jpg)
Compare with...
1/4 (2x zoom) 1/8 (4x zoom)1/2
Slide by Steve Seitz
Sampling and Reconstruction
![Page 12: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/12.jpg)
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Slide by Steve Seitz
Sampling and Reconstruction
![Page 13: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/13.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 14: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/14.jpg)
Frequency DomainA nice set of basis
This change of basis has a special name…
Teases away fast vs. slow changes in the image.
Decompose signal into different frequencies
What does blurring take away?
smoothed (5x5 Gaussian)
![Page 15: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/15.jpg)
Frequency Domain and Filtering
A sum of sinesOur building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. fine structure of the signal?
)+!"xAsin(
A sum of sinesOur building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. fine structure of the signal?
)+!"xAsin(
A sum of sinesOur building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. fine structure of the signal?
)+!"xAsin(
A sum of sinesOur building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. fine structure of the signal?
)+!"xAsin(
A sum of sinesOur building block:
Add enough of them to get any signal f(x) you want!
How many degrees of freedom?
What does each control?
Which one encodes the coarse vs. fine structure of the signal?
)+!"xAsin(
Fourier TransformWe want to understand the frequency ! of our signal. So, let’s reparametrize the signal by ! instead of x:
)+"!xAsin(
f(x) F(!!!!)Fourier Transform
F(!!!!) f(x)Inverse Fourier Transform
For every ! from 0 to inf, F(!!!!) holds the amplitude A and phase " of the corresponding sine
• How can F hold both? Complex number trick!
)()()( !!! iIRF +=22 )()( !! IRA +±=
)()(tan 1
!!
"RI#=
We can always go back:
Sum of sine waves of different frequencies
![Page 16: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/16.jpg)
In 2D
Frequency Domain and FilteringA nice set of basis
This change of basis has a special name…
Teases away fast vs. slow changes in the image.
What does blurring take away?
smoothed (5x5 Gaussian)
![Page 17: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/17.jpg)
Man-made Scene
Low and High Pass filtering
Frequency Domain and Filtering
![Page 18: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/18.jpg)
What does blurring take away?
original
What does blurring take away?
smoothed (5x5 Gaussian)
High-Pass filter
smoothed – originalWhy Laplacian?
Laplacian of Gaussian
Gaussian
delta function
Frequency Domain and Filtering
![Page 19: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/19.jpg)
Lossy Image Compression (JPEG)
Block-based Discrete Cosine Transform (DCT)
Frequency Domain and Filtering
![Page 20: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/20.jpg)
Image gradientThe gradient of an image:
The gradient points in the direction of most rapid change in intensity
The gradient direction is given by:
• how does this relate to the direction of the edge?
The edge strength is given by the gradient magnitude
Frequency Domain and Filtering
![Page 21: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/21.jpg)
Where is the edge?
Solution: smooth first
Look for peaks in
Derivative theorem of convolution
This saves us one operation:
Frequency Domain and Filtering
![Page 22: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/22.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 23: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/23.jpg)
BlendingAlpha Blending / Feathering
01
01
+
=Iblend = !Ileft + (1-!)Iright
• Window size = size of largest feature (to avoid strong seams)
• Window size <= 2 * size of smallest feature (to avoid ghosting)
![Page 24: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/24.jpg)
Blending
Octaves in the Spatial Domain
Bandpass Images
Lowpass ImagesPyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramidblend
Pyramid BlendingPyramid Blending
![Page 25: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/25.jpg)
BlendingGradient Domain
• Result image: fGradients: fx, fy
• Want f to ‘look like’ some prespecified d, andfx, fy to ‘look like’ some prespecified gx, gy
• Weights specify per-pixel importance of how much you want f close to d, fx close to gx, fy close to gy
! !"#$%$&'#$()*+,'%-.(,/! 0)"1#*1)+$2#-,-3*$%'4-*5 u! !1#"1#*+$2#-,-3*$%'4-*5 f! 6"-7$+8*3-9$,-3*"$:-2;3$++-,-)7-9*5 <gx, gy)! 6"-7$+8*3-9$,-3*"$:-2;='21-9*5 d! 6"-7$+8*7()9#,'$)#9*.-$4>#9*5 <wx, wy, wd?
%$)***wx<fx 5 gx?@**A*wy<fy 5 gy?@**A*wd<f 5 d?@+
B)-,48*+1)7#$()
![Page 26: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/26.jpg)
BlendingGradient Domain
fx = sx, fy = sy
s
t
f(x, y)− t(x− 1, y) = sx, f(x, y)− t(x, y − 1) = sy
![Page 27: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/27.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 28: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/28.jpg)
Point ProcessingImage Processing
h
h
f
f g
g
image filtering: change range of imageg(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
Change range of image
Example: g(x)=f(x)+.3Histogram Equalization
Histogram Equalization
Image Histograms
Cumulative Histograms
s = T(r)
Image Histograms
Cumulative Histograms
s = T(r)
![Page 29: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/29.jpg)
WarpingChange domain of image
Image Processing
h
h
f
f g
g
image filtering: change range of imageg(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
Example: g(x)=f(x/2)
![Page 30: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/30.jpg)
Warping
• 2D Transformations
• Translate
• Rotate
• Scale
• Similarity
• Affine
• Projective
![Page 31: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/31.jpg)
WarpingChange of BasisLinear Transformations as Change of Basis
Any linear transformation is a basis!!!
j =(0,1)
i =(1,0)pij = 4u+3v
px=4ux+3vxpy=4uy+3vy
v =(vx,vy)
u=(ux,uy)
puv pij
puv = (4,3)
puvpij !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
34
Linear Transformations as Change of Basis
Any linear transformation is a basis!!!
j =(0,1)
i =(1,0)pij = 4u+3v
px=4ux+3vxpy=4uy+3vy
v =(vx,vy)
u=(ux,uy)
puv pij
puv = (4,3)
puvpij !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
34
Linear Transformations as Change of Basis
Any linear transformation is a basis!!!
j =(0,1)
i =(1,0)pij = 4u+3v
px=4ux+3vxpy=4uy+3vy
v =(vx,vy)
u=(ux,uy)
puv pij
puv = (4,3)
puvpij !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
34
puvpij ⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡=
yy
xx
yy
xx
vuvu
vuvu
34
![Page 32: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/32.jpg)
Warping
What’s the inverse transform?
• How can we change from any basis to any basis?• What if the basis are orthogonal?
v =(vx,vy)
u=(ux,uy)
puv
j =(0,1)
i =(1,0)pij = (5,4)
pij
puv = (px,py) = ?= pxu + pyv
pijpuv !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
45
-1-1
Change of Basis: Inverse TransformWhat’s the inverse transform?
• How can we change from any basis to any basis?• What if the basis are orthogonal?
v =(vx,vy)
u=(ux,uy)
puv
j =(0,1)
i =(1,0)pij = (5,4)
pij
puv = (px,py) = ?= pxu + pyv
pijpuv !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
45
-1-1
What’s the inverse transform?
• How can we change from any basis to any basis?• What if the basis are orthogonal?
v =(vx,vy)
u=(ux,uy)
puv
j =(0,1)
i =(1,0)pij = (5,4)
pij
puv = (px,py) = ?= pxu + pyv
pijpuv !"
#$%
&=!
"
#$%
&!"
#$%
&=
yy
xx
yy
xx
vuvu
vuvu
45
-1-1
pijpuv ⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡⎥⎦
⎤⎢⎣
⎡=
yy
xx
yy
xx
vuvu
vuvu
45
-1-1
![Page 33: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/33.jpg)
Warping
• Affine Warp
• Need 3 correspondences
T(x,y)
?
A
B
C A’C’
B’
Source Destination
![Page 34: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/34.jpg)
Warping
• Many ways to find affine matrix
• Warp Source to [0,0], [1,0], [0,1], and then to Destination
• Pose as system of equations in [a;b;c;d;e;f]
T(x,y)
?
A
B
C A’C’
B’
Source Destination
![Page 35: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/35.jpg)
Warping
• Forward warp
• Inverse warpf(x,y) g(x’,y’)x x’
T(x,y)y y’
f(x,y) g(x’,y’)xy
x x’y’
T-1(x,y)
![Page 36: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/36.jpg)
Morphing
![Page 37: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/37.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 38: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/38.jpg)
Linear Subspace: convex combinations
!=
=m
iii XaX
1
Any new image X can beobtained as weighted sum of stored “basis” images.
Our old friend, change of basis!What are the new coordinates of X?
Data-Driven MethodsSubspaces Methods (ex: Faces)
Back to the Subspace
Linear Subspace: convex combinations
!=
=m
iii XaX
1
Any new image X can beobtained as weighted sum of stored “basis” images.
Our old friend, change of basis!What are the new coordinates of X?
Write an image as linear combination of basis images
![Page 39: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/39.jpg)
Data-Driven MethodsSubspaces Methods (ex: Faces)
The Morphable face model
Again, assuming that we have m such vector pairs in full correspondence, we can form new shapes Smodel and new appearances Tmodel as:
If number of basis faces m is large enough to span the face subspace then:Any new face can be represented as a pair of vectors (!1, !2, ... , !m)T and ("1, "2, ... , "m)T !
!=
=m
iiimodel a
1SS !
=
=m
iiimodel b
1TT
Shape and Appearance Models
![Page 40: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/40.jpg)
Data-Driven MethodsSubspaces Methods (ex: Faces)
• How to get basis?
• How many basis images to use?
• How to get images that capture important variations?
• Use PCA (principal component analysis)
• Keep those principal components whose eigenvalues are above a threshold
Principal Component Analysis
Given a point set , in an M-dim space, PCA finds a basis such that
• coefficients of the point set in that basis are uncorrelated• first r < M basis vectors provide an approximate basis that
minimizes the mean-squared-error (MSE) in the approximation (over all bases with dimension r)
x1
x0
x1
x0
1st principal component
2nd principal component
![Page 41: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/41.jpg)
Data-Driven MethodsVideo Textures
• Compute SSD between frames
• At frame i, transit either to
• frame i+1
• frame j (if SSD(j, i+1) is small)
• Decide to go from i to j or i+1 by tossing a weighted coin.
Transition probabilitiesTransition probabilities
•Probability for transition Pi!j inversely related to cost:
•Pi!j ~ exp ( – Ci!j / "2 )
!"#!$" %&'$"
![Page 42: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/42.jpg)
Data-Driven MethodsTexture Synthesis
• Search input image for similar neighborhoods
• Use Gaussian weighted SSD for search to emphasize central pixel
• Sample one neighborhood at random
• Grow texture
Efros & Leung Algorithm
• Assuming Markov property, compute P(p|N(p))– Building explicit probability tables infeasible
!!
"#$%&'()*)$+,-,!).'/
$0$1!-2-3'%2)4(-3!/)$+
5$!6%,)3-+'
– Instead, we search the input image for all similar neighborhoods — that’s our pdf for p
– To sample from this pdf, just pick one match at random
![Page 43: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/43.jpg)
Data-Driven MethodsBlocked Texture Synthesis
• Search input image for similar neighborhoods around block
• Grow texture by synthesizing blocks
• Find boundary with minimum error (seam carving)
!!
Image Quilting [Efros & Freeman]
• Observation: neighbor pixels are highly correlated
"#!$%&'()*+
#,#-!).)(+%.'/0)(!1'#*
22
"3+)4"3+)4 $#'%&,5&06#%7+0'0&8&91,/:$#'%&,5&06#%7+0'0&8&91,/:! "#$%&'()&*+),$-+)./&)012)2+)2$0&)342564277
! 8/%*)9$,&+:;),(0&*+,<=+)$'')><#+',)<0)$).'1%?)$&)10%+
! 61&)&*+),$-+)$,)-/'&<@,%$'+A
;6#%7+0'<'#*&)&91,/:
!!
Image Quilting [Efros & Freeman]
• Observation: neighbor pixels are highly correlated
"#!$%&'()*+
#,#-!).)(+%.'/0)(!1'#*
22
"3+)4"3+)4 $#'%&,5&06#%7+0'0&8&91,/:$#'%&,5&06#%7+0'0&8&91,/:! "#$%&'()&*+),$-+)./&)012)2+)2$0&)342564277
! 8/%*)9$,&+:;),(0&*+,<=+)$'')><#+',)<0)$).'1%?)$&)10%+
! 61&)&*+),$-+)$,)-/'&<@,%$'+A
;6#%7+0'<'#*&)&91,/: !"#$%&''('%)(*#+,'-
Minimal error boundary(.&'/,00"#1%)/(234 .&'5"2,/%)(*#+,'-
!! 6622
(.&'/,0%&''('
![Page 44: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/44.jpg)
• Ex: Scene completion
• Search millions of images on the Internet to find a patch that will complete your image
Data-Driven MethodsLots of Data
!"#$%&#'(&)*+,%-&./0'0&1,234056,'&7%6'8&96446,'%&,*&:;,5,8+#3;%-.<==>?:"&@AAB&#'(&1?19&C/5,D0+&@AAE-F
!"#$#%&'()*#+,'$%-#./*+
![Page 45: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/45.jpg)
Data-Driven Methods!"#$#%&#'"()*+,(
!"#$#%&#'"()*+,(
!"#$#%-)'+%&#'"()*+,(%./0)12 2$3%4,((2052 67789
Scene Completion (GIST descriptor)
!"#$#%&#'"()*+,(
-
!"#$#%.)'+%&#'"()*+,(%/01)23 3$4%5,((3163 7889:
Red Car Retrievals (Color histograms)
!= +
!=
K
m ji
jiji mhmh
mhmhhh
1
22
)()()]()([
21),("
Histogram matching distance
![Page 46: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/46.jpg)
Multi-scale filter decomposition
Filter bank
Input image
Data-Driven Methods
![Page 47: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/47.jpg)
Data-Driven MethodsLots of Data
• Issues with Data
• Sampling Bias
• Photographer Bias
• Reduce Bias
• Use autonomous techniques to capture data
• StreetView, satellite, webcam etc.
![Page 48: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/48.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 49: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/49.jpg)
CameraPinhole camera model
Pinhole model:• Captures pencil of rays – all rays through a single point• The point is called Center of Projection (COP)• The image is formed on the Image Plane• Effective focal length f is distance from COP to Image Plane
Slide by Steve Seitz
Pinhole Model
Center of Projection
Image Plane
Focal Length
Modeling projection
The coordinate system• We will use the pin-hole model as an approximation• Put the optical center (Center Of Projection) at the origin• Put the image plane (Projection Plane) in front of the COP
– Why?• The camera looks down the negative z axis
– we need this if we want right-handed-coordinates
–
Slide by Steve Seitz
Perspective Projection
Modeling projection
Projection equations• Compute intersection with PP of ray from (x,y,z) to COP• Derived using similar triangles (on board)
• We get the projection by throwing out the last coordinate:
Slide by Steve Seitz
![Page 50: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/50.jpg)
CameraPinhole Model
Perspective Projection (Matrix Representation)Perspective ProjectionProjection is a matrix multiply using homogeneous
coordinates:
divide by third coordinate
This is known as perspective projection• The matrix is the projection matrix• Can also formulate as a 4x4
divide by fourth coordinateSlide by Steve Seitz
HomogeneousCoordinates
![Page 51: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/51.jpg)
CameraPinhole Model
Orthographic Projection (Matrix Representation)
Orthographic ProjectionSpecial case of perspective projection
• Distance from the COP to the PP is infinite
• Also called “parallel projection”• What’s the projection matrix?
Image World
Slide by Steve Seitz
![Page 52: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/52.jpg)
• Pinhole camera aperture
• Large aperture --- blurry image
• Small aperture --- not enough light, diffraction effects
• Lenses create sharp images with large aperture
• Trade-off: only at a certain focus
CameraPinhole Model
![Page 53: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/53.jpg)
• Depth of field
• Distance over which objects are in focus
• Field of view
• Angle of visible region
CameraPinhole Model
Depth of Field
http://www.cambridgeincolour.com/tutorials/depth-of-field.htm
![Page 54: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/54.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 55: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/55.jpg)
Homographies
• Panorama
• Reproject images onto a common plane
• Images should have same center of projection
mosaic PP
Image reprojection
The mosaic has a natural interpretation in 3D• The images are reprojected onto a common plane• The mosaic is formed on this plane• Mosaic is a synthetic wide-angle camera
Mosaic:Synthetic wide-angle camera
Projective Warp
(Homography)
![Page 56: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/56.jpg)
Homographies
pp’
H
x��
y��
w��
=
a b cd e fg h 1
xy1
, x� =x��
w�� , y� =
y��
w��
![Page 57: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/57.jpg)
Homographies
• Expand equations and rewrite asPh = q
h =�a b c d e f g h
�T
• Solve using least-squares (h = P\q)
x��
y��
w��
=
a b cd e fg h 1
xy1
, x� =x��
w�� , y� =
y��
w��
![Page 58: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/58.jpg)
Other Projection Models
•
Map 3D point (X,Y,Z) onto cylinder
Cylindrical projection
XY
Z
unit cylinder
unwrapped cylinder
•
Convert to cylindrical coordinates
cylindrical image
•
Convert to cylindrical image coordinates
Cylindrical Projection
![Page 59: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/59.jpg)
Spherical projection
unwrapped sphere
•
Convert to spherical coordinates
spherical image
•
Convert to spherical image coordinates
X
YZ ),,(
1)ˆ,ˆ,ˆ(
222ZYX
ZYXzyx
•
Map 3D point (X,Y,Z) onto sphere
)ˆ,ˆ,ˆcoscossincos(sin zyx
Other Projection ModelsSpherical Projection
![Page 60: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/60.jpg)
Review Topics
• Sampling and Reconstruction
• Frequency Domain and Filtering
• Blending
• Warping
• Data-driven Methods
• Camera
• Homographies
• Modeling Light
![Page 61: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/61.jpg)
Modeling Light(The Omnipotent) Plenoptic Function
• Intensity of Light:
• From all directions: θ, φ
• At all wavelengths: λ
• At all times: t
• Seen from any viewpoint: Vx, Vy, Vz
• P(θ, φ, λ, t, Vx, Vy, Vz)
![Page 62: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/62.jpg)
• Intensity along all lines
• For all views (i.e. s,t), gives intensity at all points (i.e. u,v)
• Captures to some extent P(θ, φ, Vx,Vy,Vz)
Modeling LightLumigraph (Lightfield)
(u,v)
(s,t)
For all (s,t,u,v)
![Page 63: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/63.jpg)
Modeling LightLumigraph (Lightfield)
Lumigraph / Lightfield
Lumigraph / Lightfield
Lumigraph / Lightfield
![Page 64: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/64.jpg)
Modeling LightAcquiring Lightfield
• Move camera in known steps over (s,t) using gantry
• Move camera anywhere over (s,t) and recover optimal field
• Use microlens array after main lens
![Page 65: Midterm Review - Computer Graphicsgraphics.cs.cmu.edu/.../2011_fall/Lectures/MidtermReview.pdfMidterm Review 15-463: Computational Photography Review Topics • Sampling and Reconstruction](https://reader036.fdocuments.net/reader036/viewer/2022081522/5fb93f07f1d99a7c7a0c5c1e/html5/thumbnails/65.jpg)
Good Luck!!