Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.
-
date post
20-Dec-2015 -
Category
Documents
-
view
225 -
download
4
Transcript of Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.
![Page 1: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/1.jpg)
Motion estimation
Introduction to Computer VisionCS223B, Winter 2005
Richard Szeliski
![Page 2: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/2.jpg)
2/1/2005 Motion estimation 2
Why Visual Motion?
Visual Motion can be annoying• Camera instabilities, jitter• Measure it. Remove it.
Visual Motion indicates dynamics in the scene• Moving objects, behavior• Track objects and analyze trajectories
Visual Motion reveals spatial layout of the scene• Motion parallax
![Page 3: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/3.jpg)
2/1/2005 Motion estimation 3
Today’s lecture
Motion estimation• background: image pyramids, image warping• application: image morphing• parametric motion (review)• optic flow• layered motion models
![Page 4: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/4.jpg)
Image Pyramids
![Page 5: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/5.jpg)
2/1/2005 Motion estimation 5
Image Pyramids
![Page 6: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/6.jpg)
2/1/2005 Motion estimation 6
Pyramid Creation
“Laplacian” Pyramid• Created from Gaussian
pyramid by subtractionLl = Gl – expand(Gl+1)
filter mask
“Gaussian” Pyramid
![Page 7: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/7.jpg)
2/1/2005 Motion estimation 7
Octaves in the Spatial Domain
Bandpass Images
Lowpass Images
![Page 8: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/8.jpg)
2/1/2005 Motion estimation 8
Pyramids
Advantages of pyramids• Faster than Fourier transform• Avoids “ringing” artifacts
Many applications• small images faster to process• good for multiresolution processing• compression• progressive transmission
Known as “mip-maps” in graphics communityPrecursor to wavelets
• Wavelets also have these advantages
![Page 9: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/9.jpg)
2/1/2005 Motion estimation 9
Laplacianlevel
4
Laplacianlevel
2
Laplacianlevel
0
left pyramid right pyramid blended pyramid
![Page 10: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/10.jpg)
2/1/2005 Motion estimation 10
Pyramid Blending
![Page 11: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/11.jpg)
Image Warping
![Page 12: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/12.jpg)
2/1/2005 Motion estimation 14
Parametric (global) warping
Examples of parametric warps:
translation rotation aspect
affineperspective
cylindrical
![Page 13: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/13.jpg)
2/1/2005 Motion estimation 16
Image Warping
Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
f(x) g(x’)x x’
h(x)
![Page 14: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/14.jpg)
2/1/2005 Motion estimation 17
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
![Page 15: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/15.jpg)
2/1/2005 Motion estimation 18
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?• Answer: add “contribution” to several pixels,
normalize later (splatting)
![Page 16: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/16.jpg)
2/1/2005 Motion estimation 19
Inverse Warping
Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)
f(x) g(x’)x x’
h-1(x’)
• What if pixel comes from “between” two pixels?
![Page 17: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/17.jpg)
2/1/2005 Motion estimation 20
Inverse Warping
Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)
• What if pixel comes from “between” two pixels?• Answer: resample color value from
interpolated (prefiltered) source image
f(x) g(x’)x x’
![Page 18: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/18.jpg)
2/1/2005 Motion estimation 21
Interpolation
Possible interpolation filters:• nearest neighbor• bilinear• bicubic (interpolating)• sinc / FIR
Needed to prevent “jaggies” and “texture crawl” (see demo)
![Page 19: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/19.jpg)
2/1/2005 Motion estimation 22
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
MIP-mapping [Williams’83]:1. build pyramid (but what decimation filter?):
– block averaging– Burt & Adelson (5-tap binomial)– 7-tap wavelet-based filter (better)
2. trilinear interpolation– bilinear within each 2 adjacent levels– linear blend between levels (determined by pixel size)
![Page 20: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/20.jpg)
2/1/2005 Motion estimation 23
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
Other possibilities:• summed area tables• elliptically weighted Gaussians (EWA)
[Heckbert’86]
![Page 21: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/21.jpg)
2/1/2005 Motion estimation 24
Image Warping – non-parametric
Specify more detailed warp function
Examples: • splines• triangles• optical flow (per-pixel motion)
![Page 22: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/22.jpg)
2/1/2005 Motion estimation 25
Image Warping – non-parametric
Move control points to specify spline warp
![Page 23: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/23.jpg)
Image Morphing
![Page 24: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/24.jpg)
2/1/2005 Motion estimation 27
Image Morphing
How can we in-between two images?1. Cross-dissolve
(all examples from [Gomes et al.’99])
![Page 25: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/25.jpg)
2/1/2005 Motion estimation 28
Image Morphing
How can we in-between two images?2. Warp then cross-dissolve = morph
![Page 26: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/26.jpg)
2/1/2005 Motion estimation 29
Warp specification
How can we specify the warp?1. Specify corresponding points
• interpolate to a complete warping function
• Nielson, Scattered Data Modeling, IEEE CG&A’93]
![Page 27: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/27.jpg)
2/1/2005 Motion estimation 30
Warp specification
How can we specify the warp?2. Specify corresponding vectors
• interpolate to a complete warping function
![Page 28: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/28.jpg)
2/1/2005 Motion estimation 31
Warp specification
How can we specify the warp?2. Specify corresponding vectors
• interpolate [Beier & Neely, SIGGRAPH’92]
![Page 29: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/29.jpg)
2/1/2005 Motion estimation 32
Warp specification
How can we specify the warp?3. Specify corresponding spline control points
• interpolate to a complete warping function
![Page 30: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/30.jpg)
2/1/2005 Motion estimation 33
Final Morph Result
![Page 31: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/31.jpg)
Motion estimation
![Page 32: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/32.jpg)
2/1/2005 Motion estimation 35
Classes of Techniques
Feature-based methods• Extract salient visual features (corners, textured areas) and track
them over multiple frames• Analyze the global pattern of motion vectors of these features• Sparse motion fields, but possibly robust tracking• Suitable especially when image motion is large (10-s of pixels)
Direct-methods• Directly recover image motion from spatio-temporal image
brightness variations• Global motion parameters directly recovered without an
intermediate feature motion calculation• Dense motion fields, but more sensitive to appearance variations• Suitable for video and when image motion is small (< 10 pixels)
![Page 33: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/33.jpg)
2/1/2005 Motion estimation 36
Brightness Constancy Equation:
The Brightness Constraint
),(),( ),(),( yxyx vyuxIyxJ
Or, better still, Minimize :2)),(),((),( vyuxIyxJvuE
),(),(),(),(),(),( yxvyxIyxuyxIyxIyxJ yx Linearizing (assuming small (u,v)):
![Page 34: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/34.jpg)
2/1/2005 Motion estimation 38
Local Patch Analysis
![Page 35: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/35.jpg)
2/1/2005 Motion estimation 39
Patch Translation [Lucas-Kanade]
yx
tyx IvyxIuyxIvuE,
2),(),(),(
Minimizing
Assume a single velocity for all pixels within an image patch
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
tT IIUII
LHS: sum of the 2x2 outer product tensor of the gradient vector
![Page 36: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/36.jpg)
2/1/2005 Motion estimation 40
The Aperture Problem
TIIMLet
• Algorithm: At each pixel compute by solving
• M is singular if all gradient vectors point in the same direction• e.g., along an edge• of course, trivially singular if the summation is over a single pixel or there is no texture• i.e., only normal flow is available (aperture problem)
• Corners and textured areas are OK
and
ty
tx
II
IIb
U bMU
![Page 37: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/37.jpg)
2/1/2005 Motion estimation 41
Aperture Problem and Normal Flow
![Page 38: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/38.jpg)
2/1/2005 Motion estimation 42
Local Patch Analysis
![Page 39: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/39.jpg)
2/1/2005 Motion estimation 43
Iterative Refinement
Estimate velocity at each pixel using one iteration of Lucas and Kanade estimation
Warp one image toward the other using the estimated flow field(easier said than done)
Refine estimate by repeating the process
![Page 40: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/40.jpg)
2/1/2005 Motion estimation 44
Optical Flow: Iterative Estimation
xx0
Initial guess:
Estimate:
estimate update
![Page 41: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/41.jpg)
2/1/2005 Motion estimation 45
Optical Flow: Iterative Estimation
xx0
estimate update
Initial guess:
Estimate:
![Page 42: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/42.jpg)
2/1/2005 Motion estimation 46
Optical Flow: Iterative Estimation
xx0
Initial guess:
Estimate:
Initial guess:
Estimate:
estimate update
![Page 43: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/43.jpg)
2/1/2005 Motion estimation 47
Optical Flow: Iterative Estimation
xx0
![Page 44: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/44.jpg)
2/1/2005 Motion estimation 48
Optical Flow: Iterative EstimationSome Implementation Issues:
warping is not easy (make sure that errors in interpolation and warping are not bigger than the estimate refinement)
warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration.
often useful to low-pass filter the images before motion estimation (for better derivative estimation, and somewhat better linear approximations to image intensity)
![Page 45: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/45.jpg)
2/1/2005 Motion estimation 49
Optical Flow: Iterative Estimation
Some Implementation Issues:• warping is not easy (make sure that errors in
interpolation and warping are not bigger than the estimate refinement)
• warp one image, take derivatives of the other so you don’t need to re-compute the gradient after each iteration.
• often useful to low-pass filter the images before motion estimation (for better derivative estimation, and somewhat better linear approximations to image intensity)
![Page 46: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/46.jpg)
2/1/2005 Motion estimation 50
Optical Flow: AliasingTemporal aliasing causes ambiguities in optical flow because images can have many pixels with the same intensity.
I.e., how do we know which ‘correspondence’ is correct?
nearest match is correct (no aliasing)
nearest match is incorrect (aliasing)
To overcome aliasing: coarse-to-fine estimationcoarse-to-fine estimation.
actual shift
estimated shift
![Page 47: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/47.jpg)
2/1/2005 Motion estimation 53
image Iimage J
aJwwarp refine
a
aΔ+
Pyramid of image J Pyramid of image I
image Iimage J
Coarse-to-Fine Estimation
u=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
![Page 48: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/48.jpg)
2/1/2005 Motion estimation 54
J Jw Iwarp refine
ina
a
+
J Jw Iwarp refine
a
a+
J
pyramid construction
J Jw Iwarp refine
a+
I
pyramid construction
outa
Coarse-to-Fine Estimation
![Page 49: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/49.jpg)
2/1/2005 Motion estimation 55
Global Motion Models
2D Models:AffineQuadraticPlanar projective transform (Homography)
3D Models:Instantaneous camera motion models Homography+epipolePlane+Parallax
![Page 50: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/50.jpg)
2/1/2005 Motion estimation 56
0)()( 654321 tyx IyaxaaIyaxaaI
Example: Affine Motion
Substituting into the B.C. Equation:yaxaayxv
yaxaayxu
654
321
),(
),(
Each pixel provides 1 linear constraint in 6 global unknowns
0 tyx IvIuI
2 tyx IyaxaaIyaxaaIaErr )()()( 654321
Least Square Minimization (over all pixels):
![Page 51: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/51.jpg)
2/1/2005 Motion estimation 57
Quadratic – instantaneous approximation to planar motion
Other 2D Motion Models
287654
82
7321
yqxyqyqxqqv
xyqxqyqxqqu
yyvxxu
yhxhh
yhxhhy
yhxhh
yhxhhx
','
and
'
'
987
654
987
321
Projective – exact planar motion
![Page 52: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/52.jpg)
2/1/2005 Motion estimation 58
3D Motion Models
ZxTTxxyyv
ZxTTyxxyu
ZYZYX
ZXZYX
)()1(
)()1(2
2
yyvxxu
thyhxh
thyhxhy
thyhxh
thyhxhx
',' :and
'
'
3987
1654
3987
1321
)(1
)(1
233
133
tytt
xyv
txtt
xxu
w
w
Local Parameter:
ZYXZYX TTT ,,,,,
),( yxZ
Instantaneous camera motion:
Global parameters:
Global parameters: 32191 ,,,,, ttthh
),( yx
Homography+Epipole
Local Parameter:
Residual Planar Parallax Motion
Global parameters: 321 ,, ttt
),( yxLocal Parameter:
![Page 53: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/53.jpg)
2/1/2005 Motion estimation 59
Correlation and SSD
For larger displacements, do template matching• Define a small area around a pixel as the template• Match the template against each pixel within a
search area in next image.• Use a match measure such as correlation,
normalized correlation, or sum-of-squares difference
• Choose the maximum (or minimum) as the match• Sub-pixel interpolation also possible
![Page 54: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/54.jpg)
2/1/2005 Motion estimation 63
Discrete Search vs. Gradient Based Estimation
Consider image I translated by
21
,00
2
,1
)),(),(),((
)),(),((),(
yxvvyuuxIyxI
vyuxIyxIvuE
yx
yx
00 ,vu
),(),(),(
),(),(
1001
0
yxyxIvyuxI
yxIyxI
The discrete search method simply searches for the best estimate.The gradient method linearizes the intensity function and solves for the estimate
![Page 55: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/55.jpg)
2/1/2005 Motion estimation 69
Correlation Window Size
Small windows lead to more false matchesLarge windows are better this way, but…
• Neighboring flow vectors will be more correlated (since the template windows have more in common)
• Flow resolution also lower (same reason)• More expensive to compute
Another way to look at this:Small windows are good for local search but more precise
and less smoothLarge windows good for global search but less precise and
more smooth method
![Page 56: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/56.jpg)
2/1/2005 Motion estimation 70
Optical Flow: Robust Estimation
Issue 7:Issue 7: Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers.
Sources of outliers (multiple motions):
specularities / highlights jpeg artifacts / interlacing / motion blur multiple motions (occlusion boundaries, transparency)
velocity spacevelocity space
u1
u2
++
![Page 57: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/57.jpg)
2/1/2005 Motion estimation 71
Robust Estimation
Noise distributions are often non-Gaussian, having much heavier tails. Noise samples from the tails are called outliers.
Sources of outliers (multiple motions):• specularities / highlights
• jpeg artifacts / interlacing / motion blur
• multiple motions (occlusion boundaries, transparency)velocity spacevelocity space
u1
u2
++
![Page 58: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/58.jpg)
2/1/2005 Motion estimation 72
Robust Estimation
Standard Least Squares Estimation allows too much influence for outlying points
)()
)()(
)()(
2
mxx
x
mxx
xmE
i
ii
ii
( Influence
![Page 59: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/59.jpg)
2/1/2005 Motion estimation 73
Robust Estimation
tsysxssd IvIuIvuE ),( Robust gradient constraint
),(),(),( ssssd vyuxJyxIvuE Robust SSD
![Page 60: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/60.jpg)
2/1/2005 Motion estimation 74
Robust EstimationProblem: Least-squares estimators penalize deviations between data & model with quadratic error fn (extremely sensitive to outliers)
error penalty function influence function
Redescending error functions (e.g., Geman-McClure) help to reduce the influence of outlying measurements.
error penalty function influence function
![Page 61: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/61.jpg)
2/1/2005 Motion estimation 75
Layered Motion Models
intensity (appearance)
alpha map (opacity)warp maps (motion)
Layered models provide a 2.5 representation, like cardboard cutouts.
Key players:
![Page 62: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/62.jpg)
Layered Scene Representations
![Page 63: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/63.jpg)
2/1/2005 Motion estimation 77
Motion representations
How can we describe this scene?
![Page 64: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/64.jpg)
2/1/2005 Motion estimation 78
Block-based motion prediction
Break image up into square blocks
Estimate translation for each block
Use this to predict next frame, code difference (MPEG-2)
![Page 65: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/65.jpg)
2/1/2005 Motion estimation 79
Layered motion
Break image sequence up into “layers”:
=
Describe each layer’s motion
![Page 66: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/66.jpg)
2/1/2005 Motion estimation 80
Outline
• Why layers?• 2-D layers [Wang & Adelson 94; Weiss 97]• 3-D layers [Baker et al. 98]• Layered Depth Images [Shade et al. 98]• Transparency [Szeliski et al. 00]
![Page 67: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/67.jpg)
2/1/2005 Motion estimation 81
Layered motion
Advantages:• can represent occlusions / disocclusions• each layer’s motion can be smooth• video segmentation for semantic processing
Difficulties:• how do we determine the correct number?• how do we assign pixels?• how do we model the motion?
![Page 68: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/68.jpg)
2/1/2005 Motion estimation 82
Layers for video summarization
![Page 69: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/69.jpg)
2/1/2005 Motion estimation 83
Background modeling (MPEG-4)
Convert masked images into a background sprite for layered video coding
+ + +
=
![Page 70: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/70.jpg)
2/1/2005 Motion estimation 84
What are layers?
[Wang & Adelson, 1994]
• intensities• alphas• velocities
![Page 71: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/71.jpg)
2/1/2005 Motion estimation 87
How do we form them?
![Page 72: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/72.jpg)
2/1/2005 Motion estimation 88
How do we estimate the layers?
1. compute coarse-to-fine flow
2. estimate affine motion in blocks (regression)
3. cluster with k-means
4. assign pixels to best fitting affine region
5. re-estimate affine motions in each region…
![Page 73: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/73.jpg)
2/1/2005 Motion estimation 89
Layer synthesis
For each layer:• stabilize the sequence with the affine motion• compute median value at each pixel
Determine occlusion relationships
![Page 74: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/74.jpg)
2/1/2005 Motion estimation 90
Results
![Page 75: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/75.jpg)
2/1/2005 Motion estimation 91
What if the motion is not affine?
Use a “regularized” (smooth) motion field
[Weiss, CVPR’97]
![Page 76: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/76.jpg)
A Layered Approach To Stereo Reconstruction
Simon Baker, Richard Szeliski and P. Anandan
CVPR’98
![Page 77: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/77.jpg)
2/1/2005 Motion estimation 93
Layered Stereo
Assign pixel to different “layers” (objects, sprites)
… already covered in Stereo Lecture 2 …
![Page 78: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/78.jpg)
Layer extraction from multiple images containing reflections and transparency
Richard SzeliskiShai AvidanP. AnandanCVPR’2000
“extra bonus material”
![Page 79: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/79.jpg)
2/1/2005 Motion estimation 95
Transparent motion
Photograph (Lee) and reflection (Michael)
![Page 80: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/80.jpg)
2/1/2005 Motion estimation 96
Previous work
Physics-based vision and polarization[Shafer et al.; Wolff; Nayar et al.]
Perception of transparency [Adelson…]
Transparent motion estimation[Shizawa & Mase; Bergen et al.; Irani et al.; Darrell & Simoncelli]
3-frame layer recovery [Bergen et al.]
![Page 81: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/81.jpg)
2/1/2005 Motion estimation 97
Problem formulation
X
Y
MotionMotionX,iX,i( )( )
MotionMotionY,iY,i( )( )++
![Page 82: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/82.jpg)
2/1/2005 Motion estimation 98
Image formation model
Pure additive mixing of positive signals
mk(x) = l Wkl fl(x)
or
mk = l Wkl fl
Assume motion is planar (perspective transform, aka homography)
![Page 83: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/83.jpg)
2/1/2005 Motion estimation 99
Two processing stages
Estimate the motions and initial layer
estimates
Compute optimal layer estimates (for
known motion)
![Page 84: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/84.jpg)
2/1/2005 Motion estimation 100
Dominant motion estimation
Stabilize sequence by dominant motion
robust affine [Bergen et al. 92; Szeliski & Shum]
![Page 85: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/85.jpg)
2/1/2005 Motion estimation 101
Dominant layer estimate
How do we form composite (estimate)?
TimeTime
Inte
nsit
yIn
tens
ity
![Page 86: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/86.jpg)
2/1/2005 Motion estimation 102
Average?
TimeTime
Inte
nsit
yIn
tens
ity
![Page 87: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/87.jpg)
2/1/2005 Motion estimation 103
Median?
Hint: all layers are non-negative
TimeTime
Inte
nsit
yIn
tens
ity
![Page 88: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/88.jpg)
2/1/2005 Motion estimation 104
Min-composite
Smallest value is over-estimate of layer
TimeTime
Inte
nsit
yIn
tens
ity
![Page 89: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/89.jpg)
2/1/2005 Motion estimation 105
Difference sequence
Subtract min-composite from original image
=
original - min composite = difference imageoriginal - min composite = difference image
![Page 90: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/90.jpg)
2/1/2005 Motion estimation 106
Min composite
TimeTime
Inte
nsit
yIn
tens
ity
(overestimate of background layer)(overestimate of background layer)
![Page 91: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/91.jpg)
2/1/2005 Motion estimation 107
Difference sequence
TimeTime
Inte
nsit
yIn
tens
ity
(underestimate of foreground layer)(underestimate of foreground layer)
![Page 92: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/92.jpg)
2/1/2005 Motion estimation 108
Stabilizing secondary motion
TimeTime
Inte
nsit
yIn
tens
ity
How do we form composite (estimate)?How do we form composite (estimate)?
![Page 93: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/93.jpg)
2/1/2005 Motion estimation 109
Max-composite
TimeTime
Inte
nsit
yIn
tens
ity
Largest value is Largest value is under-estimateunder-estimate of layer of layer
![Page 94: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/94.jpg)
2/1/2005 Motion estimation 110
Min-max alternation
Subtract secondary layer (under-estimate) from original sequence
Re-compute dominant motion and better min-composite
Iterate …
Does this process converge?
![Page 95: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/95.jpg)
2/1/2005 Motion estimation 111
Min-max alternation
Does this process converge?• in theory: yes• each iteration reduces number of mis-estimated
pixels (tightens the bounds) — proof in paper
![Page 96: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/96.jpg)
2/1/2005 Motion estimation 112
Min-max alternation
Does this process converge?• in practice: no• resampling errors and noise both lead to
divergence — discussion in paper
resampling error noisy
![Page 97: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/97.jpg)
2/1/2005 Motion estimation 113
Two processing stages
Estimate the motions and initial layer
estimates
Compute optimal layer estimates (for
known motion)
![Page 98: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/98.jpg)
2/1/2005 Motion estimation 114
Optimal estimation
Recall: additive mixing of positive signals
mk = l Wkl fl
Use constrained least squares(quadratic programming)
min k | l Wkl fl – mk |2 s.t. fl 0
![Page 99: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/99.jpg)
2/1/2005 Motion estimation 115
Least squares example
background foregroundbackground foreground
blue: least squaresblue: least squares
red: constrained LSred: constrained LS
![Page 100: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/100.jpg)
2/1/2005 Motion estimation 116
Uniqueness of solution
If any layer does not have a “black” region, i.e., if fl c, then can add this offset to another layer (and subtract it from fl)
background background foreground foreground
![Page 101: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/101.jpg)
2/1/2005 Motion estimation 117
Degeneracies in solution
If motion is degenerate (e.g., horizontal), regions (scanlines) decouple (w/o MRF)
mixedmixed
scaled scaled errors errors
recovered recovered
![Page 102: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/102.jpg)
2/1/2005 Motion estimation 118
Noise sensitivity
In general, low-frequency components hard to recover for small motions
mixedmixed
recovered recovered
scaled scaled errors errors
![Page 103: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/103.jpg)
2/1/2005 Motion estimation 119
Three-layer example
3 layers with general motion works well
= + +
![Page 104: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/104.jpg)
2/1/2005 Motion estimation 120
Complete algorithm
Dominant motion with min-composites
Difference (residual) images
Non-dominant motion on differences
Improve the motion estimates
Unconstrained least-squares problem
Constrained least-squares problem
![Page 105: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/105.jpg)
2/1/2005 Motion estimation 121
Complete example
originaloriginal
stabilizedstabilizedmin-compositemin-composite
![Page 106: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/106.jpg)
2/1/2005 Motion estimation 122
Complete example
differencedifference
stabilizedstabilizedmax-compositemax-composite
![Page 107: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/107.jpg)
2/1/2005 Motion estimation 123
Final Results
= += +
![Page 108: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/108.jpg)
2/1/2005 Motion estimation 124
Another example
original stabilized min-comp. resid. 2
![Page 109: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/109.jpg)
2/1/2005 Motion estimation 125
Results: Anne and books
= += +
original background foreground (photo)original background foreground (photo)
![Page 110: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/110.jpg)
2/1/2005 Motion estimation 126
Transparent layer recovery
Pure (additive) mixing of intensities• simple constrained least squares problem• degeneracies for simple or small motions
Processing stages• dominant motion estimation• min- and max-composites to initialize• optimization of motion and layers
![Page 111: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/111.jpg)
2/1/2005 Motion estimation 127
Future work
Mitigating degeneracies (regularization)Opaque layers ( estimation)
Non-planar geometry (parallax)
![Page 112: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/112.jpg)
2/1/2005 Motion estimation 128
BibliographyL. Williams. Pyramidal parametrics.Computer Graphics, 17(3):1--11, July 1983.
L. G. Brown. A survey of image registration techniques.Computing Surveys, 24(4):325--376, December 1992.
C. D. Kuglin and D. C. Hines. The phase correlation image alignment method.In IEEE 1975 Conference on Cybernetics and Society, pages 163--165, New York, September 1975.
J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping and Morphing of Graphical Objects.
Morgan Kaufmann Publishers, San Francisco Altos, California, 1999.
G. M. Nielson. Scattered data modeling.IEEE Computer Graphics and Applications, 13(1):60--70, January 1993.
T. Beier and S. Neely. Feature-based image metamorphosis.Computer Graphics (SIGGRAPH'92), 26(2):35--42, July 1992.
![Page 113: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/113.jpg)
2/1/2005 Motion estimation 129
Bibliography
J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation. In ECCV’92, pp. 237–252, Italy, May 1992.
M. J. Black and P. Anandan. The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Comp. Vis. Image Understanding, 63(1):75–104, 1996.
H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Trans. Patt. Anal. Mach. Intel., 18(8):814–830, Aug. 1996.
Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In CVPR’97, pp. 520–526, June 1997.
![Page 114: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/114.jpg)
2/1/2005 Motion estimation 130
BibliographyJ. Y. A. Wang and E. H. Adelson. Representing moving images with
layers. IEEE Transactions on Image Processing, 3(5):625--638, September 1994.
Y. Weiss and E. H. Adelson. A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'96), pages 321--326, San Francisco, California, June 1996.
Y. Weiss. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'97), pages 520--526, San Juan, Puerto Rico, June 1997.
P. R. Hsu, P. Anandan, and S. Peleg. Accurate computation of optical flow by using layered motion representations. In Twelfth International Conference on Pattern Recognition (ICPR'94), pages 743--746, Jerusalem, Israel, October 1994. IEEE Computer Society Press
![Page 115: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/115.jpg)
2/1/2005 Motion estimation 131
BibliographyT. Darrell and A. Pentland. Cooperative robust estimation using layers of
support. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(5):474--487, May 1995.
S. X. Ju, M. J. Black, and A. D. Jepson. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'96), pages 307--314, San Francisco, California, June 1996.
M. Irani, B. Rousso, and S. Peleg. Computing occluding and transparent motions. International Journal of Computer Vision, 12(1):5--16, January 1994.
H. S. Sawhney and S. Ayer. Compact representation of videos through dominant multiple motion estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(8):814--830, August 1996.
M.-C. Lee et al. A layered video object coding system using sprite and affine motion model. IEEE Transactions on Circuits and Systems for Video Technology, 7(1):130--145, February 1997.
![Page 116: Motion estimation Introduction to Computer Vision CS223B, Winter 2005 Richard Szeliski.](https://reader036.fdocuments.net/reader036/viewer/2022062320/56649d425503460f94a1dc3e/html5/thumbnails/116.jpg)
2/1/2005 Motion estimation 132
BibliographyS. Baker, R. Szeliski, and P. Anandan. A layered approach to stereo
reconstruction. In IEEE CVPR'98, pages 434--441, Santa Barbara, June 1998.
R. Szeliski, S. Avidan, and P. Anandan. Layer extraction from multiple images containing reflections and transparency. In IEEE CVPR'2000, volume 1, pages 246--253, Hilton Head Island, June 2000.
J. Shade, S. Gortler, L.-W. He, and R. Szeliski. Layered depth images. In Computer Graphics (SIGGRAPH'98) Proceedings, pages 231--242, Orlando, July 1998. ACM SIGGRAPH.
S. Laveau and O. D. Faugeras. 3-d scene representation as a collection of images. In Twelfth International Conference on Pattern Recognition (ICPR'94), volume A, pages 689--691, Jerusalem, Israel, October 1994. IEEE Computer Society Press.
P. H. S. Torr, R. Szeliski, and P. Anandan. An integrated Bayesian approach to layer extraction from image sequences. In Seventh ICCV'98, pages 983--990, Kerkyra, Greece, September 1999.