Previously
description
Transcript of Previously
Previously
Two view geometry: epipolar geometry Stereo vision: 3D reconstruction
epipolar linesepipolar linesepipolar linesepipolar lines
BaselineBaselineOO O’O’
epipolar planeepipolar plane
' 0Tp Ep
Today
Orthographic projection Two views 3 Views: geometric interpretation >3 Views: factorization – simultaneous recovery
of motion and shape
Orthographic Projection (Reminder) Parallel projection rays,
orthogonal to image plane Focal center at infinity
x X
y Y
Two Views
Implies that
Eliminating Z
Since
Therefore
23 13 23 11 13 21 23 12 13 22 23 13' ' ( ) ( ) ( )x yr x r y r r r r x r r r r y r t r t
Q RP t
11 12 13
21 22 23
'
'x
y
x r x r y r z t
y r x r y r z t
32 23 11 13 21
31 23 12 13 22
( )r r r r r
r r r r r
23 13 32 31 23 13' ' ( ) 0x yr x r y r x r y r t r t
TR R I
Epipolar lines
This is a linear equation
23
13
32 31 23 13
0 0
' ' 1 0 0 0
( ) 1x y
r x
x y r y
r r r t r t
23 13 32 31 23 13' ' ( ) 0x yr x r y r x r y r t r t
Further Simplification
Select one point in first image and its corresponding point in the second image to be the origin of the two images
In this coordinate frame translation is 0 Expression for epipolar lines:
23 13 32 31' ' 0r x r y r x r y
Epipolar Line Recovery
We need 4 corresponding points:
1 to eliminate translation 3 to determine the 4 components of R up to scale The rest of the components cannot be determined In particular, cannot be determined from
, because these components are known only up to scale
23 13 32 31' ' 0r x r y r x r y
2 2 213 23 33 1r r r
33r
Shape Recovery from Two Views Perspective:
Translation recovered up to scale 3D shape recovered up to scale Recovery only if non-zero translation No calibration – recovery up to a projective transformation
(“projective shape”) Orthographic:
Rotation along epipolar line cannot be recovered 3D shape cannot be recovered Recovery is possible up to an affine transformation
(“affine shape”) Recovery only if non-trivial rotation Translation along line at infinity = rotation
Recovery from Three Views
Under orthographic projection metric recovery is possible from three views
Only rotation matters Rotation has three degrees of freedom Given an image, one rotation is in the image and
two are out of plane rotations Ignoring the in-plane rotation we can associate
the image with a point on the unit sphere
Recovery from Three Views
Im1
Im2
Im3
Recovery from Three Views
a
b
c
Im1
Im2
Im3
Recovery from Three Views
a, b, c are unknown –
rotation angles are known –
angles between epipolar lines Can we determine lengths
from angles?
a
b
c
Recovery from Three Views
In the plane the angles determine the sides of a triangle up to scale
Recovery from Three Views
On a sphere the sides are determined completely by the angles
Therefore three views determine all the components of rotation (up to reflection)
Sides (rotation angles) can be computed using the analogue of the cosine theorem on the sphere
Once the rotation is known structure can be recovered
a
b
c
Factorization
Simultaneous recovery of shape and motion Input:
A video sequence Tracked feature points
Assumptions: Rigid scene Orthographic projection All tracked points appear in all frames
Observation: tracked point locations satisfy linear relations that can be exploited for robust recovery
Singular Value Decomposition (SVD) Every real matrix can be decomposed to a
product of three matrices:
With diagonal U,V orthonormal
U orthonormal basis to row space of M V orthonormal basis to column space of M
nmM
Tnnnmmmnm VUM
,T TU U I V V I
Singular Value Decomposition (SVD)
1
2
0 0 0 0 0
0 0 0 0 0
0 0 0 0 0m
1
2
0 0 0
0 0 0
0 0 0
0 0 0 0
0 0 0 0
n
or
0...321 k
are called the singular values i
Tnnnmmmnm VUM
Relation to Eigenvalue Problems
2
2
2
2
2
2
( )( )
( )( )
T T T T
T
Ti i i
T T T T
T
Ti i i
M M V U U V V V
M MV V
M Mv v
MM U V V U U U
MM U U
MM u u
Singular Value Decomposition (SVD)Rank k least squares approximation of M Example: k=3
Take the 3 largest singular values:
Rank 3 approximation of M:
1
2
3
0 0
0 0
0 0
Tnmnm VUM 3333
~~~~
Factorization
Goal: given p corresponding points in f frames, compute the 3-D location of each point and the transformation between the frames
M T S
Measurements Transformation Shape (3-D locations)
Factorization
Step 1: eliminate translationSet the centroid of the points in each frame to be the
origin
Now11 12 13
21 22 23
Xr r rx
Yr r ry
Z
Factorization
11
11
y
x
Constructing M :
Factorization
1211
1211
yy
xx
Constructing M :
Factorization
p
p
yyy
xxx
11211
11211
...
...
Constructing M :
Factorization
p
p
p
p
yyy
yyy
xxx
xxx
22221
11211
22221
11211
...
...
...
...
Constructing M :
Factorization
11 12 1
1 2
11 12 1
1 2 2
...
...
...
...
p
f f fp
p
f f fp f p
x x x
x x x
y y y
y y y
Constructing M :
Factorization
p
fpffpf
p
fpf
p
ZZ
YY
XX
rrr
rrr
rrr
rrr
yy
yy
xx
xx
321
21
21
32
223
222
221
123
122
121
213
212
211
113
112
111
21
111
1
111
...
...
...
.........
.........
...
...
...
M T S
Goal: given M, find S and T
What should rank(M) be?
Factorization
Goal: given M, find S and T Compute the SVD of M
rank(M) should be 3, since rank(T)=rank(S)=3 Noise cleaning: find the rank 3 approximation of M
using the 3 largest singular values
Tpppfffpf VUM 2222
1
2
3
0 0
0 0
0 0
Tnmnm VUM 3333
~~~~
MM~
So far:
Define
The decomposition can now be written as
Factorization is not unique, since
, A invertible
Factorization
Tnmnm VUM 3333
~~~~
~~~UT TVS
~~~
STM~~~
STSAATM ˆˆ)~
()~(
~ 1
SASATT~ˆ~ˆ 1
T should contain valid rotations 3f equations, 6 unknowns:
Each row is defined as And should maintain
Factorization1 1 1
111 12 13 1
2 2 2211 12 13 2
1 1 121 22 23 1 1
2 2 221 22 23 2 2
TT
TT
T T
T T
T T
ar r r rar r r r
T T A Ar r r s b
r r r s b
,i i i iAa r Ab s
1
0
1
Ti i
Ti i
Ti i
r r
r s
s s
1
0
1
T Ti i
T Ti i
T Ti i
a A Aa
a A Ab
b A Ab
( )TR R I
Factorization
is 3x3, symmetric Linear system of equations in 6 unknowns Once B is recovered it can be factored to find A Solution is unique up to a global rotation
TB A A
TR R I
1
0
1
T Ti i
T Ti i
T Ti i
a A Aa
a A Ab
b A Ab
Input: tracked sequence
Eliminate translation, produce M
Use SVD to find the rank 3 approximation of M
Factorization ambiguous, up to invertible matrix
Find matrix A such that T contains valid rotations
Solution is unique (up to a global rotation)
Output: motion , shape
Factorization (Algorithm)
1 0 1T T T T T Ti i i i i ia A Aa a A Ab b A Ab
STSAATM ˆˆ)~
()~(
~ 1
STVUVUM TTnmnm
~~)
~~()
~~(
~~~~3333
T S
Factorization
Advantages Simultaneous recovery of shape and motion Simple algorithm, based on linear equations Robust to noise
Disadvantages Orthographic projection All points should appear in all frames
(factorization with missing data is difficult)
Summary
Shape and motion recovery under orthographic projection
Two views: Parallel epipolar lines 4 corresponding points are needed Recovery of affine shape
Three or more views Metric recovery Simultaneous recovery of shape and motion using
SVD factorization