Post on 31-Jul-2020
Affine Structure from Motion
Instructor: Jason Corso (jjcorso)!web.eecs.umich.edu/~jjcorso/t/598F14!
!
Materials on these slides have come from many sources in addition to myself (primarily Silvio Savarese to whom I am ultimately grateful); individual slides reference specific sources.!
EECS 598-08 Fall 2014!Foundations of Computer Vision!!
Readings: FP 8.2!Date: 11/5/14!!
Plan
• What is affine SFM?!• Algebraic Methods from Two Views!• Factorization!
2
Courtesy of Oxford Visual Geometry Group!
Application
Structure from motion problem!
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points !
• xij = Mi Xj , i = 1, … , m, j = 1, … , n Courtesy of Silvio Savarese.!
From the mxn correspondences xij, estimate: !• m projection matrices Mi • n 3D points Xj
x1j
x2j
xmj
Xj
motion!structure!
M1
M2
Mm
Structure from motion problem!
Courtesy of Silvio Savarese.!
Affine structure from motion!(simpler problem)!
Image!World!
Image!
From the mxn correspondences xij, estimate: !• m projection matrices Mi (affine cameras) • n 3D points Xj
Courtesy of Silvio Savarese.!
p!q!r!
R!
Q!
P!
O!
[ ]XTRKx =M
Courtesy of Silvio Savarese.!
??10
010000100001
=⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡TR
Question:!
Courtesy of Silvio Savarese.!
p!q!r!
R!
Q!
P!
O!
[ ]XTRKx =
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=10TR
010000100001
KM 3x3
Canonical perspective projection matrix!
Affine homography !(in 3D)!
Affine !Homography !(in 2D)!
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
100y0xs
K oy
ox
α
α
Courtesy of Silvio Savarese.!
Projective & Affine cameras!
[ ]XTRKx =
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=10TR
010000100001
KM⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
100y0xs
K oy
ox
α
α
Projective case!
Affine case!
Courtesy of Silvio Savarese.!
Scaling function of the distance (magnification)!
⎩⎨⎧
−=
−=
myymxx
''
Weak perspective projection!When the relative scene depth is small compared to its distance from the camera!
Courtesy of Silvio Savarese.!
Orthographic (affine) projection!
⎩⎨⎧
=
=
y'yx'x
When the camera is at a (roughly constant) distance from the scene!
– Distance from center of projection ! to image plane is infinite!
Courtesy of Silvio Savarese.!
Transformation in 2D
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1yx
H1yx
10tA
1'y'x
a
Courtesy of Silvio Savarese.!
Projective & Affine cameras!
[ ]XTRKx =
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
100yα0xsα
oy
ox
K ⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=10TR
100000100001
KM
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=10TR
010000100001
KM⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
100y0xs
K oy
ox
α
α
Projective case!
Affine case!
Parallel projection matrix !(points at infinity are mapped as points at infinity)!Magnification (scaling term)!
Courtesy of Silvio Savarese.!
Affine cameras!
[ ]XTRKx =
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
1000000
y
x
K α
α
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=10TR
100000100001
KM
⎥⎦
⎤⎢⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=×
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
×=10bA
1000]affine44[
100000100001
]affine33[ 2232221
1131211
baaabaaa
M
⎥⎦
⎤⎢⎣
⎡=+=⎟⎟
⎠
⎞⎜⎜⎝
⎛+⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎦
⎤⎢⎣
⎡=⎟⎟
⎠
⎞⎜⎜⎝
⎛=
12
1
232221
131211 XbAXx EucM
bb
ZYX
aaaaaa
yx
[Homogeneous]!
[non-homogeneous!image coordinates]!
[ ]bAMMEuc ==
;1⎥⎦
⎤⎢⎣
⎡=
PEucM
Courtesy of Silvio Savarese.!
P!p!
p’!
;1⎥⎦
⎤⎢⎣
⎡=+=⎟⎟
⎠
⎞⎜⎜⎝
⎛=
PbAPp M
vu [ ]bAM =
Affine cameras!
M = camera matrix!To recap: !from now on we define M as the camera matrix for the affine case!
Courtesy of Silvio Savarese.!
The Affine Structure-from-Motion Problem!
Given m images of n fixed points Pj (=Xi) we can write!
Problem: estimate the m 2×4 matrices Mi and!the n positions Pj from the m×n correspondences pij .!
2m × n equations in 8m+3n unknowns!
Two approaches:!- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)!- Factorization method!
How many equations and how many unknown?!
N of cameras! N of points!
Courtesy of Silvio Savarese.!
Algebraic analysis (2-view case)!
- Derive the fundamental matrix FA for the affine case!- Compute FA!- Use FA to estimate projection matrices !- Use projection matrices to estimate 3D points!
Courtesy of Silvio Savarese.!
1. Deriving the fundamental matrix FA!
P!p!
p’!
Homogeneous system!
u!
v!
Dim= ?!4x4!
Courtesy of Silvio Savarese.!Affine Epipolar Constraint!
The Affine Fundamental Matrix!!
where!
Deriving the fundamental matrix FA!
Are the epipolar lines parallel or converging?!Courtesy of Silvio Savarese.!
Courtesy of Silvio Savarese.!
Affine Epipolar Geometry!
Courtesy of Silvio Savarese.!
Estimating FA!
• From n correspondences, we obtain a linear system on! the unknown alpha, beta, etc…!
• Measurements: u, u’, v, v’!
01vuvu
1vuvu
nnnn
1111
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
ʹ′ʹ′
ʹ′ʹ′
f!!!!!
• Computed by least square and by enforcing |f|=1!• SVD!
Courtesy of Silvio Savarese.!
P!p!
p’!
Estimating projection matrices from FA!
Courtesy of Silvio Savarese.!
Affine ambiguity
( )( )PQQMPMp A-1
A==
Affine
Courtesy of Silvio Savarese.!
If Mi and Pi are solutions, !then Mi’ and Pi’ are also solutions, !
and!
2. Estimating projection matrices !from epipolar constraints!
Proof:!
where!
Q is an affine!transformation.!
Courtesy of Silvio Savarese.!
P!p!
p’!
3. Estimating projection matrices from FA!
Courtesy of Silvio Savarese.!
Estimating projection matrices from FA!
Choose Q such that…!
'~M
Where a,b,c,d can be expressed as function of the parameters of FA!
Courtesy of Silvio Savarese.!
4. Estimating the structure from FA!
A’! b’!
Can be solved by least square again!
A! b!
Courtesy of Silvio Savarese.!
[ ]Td0b~cba100
A~
=ʹ′
⎥⎦
⎤⎢⎣
⎡=ʹ′
3. Estimating projection matrices from epipolar constraints!
Choose Q such that…!
Function of the parameters of F!
[ ]T00b~010001
A~
=
⎥⎦
⎤⎢⎣
⎡=
Canonical affine cameras!
Courtesy of Silvio Savarese.!
Estimating projection matrices from epipolar constraints!
Choose Q such that…!
By re-enforcing the epipolar constraint, we can compute a, b, c, d directly from the measurements!
Courtesy of Silvio Savarese.!
Reminder: epipolar constraint!
P!p!
p’!
Homogeneous system!
u!
v!
Courtesy of Silvio Savarese.!
Estimating projection matrices from epipolar constraints!
Choose Q such that…!
A! b!
Re-enforce the Epipolar constraint!
~!~!
Courtesy of Silvio Savarese.!
Estimating projection matrices from epipolar constraints!
Choose Q such that…!
A! b!
Courtesy of Silvio Savarese.!
Estimating projection matrices from epipolar constraints!
• From at least 4 correspondences, we can solve this linear system! and compute a, b, c, d (via least square)!
Unknown: a, b, c, d!Measurements: u, u’, v, v’!
• Linear relationship between measurements and unknown!
• The cameras can be computed!
• How about the structure?!Courtesy of Silvio Savarese.!
4. Estimating the structure from FA!
A! b!
Can be solved by least square again!Courtesy of Silvio Savarese.!
First reconstruction. Mean re-projection error: 1.6pixel!
Second reconstruction. Mean re-projection error: 7.8pixel!
Courtesy of Silvio Savarese.!
A factorization method – Tomasi & Kanade algorithm
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):137-154, November 1992. !! !• Centering the data!• Factorization !
Courtesy of Silvio Savarese.!
• Centering: subtract the centroid of the image points!
( )
ji
n
1kkji
n
1kikiiji
n
1kikijij
ˆn1
n1
n1ˆ
XAXXA
bXAbXAxxx
=⎟⎠
⎞⎜⎝
⎛−=
+−+=−=
∑
∑∑
=
==
A factorization method - Centering the data
Xk!
xik xi -!
xi -!
Courtesy of Silvio Savarese.!
• Centering: subtract the centroid of the image points!
( )
ji
n
1kkji
n
1kikiiji
n
1kikijij
ˆn1
n1
n1ˆ
XAXXA
bXAbXAxxx
=⎟⎠
⎞⎜⎝
⎛−=
+−+=−=
∑
∑∑
=
==
A factorization method - Centering the data
ijiij bXAx +=
Courtesy of Silvio Savarese.!
• Centering: subtract the centroid of the image points!
( )
⎟⎠
⎞⎜⎝
⎛−=
+−+=−=
∑
∑∑
=
==
n
kkji
n
kikiiji
n
kikijij
n
nn
1
11
1
11ˆ
XXA
bXAbXAxxx
jiij XAx =ˆ
A factorization method - Centering the data
Assume that the origin of the world coordinate system is at the centroid of the 3D points!!After centering, each normalized point xij is related to the 3D point Xi by!
Courtesy of Silvio Savarese.!
X!x!
x’!
A factorization method - Centering the data!
jiij XAx =ˆ
Courtesy of Silvio Savarese.!
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
mnmm
n
n
xxx
xxxxxx
D
ˆˆˆ
ˆˆˆˆˆˆ
21
22221
11211
!"!!
cameras(2 m )!
points (n )!
A factorization method - factorization
Let’s create a 2m × n data (measurement) matrix:!
Courtesy of Silvio Savarese.!
• Let’s create a 2m × n data (measurement) matrix:!
[ ]n
mmnmm
n
n
XXX
A
AA
xxx
xxxxxx
D !"
!#!!
212
1
21
22221
11211
ˆˆˆ
ˆˆˆˆˆˆ
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
cameras(2 m × 3)!
points (3 × n )!
The measurement matrix D = M S has rank 3!(it’s a product of a 2mx3 matrix and 3xn matrix)!
A factorization method - factorization
(2 m × n)!M!
S!
Factorizing the measurement matrix
Source: M. Hebert
Factorizing the measurement matrix • Singular value decomposition of D:!
Source: M. Hebert!
Factorizing the measurement matrix • Singular value decomposition of D:!
Source: M. Hebert!
Since rank (D)=3, there are only 3 non-zero singular values!
Factorizing the measurement matrix • Obtaining a factorization from SVD:!
M = Motion (cameras) !
S = structure!
What is the issue here?!D has rank>3 because of - measurement noise !! ! ! ! - affine approximation!
Courtesy of Silvio Savarese.!
Factorizing the measurement matrix • Obtaining a factorization from SVD:!
S = structure!
D!D!
M = motion!
Courtesy of Silvio Savarese.!
Affine ambiguity
• The decomposition is not unique. We get the same D by using any 3×3 matrix C and applying the transformations M → MC, S →C-1S!
• We can enforce some Euclidean constraints to resolve !• this ambiguity (more on next lecture!)!
C C-1
M’ S’
Courtesy of Silvio Savarese.!
Algorithm summary 1. Given: m images and n features xij!2. For each image i, center the feature coordinates!3. Construct a 2m × n measurement matrix D:!
– Column j contains the projection of point j in all views!– Row i contains one coordinate of the projections of all
the n points in image i!4. Factorize D:!
– Compute SVD: D = U W VT!
– Create U3 by taking the first 3 columns of U!– Create V3 by taking the first 3 columns of V!– Create W3 by taking the upper left 3 × 3 block of W!
5. Create the motion and shape matrices:!– M = M = U3 and S = W3V3
T (or U3W3½ and S = W3
½ V3
T )!6. Eliminate affine ambiguity!
Courtesy of Silvio Savarese.!
Reconstruction results
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):137-154, November 1992.
Courtesy of Silvio Savarese.!
Next Lecture: Perspective SFM
• Readings: FP 8.3!
53