EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case...
Transcript of EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case...
![Page 1: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/1.jpg)
EECS 442 – Computer vision
Multiple view geometry
Affine structure from Motion
Reading: [HZ] Chapters: 6,14,18
[FP] Chapter: 12
Some slides of this lectures are courtesy of prof. J. Ponce,
prof FF Li, prof S. Lazebnik & prof. M. Hebert
- Affine structure from motion problem
- Algebraic methods
- Factorization methods
![Page 2: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/2.jpg)
Applications Courtesy of Oxford Visual Geometry Group
![Page 3: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/3.jpg)
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
•xij = Mi Xj , i = 1, … , m, j = 1, … , n
![Page 4: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/4.jpg)
From the mxn correspondences xij, estimate:
•m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motion
structure
M1
M2
Mm
Structure from motion problem
![Page 5: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/5.jpg)
Affine structure from motion (simpler problem)
Image World
Image
From the mxn correspondences xij, estimate:
•m projection matrices Mi (affine cameras)
•n 3D points Xj
![Page 6: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/6.jpg)
Finite cameras
p
q
r
R
Q
P
O
XTRKx
M
![Page 7: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/7.jpg)
??10
0100
0010
0001
TR
Question:
![Page 8: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/8.jpg)
Finite cameras
p
q
r
R
Q
P
O
XTRKx
10
TR
0100
0010
0001
KM 3x3
Canonical perspective projection matrix
Affine homography
(in 3D)
Affine
Homography
(in 2D)
100
y0
xs
K oy
ox
![Page 9: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/9.jpg)
Projective & Affine cameras
XTRKx
10
TR
0100
0010
0001
KM
100
y0
xs
K oy
ox
Projective case
Affine case
![Page 10: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/10.jpg)
Scaling function of the distance (magnification)
myy
mxx
'
'
Weak perspective projection When the relative scene depth is small compared to its distance from the camera
![Page 11: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/11.jpg)
Orthographic (affine) projection
y'y
x'x
When the camera is at a (roughly constant) distance from the scene
–Distance from center of projection
to image plane is infinite
![Page 12: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/12.jpg)
Transformation in 2D
Affinities:
1
y
x
H
1
y
x
10
tA
1
'y
'x
a
![Page 13: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/13.jpg)
Projective & Affine cameras
XTRKx
100
yα0
xsα
oy
ox
K
10
TR
1000
0010
0001
KM
10
TR
0100
0010
0001
KM
100
y0
xs
K oy
ox
Projective case
Affine case
Parallel projection matrix
(points at infinity are mapped as points at infinity) Magnification (scaling term)
![Page 14: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/14.jpg)
Affine cameras
XTRKx
100
00
00
y
x
K
10
TR
1000
0010
0001
KM
10
bA
1000
]affine44[
1000
0010
0001
]affine33[ 2232221
1131211
baaa
baaa
M
12
1
232221
131211 XbAXx EucM
b
b
Z
Y
X
aaa
aaa
y
x
[Homogeneous]
[non-homogeneous
image coordinates] bAMMEuc
;1
PEucM
![Page 15: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/15.jpg)
P p
p’
;1
PbAPp M
v
u bAM
Affine cameras
M = camera matrix
To recap: from now on we define M as the camera matrix for the affine case
![Page 16: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/16.jpg)
The Affine Structure-from-Motion Problem
Given m images of n fixed points Pj (=Xi) we can write
Problem: estimate the m 24 matrices Mi and
the n positions Pj from the mn correspondences pij .
2m n equations in 8m+3n unknowns
Two approaches: - Algebraic approach (affine epipolar geometry; estimate F; cameras; points)
- Factorization method
How many equations and how many unknown?
N of cameras N of points
![Page 17: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/17.jpg)
Algebraic analysis (2-view case)
- Derive the fundamental matrix FA for the
affine case
- Compute FA
- Use FA to estimate projection matrices
- Use projection matrices to estimate 3D
points
![Page 18: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/18.jpg)
1. Deriving the fundamental matrix FA
P p
p’
Homogeneous system
u
v
Dim= ? 4x4
![Page 19: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/19.jpg)
The Affine Fundamental Matrix!
where
Deriving the fundamental matrix FA
Are the epipolar lines parallel or converging?
![Page 20: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/20.jpg)
![Page 21: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/21.jpg)
Affine Epipolar Geometry
![Page 22: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/22.jpg)
Estimating FA
• From n correspondences, we obtain a linear system on
the unknown alpha, beta, etc…
• Measurements: u, u’, v, v’
0
1vuvu
1vuvu
nnnn
1111
f
• Computed by least square and by enforcing |f|=1
• SVD
![Page 23: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/23.jpg)
P p
p’
Estimating projection matrices from FA
![Page 24: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/24.jpg)
Affine ambiguity
PQQMPMp A
-1
A
Affine
![Page 25: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/25.jpg)
P p
p’
Estimating projection matrices from FA
![Page 26: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/26.jpg)
Estimating projection matrices from FA
Choose Q such
that…
'~
M
Where a,b,c,d can be expressed as function of the parameters of FA
See HZ page 348
![Page 27: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/27.jpg)
A factorization method – Tomasi & Kanade algorithm
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization
method. IJCV, 9(2):137-154, November 1992.
• Centering the data
• Factorization
![Page 28: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/28.jpg)
Centering: subtract the centroid of the image points
ji
n
1k
kji
n
1k
ikiiji
n
1k
ikijij
ˆn
1
n
1
n
1ˆ
XAXXA
bXAbXAxxx
A factorization method - Centering the data
Xk
xik xi ^
![Page 29: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/29.jpg)
Centering: subtract the centroid of the image points
ji
n
1k
kji
n
1k
ikiiji
n
1k
ikijij
ˆn
1
n
1
n
1ˆ
XAXXA
bXAbXAxxx
A factorization method - Centering the data
![Page 30: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/30.jpg)
Centering: subtract the centroid of the image points
n
k
kji
n
k
ikiiji
n
k
ikijij
n
nn
1
11
1
11ˆ
XXA
bXAbXAxxx
jiij XAx ˆ
A factorization method - Centering the data
Assume that the origin of the world coordinate system is at the centroid of
the 3D points
After centering, each normalized point xij is related to the 3D point Xi by
![Page 31: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/31.jpg)
X x
x’
A factorization method - Centering the data
jiij XAx ˆ
![Page 32: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/32.jpg)
Let’s create a 2m n data (measurement) matrix:
mnmm
n
n
xxx
xxx
xxx
D
ˆˆˆ
ˆˆˆ
ˆˆˆ
21
22221
11211
cameras
(2 m )
points (n )
A factorization method - factorization
![Page 33: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/33.jpg)
Let’s create a 2m n data (measurement) matrix:
n
mmnmm
n
n
XXX
A
A
A
xxx
xxx
xxx
D
21
2
1
21
22221
11211
ˆˆˆ
ˆˆˆ
ˆˆˆ
cameras
(2 m × 3)
points (3 × n )
The measurement matrix D = M S has rank 3 (it’s a product of a 2mx3 matrix and 3xn matrix)
A factorization method - factorization
(2 m × n)
M
S
![Page 34: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/34.jpg)
Factorizing the measurement matrix
Source: M. Hebert
![Page 35: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/35.jpg)
Factorizing the measurement matrix
Singular value decomposition of D:
Source: M. Hebert
![Page 36: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/36.jpg)
Factorizing the measurement matrix
Singular value decomposition of D:
Source: M. Hebert
Since rank (D)=3, there are only 3 non-zero singular values
![Page 37: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/37.jpg)
Factorizing the measurement matrix
Obtaining a factorization from SVD:
M = Motion (cameras)
S = structure
What is the issue here?
D has rank>3 because of - measurement noise
- affine approximation
![Page 38: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/38.jpg)
Factorizing the measurement matrix
Obtaining a factorization from SVD:
S = structure
D
D
M = motion
![Page 39: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/39.jpg)
Affine ambiguity
The decomposition is not unique. We get the same D by
using any 3×3 matrix C and applying the
transformations M → MC, S →C-1S
We can enforce some Euclidean constraints to resolve
this ambiguity (more on next lecture!)
![Page 40: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/40.jpg)
Algorithm summary
1. Given: m images and n features xij
2. For each image i, center the feature coordinates
3. Construct a 2m × n measurement matrix D:
• Column j contains the projection of point j in all views
• Row i contains one coordinate of the projections of all the n
points in image i
4. Factorize D:
• Compute SVD: D = U W VT
• Create U3 by taking the first 3 columns of U
• Create V3 by taking the first 3 columns of V
• Create W3 by taking the upper left 3 × 3 block of W
5. Create the motion and shape matrices:
• M = M = U3 and S = W3V3T (or U3W3
½ and S = W3½ V3
T )
6. Eliminate affine ambiguity
![Page 41: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/41.jpg)
Reconstruction results
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography:
A factorization method. IJCV, 9(2):137-154, November 1992.
![Page 42: EECS 442 Computer vision Multiple view geometry Affine ...x o D D Projective case Affine case Parallel projection matrix Magnification (scaling term) (points at infinity are mapped](https://reader033.fdocuments.net/reader033/viewer/2022051911/60011a7327eede65d33cb096/html5/thumbnails/42.jpg)
Next lecture
Multiple view geometry
Perspective structure from Motion