Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from...
Transcript of Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from...
![Page 1: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/1.jpg)
• Project 3 Due Friday• Artifact due at the same time
• We are planning to announce new grading guidelines this week
Announcements
![Page 2: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/2.jpg)
The Fundamental Matrix & Structure from Motion
![Page 3: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/3.jpg)
• One algebraic form can have many geometric interpretations• Let’s review the algebra, and some of the geometry relevant for
today…
The Many Meanings of Vectors and Cross Products
![Page 4: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/4.jpg)
• Compute 3D vector c that has a dot product of zero with both 3D vectors a, and b
Cross-Products: The Algebra
![Page 5: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/5.jpg)
Cross-Product as Linear Operator
Useful fact: Cross product with a vector t can be represented as multiplication with a (skew-symmetric) 3x3 matrix
![Page 6: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/6.jpg)
• The cross product of two vectors is the normal vector to plane between that contains them
Cross Products: Geometric Interpretation
![Page 7: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/7.jpg)
• A point in 3D
• A line in 3D is defined by the equation
• Points and lines are both vectors of the same dimension with a homogeneous coordinate
Point-Line Duality and the Cross-Product
![Page 8: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/8.jpg)
• The line between two points is just the cross product of the two points
Point-Line Duality and the Cross-Product
![Page 9: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/9.jpg)
• The intersection of two lines is the cross produce of those lines
Point-Line Duality and the Cross-Product
![Page 10: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/10.jpg)
• One vector can have multiple interpretations• I suggest:
• First think about the geometry• Then the representation• Then the algebra
Point-Line Duality and the Cross-Product
![Page 11: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/11.jpg)
Back to stereo
• Where do epipolar lines come from?
![Page 12: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/12.jpg)
Two-view geometry
• Where do epipolar lines come from?
epipolar plane
epipolarepipolar lineepipolar line
0
3D point lies somewhere along r
(projection of r)
Image 1 Image 2
![Page 13: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/13.jpg)
Fundamental matrix
• This epipolar geometry of two views is described by a Very Special 3x3 matrix , called the fundamental matrix
• maps (homogeneous) points in image 1 to lines in image 2!
• The epipolar line (in image 2) of point p is:
• Epipolar constraint on corresponding points:
epipolar plane
epipolarepipolar lineepipolar line
0
(projection of ray)
Image 1 Image 2
![Page 14: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/14.jpg)
Fundamental matrix
• Two Special points: e1 and e2 (the epipoles): projection of one camera into the other
epipolar plane
epipolarepipolar lineepipolar line
0
(projection of ray)
![Page 15: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/15.jpg)
Fundamental matrix
• Two Special points: e1 and e2 (the epipoles): projection of one camera into the other
• All of the epipolar lines in an image pass through the epipole
0
![Page 16: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/16.jpg)
Epipoles
![Page 17: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/17.jpg)
Properties of the Fundamental Matrix
• is the epipolar line associated with
• is the epipolar line associated with
• and
• is rank 2
• How many degrees of freedom does F have?
18
T
![Page 18: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/18.jpg)
Fundamental matrix
• Why does F exist?• Let’s derive it…
0
![Page 19: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/19.jpg)
Fundamental matrix – calibrated case
0
: intrinsics of camera 1 : intrinsics of camera 2
: rotation of image 2 w.r.t. camera 1
: ray through p in camera 1’s (and world) coordinate system
: ray through q in camera 2’s coordinate system
: translation of cam 2 wrt cam 1
![Page 20: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/20.jpg)
Fundamental matrix – calibrated case
• , , and are coplanar• epipolar plane can be represented as
0
![Page 21: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/21.jpg)
Fundamental matrix – calibrated case
0
![Page 22: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/22.jpg)
Fundamental matrix – calibrated case
• One more substitution:• Cross product with t (on left) can be represented as a 3x3
matrix
0
![Page 23: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/23.jpg)
Fundamental matrix – calibrated case
0
![Page 24: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/24.jpg)
Fundamental matrix – calibrated case
0
: ray through p in camera 1’s (and world) coordinate system
: ray through q in camera 2’s coordinate system
{the “Essential matrix”
![Page 25: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/25.jpg)
Fundamental matrix – uncalibrated case
0
the Fundamental matrix
: intrinsics of camera 1 : intrinsics of camera 2
: rotation of image 2 w.r.t. camera 1: translation of cam 2 wrt cam 1
![Page 26: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/26.jpg)
Putting the “Fun” in “Fundamental Matrix”
https://www.youtube.com/watch?v=DgGV3l82NTk
![Page 27: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/27.jpg)
Rectified case
![Page 28: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/28.jpg)
• reproject image planes onto a common plane parallel to the line between optical centers
• pixel motion is horizontal after this transformation
• two homographies (3x3 transform), one for each input image reprojection
• C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
Stereo image rectification
![Page 29: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/29.jpg)
Original stereo pair
After rectification
![Page 30: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/30.jpg)
Relationship with homography?
Images taken from the same center of projection? Use a homography!
![Page 31: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/31.jpg)
Questions?
![Page 32: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/32.jpg)
Estimating F
• If we don’t know K1, K2, R, or t, can we estimate F for two images?
• Yes, given enough correspondences
![Page 33: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/33.jpg)
Estimating F – 8-point algorithm
• The fundamental matrix F is defined by
0=ΤFxx'for any pair of matches x and x’ in two images.
• Let x=(u,v,1)T and x’=(u’,v’,1)T,
=
333231
232221
131211
fffffffff
F
each match gives a linear equation
0'''''' 333231232221131211 =++++++++ fvfuffvfvvfuvfufvufuu
![Page 34: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/34.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
=
fffffffff
vuvvvvuuuvuu
vuvvvvuuuvuuvuvvvvuuuvuu
nnnnnnnnnnnn
• Like with homographies, instead of solving , we seek f to minimize , least eigenvector of .
0=AfAf
AAΤ
![Page 35: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/35.jpg)
8-point algorithm – Problem?
• F should have rank 2• To enforce that F is of rank 2, F is replaced by F’ that
minimizes subject to the rank constraint. 'FF −
• This is achieved by SVD. Let , where
, let
then is the solution.
Τ= VUF Σ
=
3
2
1
000000
Σσ
σσ
=
0000000
Σ' 2
1
σσ
Τ= VUF Σ''
![Page 36: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/36.jpg)
8-point algorithm% Build the constraint matrix
A = [x2(1,:)'.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...
x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...
x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
% Extract fundamental matrix from the column of V
% corresponding to the smallest singular value.
F = reshape(V(:,9),3,3)';
% Enforce rank2 constraint
[U,D,V] = svd(F);
F = U*diag([D(1,1) D(2,2) 0])*V';
![Page 37: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/37.jpg)
8-point algorithm
• Pros: it is linear, easy to implement and fast• Cons: susceptible to noise
![Page 38: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/38.jpg)
0
1´´´´´´
1´´´´´´1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
=
fffffffff
vuvvvvuuuvuu
vuvvvvuuuvuuvuvvvvuuuvuu
nnnnnnnnnnnn
Problem with 8-point algorithm
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!Orders of magnitude differencebetween column of data matrix→ least-squares yields poor results
![Page 39: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/39.jpg)
Normalized 8-point algorithm
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
−
−
1
1500
2
10700
2
normalized least squares yields good resultsTransform image to ~[-1,1]x[-1,1]
![Page 40: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/40.jpg)
Normalized 8-point algorithm
1. Transform input by ,2. Call 8-point on to obtain3.
ii Txx =ˆ 'i
'i Txx =ˆ
'ii xx ˆ,ˆ
TFTF ˆΤ'=F̂
0=ΤFxx'
0ˆ'ˆ 1 =−Τ−Τ xFTTx'
F̂
![Page 41: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/41.jpg)
Normalized 8-point algorithm
A = [x2(1,:)'.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...
x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...
x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
F = reshape(V(:,9),3,3)';
[U,D,V] = svd(F);
F = U*diag([D(1,1) D(2,2) 0])*V';
% DenormaliseF = T2'*F*T1;
[x1, T1] = normalise2dpts(x1);[x2, T2] = normalise2dpts(x2);
![Page 42: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/42.jpg)
Results (ground truth)
![Page 43: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/43.jpg)
Results (8-point algorithm)
![Page 44: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/44.jpg)
Results (normalized 8-point algorithm)
![Page 45: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/45.jpg)
What about more than two views?
• The geometry of three views is described by a 3 x 3 x 3 tensor called the trifocal tensor
• The geometry of four views is described by a 3 x 3 x 3 x 3 tensor called the quadrifocal tensor
• After this it starts to get complicated…
![Page 46: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/46.jpg)
Large-scale structure from motion
Dubrovnik, Croatia. 4,619 images (out of an initial 57,845).Total reconstruction time: 23 hoursNumber of cores: 352
![Page 47: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/47.jpg)
Structure from motion
CS5670: Computer VisionAbe Davis, Slides by Noah Snavely
![Page 48: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/48.jpg)
Readings
• Szeliski, Chapter 7.1 – 7.4
![Page 49: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/49.jpg)
Structure from motion
• Multi-view stereo assumes that cameras are calibrated• Extrinsics and intrinsics are known for all views
• How do we compute calibration if we don’t know it? In general, this is called structure from motion
![Page 50: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/50.jpg)
Large-scale structure from motion
Dubrovnik, Croatia. 4,619 images (out of an initial 57,845). Total reconstruction time: 23 hours Number of cores: 352
![Page 51: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/51.jpg)
Two views
• Solve for Fundamental matrix / Essential matrix• Factorize into intrinsics, rotation, and translation
![Page 52: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/52.jpg)
What about more than two views?
• The geometry of three views is described by a 3 x 3 x 3 tensor called the trifocal tensor
• The geometry of four views is described by a 3 x 3 x 3 x 3 tensor called the quadrifocal tensor
• After this it starts to get complicated…• Instead, we explicitly solve for camera poses and scene
geometry
![Page 53: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/53.jpg)
• Given many images, how can we • a) figure out where they were all taken from?• b) build a 3D model of the scene?
• This is (roughly) the structure from motion problem
Structure from motion
![Page 54: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/54.jpg)
Structure from motion
• Input: images with points in correspondence pi,j = (ui,j,vi,j)
• Output• structure: 3D location xi for each point pi• motion: camera parameters Rj , tj possibly Kj
• Objective function: minimize reprojection error
Reconstruction (side) (top)
![Page 55: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/55.jpg)
Also doable from video
![Page 56: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/56.jpg)
What we’ve seen so far…
2D transformations between images
• Translations, affine transformations, homographies…
Fundamental matrices
• Represent relationships between 2D images in the form of corresponding 2D lines
What’s new: Explicitly representing 3D geometry of cameras and points
![Page 57: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/57.jpg)
Input
![Page 58: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/58.jpg)
Camera calibration and triangulation
• Suppose we know 3D points• And have matches between these points and an image• How can we compute the camera parameters?
• Suppose we have known camera parameters, each of which observes a point
• How can we compute the 3D location of that point?
![Page 59: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/59.jpg)
Structure from motion
• SfM solves both of these problems at once• A kind of chicken-and-egg problem
• (but solvable)
![Page 60: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/60.jpg)
Photo Tourism
![Page 61: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/61.jpg)
First step: how to get correspondence?
• Feature detection and matching
![Page 62: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/62.jpg)
• Detect features using SIFT [Lowe, IJCV 2004]
Feature Detection
![Page 63: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/63.jpg)
• Detect features using SIFT [Lowe, IJCV 2004]
Feature Detection
![Page 64: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/64.jpg)
• Detect features using SIFT [Lowe, IJCV 2004]
• Match features between each pair of images
• Refine matching using RANSAC to estimate fundamental matrix between each pair
Feature Matching
![Page 65: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/65.jpg)
• Graph of connectivity based on matched features
Image connectivity graph
(graph layout produced using the Graphviz toolkit: http://www.graphviz.org/)
![Page 66: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/66.jpg)
Correspondence estimation
• Link up pairwise matches to form connected components of matches across several images
Image 1 Image 2 Image 3 Image 4
![Page 67: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/67.jpg)
Input
![Page 68: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/68.jpg)
Structure from motion
Camera 1
Camera 2
Camera 3R1,t1
R2,t2
R3,t3
X1
X4
X3
X2
X5
X6
X7
minimizeg(R,T,X)
p1,1
p1,2
p1,3
X non-linear least squares
![Page 69: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/69.jpg)
Problem size
• What are the variables? • How many variables per camera?• How many variables per point?
• Trevi Fountain collection466 input photos
+ > 100,000 3D points= very large optimization problem
![Page 70: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/70.jpg)
Structure from motion
• Minimize sum of squared reprojection errors:
• Minimizing this function is called bundle adjustment
• Optimized using non-linear least squares, e.g. Levenberg-Marquardt
predictedimage location
observedimage location
indicator variable:is point i visible in image j ?
![Page 71: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/71.jpg)
Is SfM always uniquely solvable?
![Page 72: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/72.jpg)
Is SfM always uniquely solvable?
• No…
![Page 73: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/73.jpg)
Incremental structure from motion
![Page 74: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/74.jpg)
Incremental structure from motion
![Page 75: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/75.jpg)
Incremental structure from motion
![Page 76: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/76.jpg)
Incremental structure from motion
Time-lapse reconstruction of Dubrovnik, Croatia, viewed from above
![Page 77: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/77.jpg)
Photo Explorer (Part of Noah’s PhD work)
![Page 78: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/78.jpg)
![Page 79: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/79.jpg)
![Page 80: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/80.jpg)
![Page 81: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/81.jpg)
https://en.wikipedia.org/wiki/Libration
![Page 82: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/82.jpg)
Questions?
![Page 83: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/83.jpg)
SfM – Failure cases• Necker reversal
![Page 84: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/84.jpg)
![Page 85: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/85.jpg)
SfM applications
• 3D modeling• Surveying• Robot navigation and mapmaking• Visual effects…
– (see video)
![Page 86: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/86.jpg)
SfM applications
• 3D modeling• Surveying• Robot navigation and mapmaking• Virtual and augmented reality• Visual effects (“Match moving”)
– https://www.youtube.com/watch?v=RdYWp70P_kY
![Page 87: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/87.jpg)
Applications – Hyperlapse
https://www.youtube.com/watch?v=SOpwHaQnRSYhttps://www.youtube.com/watch?v=sA4Za3Hv6ng
![Page 88: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/88.jpg)
PhotoTourism
![Page 89: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/89.jpg)
Applications: Visual Reality & Augmented Reality
Hololenshttps://www.youtube.com/watch?v=FMtvrTGnP04
Oculushttps://www.youtube.com/watch?v=KOG7yTz1iTA
![Page 90: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/90.jpg)
Applications: Simultaneous
localization and mapping (SLAM)
https://www.youtube.com/watch?v=k43xJs3Roqg https://www.youtube.com/watch?v=ZR1yXFAslSk
![Page 91: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/91.jpg)
Application: Simultaneous localization and mapping (SLAM)
Scape: Building the ‘AR Cloud’: Part Three —3D Maps, the Digital Scaffolding of the 21st Centuryhttps://medium.com/scape-technologies/building-the-ar-cloud-part-three-3d-maps-the-digital-scaffolding-of-the-21st-century-465fa55782dd
![Page 92: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/92.jpg)
Application: AR walking directions
![Page 93: Announcements - cs.cornell.edu€¦ · Announcements. The Fundamental Matrix & Structure from Motion • One algebraic form can have many geometric interpretations • Let’s review](https://reader033.fdocuments.net/reader033/viewer/2022052017/602fb79080f4d55cab1df975/html5/thumbnails/93.jpg)
Questions?