CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction...
Transcript of CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction...
![Page 1: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/1.jpg)
CS664 Lecture #19: Layers, RANSAC, panoramas, epipolar geometry
Some material taken from:
David Lowe, UBC
Jiri Matas, CMP Praguehttp://cmp.felk.cvut.cz/~matas/papers/presentations/matas_beyond-ransac_cvprac05.ppt
![Page 2: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/2.jpg)
2
Announcements
Paper report due on 11/15Please choose a final project soon– Email me a proposal after you get your paper
report graded – Which will be soon after 11/15!
Project must have a research component– Talk to me if you have questions
Next quiz Thursday 11/3– coverage through last lecture
PS#2 due November 8
![Page 3: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/3.jpg)
3
Handling large motions
None of these optical-flow based techniques work well for large motions or for really textured scenes– Need “just right” amount of texture!
A standard solution for larger motions is to do this “coarse to fine”– Run on a low resolution version of the image– Apply this as a warp, then increase resolution– These methods have some ugly properties
• But a number of people think they work
![Page 4: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/4.jpg)
4
Layered motion estimation
Suppose there are multiple motions– Think of “Bugs Bunny” style animation
![Page 5: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/5.jpg)
5
Layered motion algorithms
Global affine fit with IRLS should give us the dominant motion– Assuming it’s >50% of the image– Outlier pixels (low IRLS weights) are doing
something else, so we can focus on them
Need to find what the motions are, which pixels belong to each region– If we know one, we could compute the other– Classical application of EM– To initialize, compute local affine fits and
cluster in 6-dimensional space
![Page 6: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/6.jpg)
6
RANSAC
Extremely popular way to do model fitting in the presence of noisy data– One of the best algorithms in the vision toolkit
Algorithm:– Select a random sample of data points– Find model that best fits these points(LS)– Find all other data points that like this model
• Consensus set
Parameters: number of iterations, sample size, consensus set “tolerance”
![Page 7: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/7.jpg)
7
Example: registration
![Page 8: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/8.jpg)
8
Example: multiple motions
![Page 9: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/9.jpg)
9
RANSAC for sparse registration
Given an interest-point operator– Corner detector, or SIFT (we will cover this)
Assume we’re looking at a plane– Planar homography
• Homography = projective transformation• Planar homography = 2D affine homography
– Application: recognizing panoramas • Brown & Lowe, ICCV 2003
http://www.cs.ubc.ca/~mbrown/panorama
![Page 10: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/10.jpg)
10
Recognizing panoramas
![Page 11: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/11.jpg)
11
Feature extraction (SIFT)
![Page 12: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/12.jpg)
12
Consensus set
![Page 13: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/13.jpg)
13
Panorama
![Page 14: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/14.jpg)
14
Completing the panorama
We have a lot of pairwise panoramas– How do we create a complete panorama?– A connected set should be a panorama
How to create the image?– Pairwise registrations aren’t globally consistent
Camera parameters come in two classes– Intrinsic: focal length, pixel size & spacing– External: rigid body motion (6 d.o.f.)– For a given panorama, we will assume the
parameters are only rotation & focal length
![Page 15: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/15.jpg)
15
Bundle adjustment
We have a set of cameras with their parameters θold, and need to add a new image and estimate its parameters θ– Find the θ that makes the feature points line
up as closely as possible• Minimize the “reprojection error”• Robustify this, to handle outliers
– Actually, we minimize both θold and θ
Non-linear least squares problem– Usually solved with Levenberg-Marquadt
![Page 16: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/16.jpg)
16
Bundle adjustment in action
![Page 17: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/17.jpg)
17
Results
![Page 18: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/18.jpg)
18
Epipolar geometry
Where could a point in I1 appear in I2?– Motion: anywhere nearby– Stereo: anywhere horizontally nearby
• Why just horizontal?
– Assume a stationary scene
![Page 19: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/19.jpg)
19
Two View Geometry
Point X in world and two camera centers C, C’ define the epipolar plane
– Images x,x’ of X intwo image planeslie on this plane
– Intersection ofline CC’ withimage planesdefine specialpoints calledepipoles, e,e’
e e′
![Page 20: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/20.jpg)
20
Epipolar Lines
Set of points that project to x in I defineline l’ in I’
– Called epipolar line
– Goes throughepipole e’
– A point x in Ithus maps to apoint on l’ in I’
• Rather thanto a point anywhere in I
![Page 21: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/21.jpg)
21
Epipolar Geometry
Two-camera system defines one parameter family (pencil) of planes through baseline CC’
– Each such planedefines matchingepipolar lines intwo image planes
– One parameterfamily of lines through each epipole
– Correspondence between images
![Page 22: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/22.jpg)
22
Converging Stereo Cameras
Corresponding points lie on
corresponding epipolar lines
Known camera geometry so 1D not 2D
search!
![Page 23: CS664 Lecture #19: Layers, RANSAC, panoramas, …...Recognizing panoramas 11 Feature extraction (SIFT) 12 Consensus set 13 Panorama 14 Completing the panorama We have a lot of pairwise](https://reader033.fdocuments.net/reader033/viewer/2022050508/5f998548b99f9524ef5f1f62/html5/thumbnails/23.jpg)
23
Epipoles in direction of motion
e
e’