Image Stitching - courses.engr.illinois.edu
Transcript of Image Stitching - courses.engr.illinois.edu
![Page 1: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/1.jpg)
10/19/10
Image Stitching
Computational PhotographyDerek Hoiem, University of Illinois
Photos by Russ Hewett
![Page 2: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/2.jpg)
Last Class: Keypoint Matching
K. Grauman, B. Leibe
Af Bf
A1
A2 A3
Tffd BA <),(
1. Find a set of distinctive key-points
3. Extract and normalize the region content
2. Define a region around each keypoint
4. Compute a local descriptor from the normalized region
5. Match local descriptors
![Page 3: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/3.jpg)
Last Class: Summary
• Keypoint detection: repeatable and distinctive– Corners, blobs– Harris, DoG
• Descriptors: robust and selective– SIFT: spatial histograms of gradient
orientation
![Page 4: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/4.jpg)
Today: Image Stitching• Combine two or more overlapping images to
make one larger image
Add example
Slide credit: Vaibhav Vaish
![Page 5: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/5.jpg)
Panoramic Imaging
• Higher resolution photographs, stitched from multiple images
• Capture scenes that cannot be captured in one frame
• Cheaply and easily achieve effects that used to cost a lot of money
• Like HDR and Focus Stacking, use computational methods to go beyondthe physical limitations of the camera
![Page 6: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/6.jpg)
Photo: Russell J. Hewett
Pike’s Peak Highway, CO
Nikon D70s, Tokina 12-24mm @ 16mm, f/22, 1/40s
![Page 7: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/7.jpg)
Photo: Russell J. Hewett
Pike’s Peak Highway, CO
(See Photo On Web)
![Page 8: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/8.jpg)
Software
• Panorama Factory (not free)
• Hugin (free)
• (But you are going to write your own, right?)
![Page 9: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/9.jpg)
Photo: Russell J. Hewett
360 Degrees, Tripod Leveled
Nikon D70, Tokina 12-24mm @ 12mm, f/8, 1/125s
![Page 10: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/10.jpg)
Photo: Russell J. Hewett
Howth, Ireland
(See Photo On Web)
![Page 11: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/11.jpg)
Capturing Panoramic Images
• Tripod vs Handheld• Help from modern cameras• Leveling tripod• Gigapan• Or wing it
• Exposure• Consistent exposure between frames• Gives smooth transitions• Manual exposure• Makes consistent exposure of dynamic scenes easier• But scenes don’t have constant intensity everywhere
• Caution• Distortion in lens (Pin Cushion, Barrel, and Fisheye)• Polarizing filters• Sharpness in image edge / overlap region
• Image Sequence• Requires a reasonable amount of overlap (at least 15-30%)• Enough to overcome lens distortion
![Page 12: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/12.jpg)
Photo: Russell J. Hewett
Handheld Camera
Nikon D70s, Nikon 18-70mm @ 70mm, f/6.3, 1/200s
![Page 13: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/13.jpg)
Photo: Russell J. Hewett
Handheld Camera
![Page 14: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/14.jpg)
Photo: Russell J. Hewett
Les Diablerets, Switzerland
(See Photo On Web)
![Page 15: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/15.jpg)
Photo: Russell J. Hewett & Bowen Lee
Macro
Nikon D70s, Tamron 90mm Micro @ 90mm, f/10, 15s
![Page 16: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/16.jpg)
Photo: Russell J. Hewett & Bowen Lee
Side of Laptop
![Page 17: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/17.jpg)
Considerations For Stitching
• Variable intensity across the total scene
• Variable intensity and contrast between frames
• Lens distortion• Pin Cushion, Barrel, and Fisheye• Profile your lens at the chosen focal length (read from EXIF)• Or get a profile from LensFun
• Dynamics/Motion in the scene• Causes ghosting• Once images are aligned, simply choose from one or the other
• Misalignment• Also causes ghosting• Pick better control points
• Visually pleasing result• Super wide panoramas are not always ‘pleasant’ to look at• Crop to golden ratio, 10:3, or something else visually pleasing
![Page 18: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/18.jpg)
Photo: Russell J. Hewett
Ghosting and Variable Intensity
Nikon D70s, Tokina 12-24mm @ 12mm, f/8, 1/400s
![Page 19: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/19.jpg)
Photo: Russell J. Hewett
![Page 20: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/20.jpg)
Photo: Bowen Lee
Ghosting From Motion
Nikon e4100 P&S
![Page 21: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/21.jpg)
Photo: Russell J. Hewett Nikon D70, Nikon 70-210mm @ 135mm, f/11, 1/320s
Motion Between Frames
![Page 22: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/22.jpg)
Photo: Russell J. Hewett
![Page 23: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/23.jpg)
Photo: Russell J. Hewett
Gibson City, IL
(See Photo On Web)
![Page 24: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/24.jpg)
Photo: Russell J. Hewett
Mount Blanca, CO
Nikon D70s, Tokina 12-24mm @ 12mm, f/22, 1/50s
![Page 25: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/25.jpg)
Photo: Russell J. Hewett
Mount Blanca, CO
(See Photo On Web)
![Page 26: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/26.jpg)
Problem basics• Do on board
![Page 27: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/27.jpg)
Basic problem • x = K [R t] X• x’ = K’ [R’ t’] X’• t=t’=0
• x‘=Hx where H = K’ R’ R-1 K-1
• Typically only R and f will change (4 parameters), but, in general, H has 8 parameters
f f'
.x
x'
X
![Page 28: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/28.jpg)
Views from rotating camera
Camera Center
![Page 29: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/29.jpg)
Image Stitching Algorithm Overview
1. Detect keypoints2. Match keypoints3. Estimate homography with four matched
keypoints (using RANSAC)4. Project onto a surface and blend
![Page 30: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/30.jpg)
Image Stitching Algorithm Overview
1. Detect keypoints (e.g., SIFT)2. Match keypoints (most similar features,
compared to 2nd most similar)
![Page 31: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/31.jpg)
Computing homography
Assume we have four matched points: How do we compute homography H?
Direct Linear Transformation (DLT)
Hxx ='
0h =
′′′−−−
′′′−−−vvvvuvuuuvuuvu
10000001
=
'''''
'w
vwuw
x
=
987
654
321
hhhhhhhhh
H
=
9
8
7
6
5
4
3
2
1
hhhhhhhhh
h
![Page 32: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/32.jpg)
Computing homographyDirect Linear Transform
• Apply SVD: UDVT = A• h = Vsmallest (column of V corr. to smallest singular value)
=
=
987
654
321
9
2
1
hhhhhhhhh
h
hh
Hh
0Ah0h =⇒=
′′′−−−
′′′−−−
′′′−−−
nnnnnnn vvvvuvu
vvvvuvuuuvuuvu
1000
10000001
1111111
1111111
Matlab[U, S, V] = svd(A);h = V(:, end);
![Page 33: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/33.jpg)
Computing homographyAssume we have four matched points: How do we compute homography H?
Normalized DLT1. Normalize coordinates for each image
a) Translate for zero meanb) Scale so that average distance to origin is sqrt(2)
– This makes problem better behaved numerically (see Hartley and Zisserman p. 107-108)
2. Compute H using DLT in normalized coordinates3. Unnormalize:
Txx =~ xTx ′′=′~
THTH ~1−′=
ii Hxx =′
![Page 34: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/34.jpg)
Computing homography
• Assume we have matched points with outliers: How do we compute homography H?
Automatic Homography Estimation with RANSAC
![Page 35: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/35.jpg)
RANSAC: RANdom SAmple ConsensusScenario: We’ve got way more matched points than needed to fit the parameters, but we’re not sure which are correct
RANSAC Algorithm• Repeat N times
1. Randomly select a sample– Select just enough points to recover the parameters2. Fit the model with random sample3. See how many other points agree
• Best estimate is one with most agreement– can use agreeing points to refine estimate
![Page 36: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/36.jpg)
Computing homography• Assume we have matched points with outliers: How do
we compute homography H?
Automatic Homography Estimation with RANSAC1. Choose number of samples N2. Choose 4 random potential matches3. Compute H using normalized DLT4. Project points from x to x’ for each potentially
matching pair:5. Count points with projected distance < t
– E.g., t = 3 pixels6. Repeat steps 2-5 N times
– Choose H with most inliers
HZ Tutorial ‘99
ii Hxx =′
![Page 37: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/37.jpg)
Automatic Image Stitching
1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched points and RANSAC with normalized DLT
4. Project each image onto the same surface and blend
![Page 38: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/38.jpg)
Choosing a Projection SurfaceMany to choose: planar, cylindrical, spherical, cubic, etc.
![Page 39: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/39.jpg)
Planar vs. Cylindrical Projection
Planar
Photos by Russ Hewett
![Page 40: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/40.jpg)
Planar vs. Cylindrical Projection
Planar
![Page 41: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/41.jpg)
Planar vs. Cylindrical Projection
Cylindrical
![Page 42: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/42.jpg)
Planar vs. Cylindrical Projection
Cylindrical
![Page 43: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/43.jpg)
Planar
Cylindrical
![Page 44: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/44.jpg)
Simple gain adjustment
![Page 45: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/45.jpg)
Automatically choosing images to stitch
![Page 46: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/46.jpg)
Recognizing Panoramas
Brown and Lowe 2003, 2007Some of following material from Brown and Lowe 2003 talk
![Page 47: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/47.jpg)
Recognizing PanoramasInput: N images1. Extract SIFT points, descriptors from all
images2. Find K-nearest neighbors for each point (K=4)3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched image
![Page 48: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/48.jpg)
Recognizing PanoramasInput: N images1. Extract SIFT points, descriptors from all
images2. Find K-nearest neighbors for each point (K=4)3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched imagec) Decide if match is valid (ni > 8 + 0.3 nf )
# inliers # keypoints in overlapping area
![Page 49: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/49.jpg)
RANSAC for Homography
Initial Matched Points
![Page 50: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/50.jpg)
RANSAC for Homography
Final Matched Points
![Page 51: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/51.jpg)
Verification
![Page 52: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/52.jpg)
RANSAC for Homography
![Page 53: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/53.jpg)
Recognizing Panoramas (cont.)(now we have matched pairs of images)4. Find connected components
![Page 54: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/54.jpg)
Finding the panoramas
![Page 55: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/55.jpg)
Finding the panoramas
![Page 56: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/56.jpg)
Finding the panoramas
![Page 57: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/57.jpg)
Recognizing Panoramas (cont.)(now we have matched pairs of images)4. Find connected components5. For each connected component
a) Perform bundle adjustment to solve for rotation (θ1, θ2, θ3) and focal length f of all cameras
b) Project to a surface (plane, cylinder, or sphere)c) Render with multiband blending
![Page 58: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/58.jpg)
Bundle adjustment for stitching• Non-linear minimization of re-projection error
• where H = K’ R’ R-1 K-1
• Solve non-linear least squares (Levenberg-Marquardt algorithm)– See paper for details
)ˆ,(1∑∑∑ ′′=
N M
j k
i
disterror xx
Hxx =′ˆ
![Page 59: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/59.jpg)
Bundle AdjustmentNew images initialized with rotation, focal length of the best matching image
![Page 60: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/60.jpg)
Bundle AdjustmentNew images initialized with rotation, focal length of the best matching image
![Page 61: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/61.jpg)
Blending
• Gain compensation: minimize intensity difference of overlapping pixels
• Blending– Pixels near center of image get more weight– Multiband blending to prevent blurring
![Page 62: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/62.jpg)
Multi-band Blending (Laplacian Pyramid)
• Burt & Adelson 1983– Blend frequency bands over range ∝ λ
![Page 63: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/63.jpg)
Multiband blending
![Page 64: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/64.jpg)
Blending comparison (IJCV 2007)
![Page 65: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/65.jpg)
Blending Comparison
![Page 66: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/66.jpg)
StraighteningRectify images so that “up” is vertical
![Page 67: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/67.jpg)
Further readingHarley and Zisserman: Multi-view Geometry book• DLT algorithm: HZ p. 91 (alg 4.2), p. 585• Normalization: HZ p. 107-109 (alg 4.2)• RANSAC: HZ Sec 4.7, p. 123, alg 4.6• Tutorial:
http://users.cecs.anu.edu.au/~hartley/Papers/CVPR99-tutorial/tut_4up.pdf
• Recognising Panoramas: Brown and Lowe, IJCV 2007 (also bundle adjustment)
![Page 68: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/68.jpg)
Things to remember
• Homography relates rotating cameras
• Recover homography using RANSAC and normalized DLT
• Can choose surface of projection: cylinder, plane, and sphere are most common
• Lots of room for tweaking (blending, straightening, etc.)
![Page 69: Image Stitching - courses.engr.illinois.edu](https://reader035.fdocuments.net/reader035/viewer/2022062401/62aab09b68972c2962240046/html5/thumbnails/69.jpg)
Next class
• Using interest points to find objects in datasets