Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2...
Transcript of Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2...
![Page 1: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/1.jpg)
Stereo
16-385 Computer VisionSpring 2018, Lecture 11http://www.cs.cmu.edu/~16385/
![Page 2: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/2.jpg)
Course announcements
• Homework 2 is due on February 23rd.- Any questions about the homework?- How many of you have looked at/started/finished homework 2?
• Yannis’ office hours on Friday are 1-3:30pm.
• Yannis has extra office hours on Wednesday 3-5pm – right after the class.
![Page 3: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/3.jpg)
Computer vision talks at CMU• VASC Seminar: https://www.ri.cmu.edu/events/category/vasc-seminar/month/
- VASC stands for Vision and Autonomous Systems Center (no idea).- One of the longest-running and best-known departmental vision seminars.
• Faculty candidate talks: https://www.ri.cmu.edu/events/- Many vision talks this year.- Some of the best our field has to offer – the rising stars of computer vision.
Feb 27: Pulkit AgrawalMarch 05: Judy HoffmanMarch 07: Manolis SavvaMarch 08: David FouheyMarch 21: Katie BoumanMarch 27: Saurabh GuptaApril 03: Jia Deng
• Robotics Institute Seminar:https://www.ri.cmu.edu/events/category/robotics-seminar/month/- Most of the time about robotic, but sometimes about vision.
• Talks in other departments (MLD, CSD).
![Page 4: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/4.jpg)
Computer vision talks at CMU
• Find all events: https://www.ri.cmu.edu/events/
• I will be sending notifications on Piazza about vision talks.
• I will also be starting discussion threads on Piazza about vision talks.
• Attending and commenting on talks regularly can count towards an extra 5% participation credit.
![Page 5: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/5.jpg)
Overview of today’s lecture
• Plug: epipolar imaging and computational photography.
• Revisiting triangulation.
• Disparity.
• Stereo rectification.
• Stereo matching.
• Improving stereo matching.
• Structured light.
![Page 6: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/6.jpg)
Slide credits
Some of these slides were adapted directly from:
• Kris Kitani (16-385, Spring 2017).• Srinivasa Narasimhan (16-823, Spring 2017).
![Page 7: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/7.jpg)
Plug: epipolar imaging.
![Page 8: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/8.jpg)
A couple of interesting videosNotice anything unusual about them?
Video stream 1 Video stream 2
![Page 9: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/9.jpg)
Epipolar imaging camera
Built here at CMU.
![Page 10: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/10.jpg)
Regular Imaging
Light Source Sensor
![Page 11: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/11.jpg)
Regular Imaging
Light Source Sensor
![Page 12: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/12.jpg)
Regular Imaging
Light Source Sensor
![Page 13: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/13.jpg)
Regular Imaging
Light Source Sensor
![Page 14: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/14.jpg)
Epipolar Imaging
Light Source Sensor & Mask
Epipolar Plane
![Page 15: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/15.jpg)
Epipolar Imaging
Light Source Sensor & Mask
![Page 16: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/16.jpg)
Epipolar Imaging
Light Source Complete Image
![Page 17: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/17.jpg)
Epipolar Imaging
Light Source Sensor & Mask
![Page 18: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/18.jpg)
Epipolar Imaging
Light Source Sensor & Mask
![Page 19: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/19.jpg)
![Page 20: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/20.jpg)
![Page 21: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/21.jpg)
![Page 22: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/22.jpg)
![Page 23: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/23.jpg)
![Page 24: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/24.jpg)
![Page 25: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/25.jpg)
![Page 26: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/26.jpg)
![Page 27: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/27.jpg)
cameras that see around corners
cameras that measure depth in real time
15-463/15-663/15-862 Computational PhotographyLearn about this and other unconventional cameras – and build some on your own!
http://graphics.cs.cmu.edu/courses/15-463/
cameras that take video at the speed of light
cameras that capture entire focal stacks
![Page 28: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/28.jpg)
Revisiting triangulation
![Page 29: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/29.jpg)
How would you reconstruct 3D points?
Left image Right image
![Page 30: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/30.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)
![Page 31: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/31.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)2. Form epipolar line for that point in second image (how?)
![Page 32: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/32.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)2. Form epipolar line for that point in second image (how?)3. Find matching point along line (how?)
![Page 33: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/33.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)2. Form epipolar line for that point in second image (how?)3. Find matching point along line (how?)4. Perform triangulation (how?)
![Page 34: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/34.jpg)
Triangulation
right imageleft image
3D point
left camera with matrix right camera with matrix
![Page 35: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/35.jpg)
How would you reconstruct 3D points?
Left image Right image
1. Select point in one image (how?)2. Form epipolar line for that point in second image (how?)3. Find matching point along line (how?)4. Perform triangulation (how?)
What are the disadvantages of this procedure?
![Page 36: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/36.jpg)
Stereo rectification
![Page 37: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/37.jpg)
What’s different between these two images?
![Page 38: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/38.jpg)
![Page 39: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/39.jpg)
![Page 40: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/40.jpg)
Objects that are close move more or less?
![Page 41: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/41.jpg)
The amount of horizontal movement is
inversely proportional to …
![Page 42: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/42.jpg)
The amount of horizontal movement is
inversely proportional to …
… the distance from the camera.
More formally…
![Page 43: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/43.jpg)
image plane
camera center camera center
3D point
![Page 44: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/44.jpg)
image plane
![Page 45: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/45.jpg)
![Page 46: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/46.jpg)
(baseline)
![Page 47: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/47.jpg)
(baseline)
How is X related to x?
![Page 48: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/48.jpg)
(baseline)
![Page 49: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/49.jpg)
(baseline)
How is X related to x’?
![Page 50: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/50.jpg)
(baseline)
![Page 51: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/51.jpg)
(baseline)
Disparity
(wrt to camera origin of image plane)
![Page 52: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/52.jpg)
(baseline)
Disparityinversely proportional
to depth
![Page 53: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/53.jpg)
Nomad robot searches for meteorites in Antarticahttp://www.frc.ri.cmu.edu/projects/meteorobot/index.html
Real-time stereo sensing
![Page 54: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/54.jpg)
![Page 55: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/55.jpg)
Subaru
Eyesight system
Pre-collision
braking
![Page 56: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/56.jpg)
![Page 57: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/57.jpg)
What other vision system uses disparity for depth sensing?
![Page 58: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/58.jpg)
![Page 59: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/59.jpg)
![Page 60: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/60.jpg)
![Page 61: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/61.jpg)
![Page 62: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/62.jpg)
![Page 63: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/63.jpg)
This is how 3D movies work
![Page 64: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/64.jpg)
Is disparity the only depth cue
the human visual system uses?
![Page 65: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/65.jpg)
So can I compute depth from any two
images of the same object?
![Page 66: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/66.jpg)
So can I compute depth from any two
images of the same object?
1. Need sufficient baseline
2. Images need to be ‘rectified’ first (make epipolar lines horizontal)
![Page 67: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/67.jpg)
1. Rectify images
(make epipolar lines horizontal)
2. For each pixel
a. Find epipolar line
b. Scan line for best match
c. Compute depth from disparity
![Page 68: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/68.jpg)
How can you make the epipolar lines horizontal?
![Page 69: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/69.jpg)
image plane
camera center camera center
3D point
What’s special about these two cameras?
![Page 70: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/70.jpg)
t
x
x’
R = I t = (T, 0, 0)
When this relationship holds:
When are epipolar lines horizontal?
![Page 71: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/71.jpg)
![Page 72: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/72.jpg)
t
x
x’
When are epipolar lines horizontal?
R = I t = (T, 0, 0)
Let’s try this out…
This always has to hold for rectified images
When this relationship holds:
![Page 73: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/73.jpg)
t
x
x’
R = I t = (T, 0, 0)
Write out the constraint
Let’s try this out…
When this relationship holds:
This always has to hold for rectified images
When are epipolar lines horizontal?
![Page 74: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/74.jpg)
t
x
x’
R = I t = (T, 0, 0)
y coordinate is
always the same!
Write out the constraint
Let’s try this out…
This always has to hold
The image of a 3D point will
always be on the same
horizontal line
When this relationship holds:
When are epipolar lines horizontal?
![Page 75: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/75.jpg)
It’s hard to make the image planes exactly parallel
![Page 76: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/76.jpg)
How can you make the epipolar lines horizontal?
![Page 77: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/77.jpg)
Use stereo rectification?
![Page 78: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/78.jpg)
What is stereo rectification?
![Page 79: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/79.jpg)
What is stereo rectification?
Reproject image
planes onto a
common plane
parallel to the line
between camera
centers
How can you do this?
![Page 80: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/80.jpg)
What is stereo rectification?
Reproject image
planes onto a
common plane
parallel to the line
between camera
centers
Need two
homographies (3x3
transform), one for
each input image
reprojection
C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision.Computer Vision and Pattern Recognition, 1999.
![Page 81: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/81.jpg)
Stereo Rectification
1. Rotate the right camera by R
(aligns camera coordinate system orientation only)
2. Rotate (rectify) the left camera so that the epipole
is at infinity
3. Rotate (rectify) the right camera so that the epipole
is at infinity
4. Adjust the scale
![Page 82: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/82.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 83: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/83.jpg)
rotate by R
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 84: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/84.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 85: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/85.jpg)
rotate by Rrect
rotate by Rrect
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 86: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/86.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 87: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/87.jpg)
scale by H
scale by H
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 88: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/88.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 89: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/89.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 90: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/90.jpg)
Step 1: Compute E to get R
SVD: Let
We get FOUR solutions:
two possible rotations two possible translations
![Page 91: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/91.jpg)
We get FOUR solutions:
Compute determinant of R, valid solution must be equal to 1(note: det(R) = -1 means rotation and reflection)
Compute 3D point using triangulation, valid solution has positive Z value(Note: negative Z means point is behind the camera )
Which one do we choose?
![Page 92: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/92.jpg)
camera center
image plane
op
tica
l axis
Camera Icon
Find the configuration where the points is in front of both cameras
Let’s visualize the four configurations…
![Page 93: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/93.jpg)
Find the configuration where the points is in front of both cameras
![Page 94: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/94.jpg)
Find the configuration where the points is in front of both cameras
![Page 95: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/95.jpg)
1. Compute E to get R
2. Rotate right image by R
3. Rotate both images by Rrect
4. Scale both images by H
Stereo Rectification:
![Page 96: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/96.jpg)
When do epipolar
lines become
horizontal?
![Page 97: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/97.jpg)
Parallel cameras
Where is the epipole?
![Page 98: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/98.jpg)
Parallel cameras
epipole at infinity
![Page 99: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/99.jpg)
Setting the epipole to infinity(Building Rrect from e)
Let
epipole coincides with translation vector
cross product of e and
the direction vector of
the optical axis
orthogonal vector
Given: epipole e
(using SVD on E)
(translation from E)
![Page 100: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/100.jpg)
If and orthogonal
then
![Page 101: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/101.jpg)
If and orthogonal
then
Where is this point located on the image plane?
![Page 102: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/102.jpg)
If and orthogonal
then
Where is this point located on the image plane?
At x-infinity
![Page 103: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/103.jpg)
Stereo Rectification Algorithm
1. Estimate E using the 8 point algorithm (SVD)
2. Estimate the epipole e (SVD of E)
3. Build Rrect from e
4. Decompose E into R and T
5. Set R1=Rrect and R2 = RRrect
6. Rotate each left camera point (warp image)
[x’ y’ z’] = R1 [x y z]
7. Rectified points as p = f/z’[x’ y’ z’]
8. Repeat 6 and 7 for right camera points using R2
![Page 104: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/104.jpg)
Stereo Rectification Algorithm
1. Estimate E using the 8 point algorithm
2. Estimate the epipole e (solve Ee=0)
3. Build Rrect from e
4. Decompose E into R and T
5. Set R1=Rrect and R2 = RRrect
6. Rotate each left camera point x’~ Hx where H = KR1*You may need to alter the focal length (inside K) to keep points within the
original image size
7. Repeat 6 and 7 for right camera points using R2
![Page 105: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/105.jpg)
Unrectified
![Page 106: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/106.jpg)
Unrectified
Rectified
![Page 107: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/107.jpg)
Unrectified
Rectified
![Page 108: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/108.jpg)
What can we do after
rectification?
![Page 109: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/109.jpg)
Stereo matching
![Page 110: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/110.jpg)
Depth Estimation via Stereo Matching
![Page 111: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/111.jpg)
1. Rectify images
(make epipolar lines horizontal)
2. For each pixel
a. Find epipolar line
b. Scan line for best match
c. Compute depth from disparity
How would
you do this?
![Page 112: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/112.jpg)
Reminder from filtering
How do we detect an edge?
![Page 113: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/113.jpg)
Reminder from filtering
How do we detect an edge?• We filter with something that looks like an edge.
original
horizontal edge filter
vertical edge filter
1 0 -1
1
0
-1*
*
We can think of linear filtering as a way to evaluate how similar an image is locally to some template.
![Page 114: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/114.jpg)
Find this template
How do we detect the template in he following image?
![Page 115: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/115.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output What will the output look like?
![Page 116: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/116.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output
What went wrong?
![Page 117: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/117.jpg)
Find this template
How do we detect the template in he following image?
Solution 1: Filter the image using the template as filter kernel.
filter
image
output
Increases for higher local intensities.
![Page 118: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/118.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output What will the output look like?
template mean
![Page 119: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/119.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output
template mean
True detection
False detections
output
thresholding
What went wrong?
![Page 120: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/120.jpg)
Find this template
How do we detect the template in he following image?
Solution 2: Filter the image using a zero-mean template.
filter
image
output
template mean
output
Not robust to high-contrast areas
![Page 121: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/121.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output What will the output look like?
![Page 122: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/122.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output True detection
1-output
thresholding
What could go wrong?
![Page 123: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/123.jpg)
Find this template
How do we detect the template in he following image?
Solution 3: Use sum of squared differences (SSD).
filter
image
output
1-output
Not robust to local intensity changes
![Page 124: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/124.jpg)
Find this template
How do we detect the template in he following image?
Observations so far:
• subtracting mean deals with brightness bias
• dividing by standard deviation removes contrast bias
Can we combine the two effects?
![Page 125: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/125.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
filter
image
output
template mean
local patch mean
What will the output look like?
![Page 126: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/126.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 127: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/127.jpg)
Find this template
How do we detect the template in he following image?
Solution 4: Normalized cross-correlation (NCC).
True detections
1-output
thresholding
![Page 128: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/128.jpg)
What is the best method?
It depends on whether you care about speed or invariance.
• Zero-mean: Fastest, very sensitive to local intensity.
• Sum of squared differences: Medium speed, sensitive to intensity offsets.
• Normalized cross-correlation: Slowest, invariant to contrast and brightness.
![Page 129: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/129.jpg)
Stereo Block Matching
Matching cost
disparity
Left Right
scanline
• Slide a window along the epipolar line and compare contents of
that window with the reference window in the left image
• Matching cost: SSD or normalized correlation
![Page 130: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/130.jpg)
SSD
![Page 131: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/131.jpg)
Normalized cross-correlation
![Page 132: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/132.jpg)
Similarity Measure Formula
Sum of Absolute Differences (SAD)
Sum of Squared Differences (SSD)
Zero-mean SAD
Locally scaled SAD
Normalized Cross Correlation (NCC)
SAD SSD NCC Ground truth
![Page 133: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/133.jpg)
Effect of window size
W = 3 W = 20
![Page 134: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/134.jpg)
Effect of window size
W = 3 W = 20
Smaller window
+ More detail
- More noise
Larger window
+ Smoother disparity maps
- Less detail
- Fails near boundaries
![Page 135: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/135.jpg)
When will stereo block matching fail?
![Page 136: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/136.jpg)
When will stereo block matching fail?
textureless regions repeated patterns
specularities
![Page 137: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/137.jpg)
Improving stereo matching
![Page 138: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/138.jpg)
Block matching Ground truth
What are some problems with the result?
![Page 139: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/139.jpg)
How can we improve depth estimation?
![Page 140: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/140.jpg)
How can we improve depth estimation?
Too many discontinuities.
We expect disparity values to change slowly.
Let’s make an assumption:
depth should change smoothly
![Page 141: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/141.jpg)
Stereo matching as …
Energy Minimization
What defines a good stereo correspondence?1. Match quality
– Want each pixel to find a good match in the other image2. Smoothness
– If two pixels are adjacent, they should (usually) move about the same amount
![Page 142: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/142.jpg)
{ {
(block matching result) (smoothness function)
Want each pixel to find a good match
in the other image
Adjacent pixels should (usually)
move about the same amount
data term smoothness term
energy function
(for one pixel)
![Page 143: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/143.jpg)
SSD distance between windows
centered at I(x, y) and J(x+ d(x,y), y)
data term
![Page 144: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/144.jpg)
4-connected
neighborhood
8-connected
neighborhood
: set of neighboring pixels
SSD distance between windows
centered at I(x, y) and J(x+ d(x,y), y)
smoothness term
![Page 145: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/145.jpg)
“Potts model”
L1 distance
smoothness term
![Page 146: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/146.jpg)
Dynamic Programming
Can minimize this independently per scanline using dynamic programming (DP)
: minimum cost of solution such that d(x,y) = d
One possible solution…
![Page 147: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/147.jpg)
Match only Match & smoothness (via graph cut)
Ground Truth
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
![Page 148: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/148.jpg)
All of these cases remain difficult, what can we do?
textureless regions repeated patterns
specularities
![Page 149: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/149.jpg)
Structured light
![Page 150: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/150.jpg)
Use controlled (“structured”) light to make correspondences easier
Disparity between laser points on the same scanline in the images determines the 3-D coordinates of the laser point on object
![Page 151: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/151.jpg)
Use controlled (“structured”) light to make correspondences easier
![Page 152: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/152.jpg)
Structured light and two cameras
I J
laser
![Page 153: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/153.jpg)
I J
Structured light and one camera
Projector acts like “reverse” camera
![Page 154: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/154.jpg)
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Example: Laser scanner
![Page 155: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/155.jpg)
![Page 156: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/156.jpg)
![Page 157: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/157.jpg)
![Page 158: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/158.jpg)
![Page 159: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/159.jpg)
![Page 160: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/160.jpg)
cameras that see around corners
cameras that measure depth in real time
15-463/15-663/15-862 Computational PhotographyLearn about structured light and other cameras – and build some on your own!
http://graphics.cs.cmu.edu/courses/15-463/
cameras that take video at the speed of light
cameras that capture entire focal stacks
![Page 161: Introduction and course overvie16385/s18/lectures/lecture11.pdfCourse announcements • Homework 2 is due on February 23rd. ... Overview of today’s lecture • Plug: epipolar imaging](https://reader035.fdocuments.net/reader035/viewer/2022070320/60419bed97c0b340656617fe/html5/thumbnails/161.jpg)
Basic reading:• Szeliski textbook, Section 8.1 (not 8.1.1-8.1.3), Chapter 11, Section 12.2.• Hartley and Zisserman, Section 11.12.
References