Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun...
-
date post
23-Jan-2016 -
Category
Documents
-
view
220 -
download
0
Transcript of Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun...
![Page 1: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/1.jpg)
Stanford CS223B Computer Vision, Winter 2007
Lecture 6 Advanced Stereo
Professors Sebastian Thrun and Jana Košecká
CAs: Vaibhav Vaish and David Stavens
StereoStereo
![Page 2: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/2.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Summary Stereo Epipolar Geometry Fundamental/Essential Matrix
plp
r
P
Ol Orel er
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles
0lTr Epp
![Page 3: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/3.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence: Where to search?
Search window?
![Page 4: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/4.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields
![Page 5: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/5.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Rectification Problem: Epipolar lines not parallel to scan lines
plp
r
P
Ol Orel er
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles
![Page 6: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/6.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Rectification Problem: Epipolar lines not parallel to scan lines
plp
r
P
Ol Or
Pl Pr
Epipolar Plane
Epipolar Lines
Epipoles at ininity
Rectified Images
![Page 7: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/7.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Epipolar Rectified Stereo Images
Epipolar line
![Page 8: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/8.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Epipolar Rectified Images
Source: A. Fusiello, Verona, 2000]
![Page 9: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/9.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Example Rectification
![Page 10: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/10.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Final Step: Image Normalization
Even when the cameras are identical models, there can be differences in gain and sensitivity.
The cameras do not see exactly the same surfaces, so their overall light levels can differ.
For these reasons and more, it is a good idea to normalize the pixels in each window:
pixel Normalized ),(
),(ˆ
magnitude Window )],([
pixel Average ),(
),(
),(),(
2
),(
),(),(),(
1
yxW
yxWvuyxW
yxWvuyxW
m
mm
m
m
II
IyxIyxI
vuII
vuII
![Page 11: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/11.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields
![Page 12: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/12.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence
1P1Oy
x
z
f
2Oy
x
z
1.lp
1,rp
1P
Phantom points
![Page 13: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/13.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence via Correlation
Rectified images
Left Right
scanline
SSD error
disparity
(Same as max-correlation / max-cosine for normalized image patch)
![Page 14: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/14.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Images as Vectors
Left Right
LwRw
Each window is a vectorin an m2 dimensionalvector space.Normalization makesthem unit length.
![Page 15: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/15.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Image Metrics
Lw)(dwR
2
),(),(
2SSD
)(
)],(ˆ),(ˆ[)(
dww
vduIvuIdC
RL
yxWvuRL
m
(Normalized) Sum of Squared Differences
Normalized Correlation
cos)(
),(ˆ),(ˆ)(),(),(
NC
dww
vduIvuIdC
RL
yxWvuRL
m
)(maxarg)(minarg2* dwwdwwd RLdRLd
![Page 16: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/16.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence Using Correlation
Left Disparity Map
Images courtesy of Point Grey Research
![Page 17: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/17.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
LEFT IMAGE
corner line
structure
Correspondence By Features
![Page 18: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/18.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence By Features
RIGHT IMAGE
corner line
structure
Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum
![Page 19: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/19.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Correspondences
… …Left scan line Right scan line
![Page 20: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/20.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Correspondences
… …Left scanline Right scanline
Match
Match
MatchOcclusion Disocclusion
![Page 21: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/21.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Search Over Correspondences
Three cases:–Sequential – cost of match–Occluded – cost of no match–Disoccluded – cost of no match
Left scanline
Right scanline
Occluded Pixels
Disoccluded Pixels
![Page 22: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/22.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.
Occluded Pixels
Left scanline
Dis-occluded Pixels
Right scanline
Terminal
Stereo Matching with Dynamic Programming
![Page 23: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/23.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
DP Algorithm:
V[0,0] = 0
V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }
Occluded Pixels
Left scanline
Dis-occluded Pixels
Right scanline
Terminal
Stereo Matching with Dynamic Programming
![Page 24: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/24.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Occluded Pixels
Left scanline
Dis-occluded Pixels
Right scanline
Terminal
Stereo Matching with Dynamic Programming
DP Algorithm:
V[0,0] = 0
V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }
![Page 25: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/25.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Find Stereo Alignment
D=[X,Y]repeat until D=[1,1] add D to alignment D = d[D]end
Occluded Pixels
Left scanline
Dis-occluded Pixels
Right scanline
Terminal
Stereo Matching with Dynamic Programming
![Page 26: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/26.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Commercial-Grade Stereo Tyzx, a leading stereo camera manufacturer (here strapped on our DARPA Grand Challenge vehicle)
Disparity map
![Page 27: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/27.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Dense Stereo Matching: Examples
View extrapolation results
input depth image novel view [Matthies,Szeliski,Kanade’88]
![Page 28: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/28.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Dense Stereo Matching Some other view extrapolation results
input depth image novel view
![Page 29: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/29.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Dense Stereo Matching Compute certainty map from correlations
input depth map certainty map
![Page 30: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/30.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
DP for Correspondence Does this always work? When would it fail?
– Failure Example 1– Failure Example 2– Failure Example 3– Failure Example 4
![Page 31: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/31.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence Problem 1 Ambiguities
![Page 32: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/32.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence Problem 2 Multiple occluding objects
Figure fromForsyth & Ponce
![Page 33: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/33.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence Problem 3 Correspondence fail for smooth surfaces (edge =
occlusion boundary, poorly localized)
There is currently no good solution to this correspondence problem
![Page 34: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/34.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Correspondence Problem 4
Regions without texture Highly Specular surfaces Translucent objects
![Page 35: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/35.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Some Stereo Results
Side view
Top view
![Page 36: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/36.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
More Stereo Results
Side view
![Page 37: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/37.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
A True Challenge!
http://www.well.com/user/jimg/stereo/stereo_list.html
![Page 38: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/38.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields
![Page 39: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/39.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
How can We Improve Stereo?
Space-time stereo scanneruses unstructured light to aidin correspondence
Result: Dense 3D mesh (noisy)
![Page 40: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/40.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Active Stereo: Adding Texture to Scene
By James Davis, Honda Research,
Now UCSC
![Page 41: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/41.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
rect
ified
Active Stereo (Structured Light)
![Page 42: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/42.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Structured Light: 3-D Result
3D Snapshot
By James Davis, Honda Research
![Page 43: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/43.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
![Page 44: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/44.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
![Page 45: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/45.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
![Page 46: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/46.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
![Page 47: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/47.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Time of Flight Sensor: Shutter
http://www.3dvsystems.com
![Page 48: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/48.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Scanning Laser Range Finders
![Page 49: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/49.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Scanning Laser Results
![Page 50: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/50.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Scanning Laser Results
![Page 51: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/51.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields
![Page 52: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/52.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo Assign pixel to different “layers” (objects, sprites)
![Page 53: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/53.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo
Track each layer from frame to frame, compute plane eqn. and composite mosaic
Re-compute pixel assignment by comparing original images to sprites
![Page 54: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/54.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo Re-synthesize original or novel images from
collection of sprites
![Page 55: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/55.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo Advantages:
– can represent occluded regions– can represent transparent and border (mixed) pixels
(sprites have alpha value per pixel)– works on texture-less interior regions
Limitations:– fails for high depth-complexity scenes
![Page 56: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/56.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Fitting Planar Surfaces (with EM)
******
![Page 57: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/57.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Expectation Maximization 3D Model:
},,,{ 21 J
3, jjj Planar surface in 3D
jijij zz ),dist(
Distance point-surfacesurface
surface normal a
y
x
z
displacement b
![Page 58: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/58.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Mixture Measurement Model Case 1: Measurement zi caused by plane j
2
2)(
2
1
22
1)|(
jij z
ji ezp
2
2max
2ln
2
1
2max
*2
11)|(
z
i ez
zp
§ Case 2: Measurement zi caused by something else
![Page 59: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/59.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Measurement Model with Correspondences
J
j
jijj
zc
zc
Ji eccczp 12
2
2
2max
*
)(
2ln
2
1
2*12
1),,,,|(
correspondence variables C:
}
J
jj
j
cc
cc
1*
*
1
}1,0{,
1
)(
2ln
2
1
2
12
2
2
2max
*
2
1),|(
i
zc
zc
J
j
jijiji
eCZp
![Page 60: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/60.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Expected Log-Likelihood Function
1
)(
2ln
2
1
2
12
2
2
2max
*
2
1),|(
i
zc
zc
J
j
jijiji
eCZp
i
J
j
jijij
ic
zcE
zcE
J
CZpE
12
2
2
2max
*
2
)(][
2
1
2ln][
2
1
2)1(
1ln
)|,(ln
…after some simple math
mapping with known data association
probabilistic data association
![Page 61: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/61.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
The EM Algorithm
E-step: given plane params, compute
M-step: given expectations, compute
i
J
j
jijijc
zcECZpE
12
2)(][const)|,(ln
][ ijcE
},{ jja
![Page 62: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/62.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Choosing the “Right” Number of Planes: AIC
J=2 J=3 J=5J=0 J=1 J=4
increased data likelihood
increased prior probability
)(log)|(log)|(log JpJdpconstdJp
![Page 63: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/63.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Determining Number of Surfaces
J =1
First model component
*
*
J =1
E-Step
*
*
J =3
Add model components
J =3
E-Step
J =3
M-step
J =1
Prune model
J =3
Add model components
J =3
E/M Steps
*J =2
Prune model
![Page 64: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/64.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo Resulting sprite collection
![Page 65: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/65.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Layered Stereo
Estimated depth map
![Page 66: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/66.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Example (here with laser range finder)
![Page 67: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/67.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Example (here with laser range finder)
Another Example
![Page 68: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/68.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields
![Page 69: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/69.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Motivation and Goals
James Diebel
![Page 70: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/70.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Motivation and Goals
James Diebel
![Page 71: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/71.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Network of Constraints (Markov Random Field)
Vertex Node
Edge Node
Face Node
Vertex Node
Edge Node
Face Node
Vertex Node
Edge Node
Face Node
DirectionsDirectionsDirections
James Diebel
![Page 72: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/72.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
MRF Approach to Smoothing Potential function: contains a sensor-model term
and a surface prior
The edge potential is important! Minimize by conjugate gradient
– Optimize systems with tens of thousands of parameters in just a couple seconds
– Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes)
j
ji
iiiT
ii nnxxxx 2100 1
Diebel/Thrun, 2006
![Page 73: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/73.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Possible Edge Potential Functions
![Page 74: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/74.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
![Page 75: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/75.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
![Page 76: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/76.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
![Page 77: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/77.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Results: Smoothing
James Diebel
![Page 78: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/78.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Movies…
Movies in Windows Media Player
![Page 79: Stanford CS223B Computer Vision, Winter 2007 Lecture 6 Advanced Stereo Professors Sebastian Thrun and Jana Košecká CAs: Vaibhav Vaish and David Stavens.](https://reader035.fdocuments.net/reader035/viewer/2022062500/56649d405503460f94a1b081/html5/thumbnails/79.jpg)
Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007
Summary Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields