Epipolar lines
description
Transcript of Epipolar lines
![Page 1: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/1.jpg)
Epipolar lines
epipolar linesepipolar lines
BaselineO O’
epipolar plane
𝑝 ′𝑇 𝐸𝑝=0
![Page 2: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/2.jpg)
![Page 3: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/3.jpg)
Rectification
• Rectification: rotation and scaling of each camera’s coordinate frame to make the epipolar lines horizontal and equi-height,by bringing the two image planes to be parallel to the baseline
• Rectification is achieved by applying homography to each of the two images
![Page 4: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/4.jpg)
Rectification
BaselineO O’
𝑞 ′𝑇𝐻 𝑙−𝑇 𝐸𝐻𝑟
−1𝑞=0
𝐻 𝑙 𝐻𝑟
![Page 5: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/5.jpg)
Cyclopean coordinates• In a rectified stereo rig with baseline of length ,
we place the origin at the midpoint between the camera centers.
• a point is projected to:– Left image: , – Right image: ,
• Cyclopean coordinates:
![Page 6: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/6.jpg)
Disparity
• Disparity is inverse proportional to depth• Constant disparity constant depth• Larger baseline, more stable reconstruction of depth
(but more occlusions, correspondence is harder)
(Note that disparity is defined in a rectified rig in a cyclopean coordinate frame)
![Page 7: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/7.jpg)
Random dot stereogram
• Depth can be perceived from a random dot pair of images (Julesz)
• Stereo perception is based solely on local information (low level)
![Page 8: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/8.jpg)
Moving random dots
![Page 9: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/9.jpg)
Compared elements
• Pixel intensities• Pixel color• Small window (e.g. or ), often using
normalized correlation to offset gain• Features and edges (less common)• Mini segments
![Page 10: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/10.jpg)
Dynamic programming
• Each pair of epipolar lines is compared independently
• Local cost, sum of unary term and binary term– Unary term: cost of a single match– Binary term: cost of change of disparity (occlusion)
• Analogous to string matching (‘diff’ in Unix)
![Page 11: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/11.jpg)
String matching• Swing → String
S t r i n g
S w i n g
Start
End
![Page 12: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/12.jpg)
String matching• Cost: #substitutions + #insertions + #deletions
S t r i n g
S w i n g
![Page 13: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/13.jpg)
![Page 14: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/14.jpg)
Dynamic Programming
• Shortest path in a grid• Diagonals: constant disparity• Moving along the diagonal – pay unary cost
(cost of pixel match)• Move sideways – pay binary cost, i.e. disparity
change (occlusion, right or left)• Cost prefers fronto-parallel planes. Penalty is
paid for tilted planes
![Page 15: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/15.jpg)
Dynamic Programming
Start
, Complexity?
![Page 16: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/16.jpg)
Probability interpretation: Viterbi algorithm
• Markov chain
• States: discrete set of disparity
• Log probabilities: product sum
![Page 17: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/17.jpg)
Probability interpretation: Viterbi algorithm
• Markov chain
• States: discrete set of disparity
• Maximum likelihood: minimize sum of negative logs• Viterbi algorithm: equivalent to shortest path
![Page 18: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/18.jpg)
Dynamic Programming: Pros and Cons
• Advantages:– Simple, efficient– Achieves global optimum– Generally works well
• Disadvantages:
![Page 19: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/19.jpg)
Dynamic Programming: Pros and Cons
• Advantages:– Simple, efficient– Achieves global optimum– Generally works well
• Disadvantages:– Works separately on each epipolar line,
does not enforce smoothness across epipolars– Prefers fronto-parallel planes– Too local? (considers only immediate neighbors)
![Page 20: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/20.jpg)
Markov Random Field
• Graph In our case: graph isa 4-connected gridrepresenting one image
• States: disparity
• Minimize energy of the form
• Interpreted as negative log probabilities
![Page 21: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/21.jpg)
Iterated Conditional Modes (ICM)
• Initialize states (= disparities) for every pixel• Update repeatedly each pixel by the most likely
disparity given the values assigned to its neighbors:
• Markov blanket: the state of a pixel only depends on the states of its immediate neighbors
• Similar to Gauss-Seidel iterations• Slow convergence to (often bad) local minimum
![Page 22: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/22.jpg)
Graph cuts: expansion moves
• Assume is non-negative and is metric:
• We can apply more semi-global moves using minimal s-t cuts
• Converges faster to a better (local) minimum
![Page 23: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/23.jpg)
α-Expansion
• In any one round, expansion move allows each pixel to either – change its state to α, or– maintain its previous state
Each round is implemented via max flow/min cut• One iteration: apply expansion moves
sequentially with all possible disparity values• Repeat till convergence
![Page 24: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/24.jpg)
α-Expansion• Every round achieves a globally optimal solution over
one expansion move• Energy decreases (non-increasing) monotonically
between rounds• At convergence energy is optimal with respect to all
expansion moves, and within a scale factor from the global optimum:
where
![Page 25: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/25.jpg)
α-Expansion (1D example)
![Page 26: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/26.jpg)
α-Expansion (1D example)
𝛼
𝛼
![Page 27: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/27.jpg)
α-Expansion (1D example)
𝛼
𝛼
![Page 28: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/28.jpg)
α-Expansion (1D example)
𝐷𝑝(𝛼) 𝐷𝑞 (𝛼)
𝛼
𝛼
𝑉 𝑝𝑞 (𝛼 ,𝛼 )=0
![Page 29: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/29.jpg)
α-Expansion (1D example)
𝛼
𝛼 𝐷𝑝(𝑑𝑝) 𝐷𝑞 (𝑑𝑞)
But what about?
![Page 30: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/30.jpg)
α-Expansion (1D example)
𝛼
𝛼 𝐷𝑝(𝑑𝑝) 𝐷𝑞 (𝑑𝑞)
𝑉 𝑝𝑞(𝑑𝑝 ,𝑑𝑞)
![Page 31: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/31.jpg)
α-Expansion (1D example)
𝛼
𝛼 𝐷𝑝(𝑑𝑝)
𝑉 𝑝𝑞(𝑑𝑝 ,𝛼)𝐷𝑞 (𝛼)
![Page 32: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/32.jpg)
α-Expansion (1D example)
𝛼
𝛼 𝐷𝑞 (𝑑𝑞)
𝑉 𝑝𝑞(𝛼 ,𝑑𝑞)𝐷𝑝(𝛼)
![Page 33: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/33.jpg)
α-Expansion (1D example)
𝛼
𝛼
𝑉 𝑝𝑞(𝛼 ,𝑑𝑞)𝑉 𝑝𝑞(𝑑𝑝 ,𝛼)
𝑉 𝑝𝑞(𝑑𝑝 ,𝑑𝑞)
Such a cut cannot be obtained due to triangle inequality:
![Page 34: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/34.jpg)
Common Metrics
• Potts model:
• Truncated :
• Truncated squared difference is not a metric
![Page 35: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/35.jpg)
Reconstruction with graph-cuts
Original Result Ground truth
![Page 36: Epipolar lines](https://reader036.fdocuments.net/reader036/viewer/2022062323/5681610a550346895dd057d3/html5/thumbnails/36.jpg)
A different application: detect skyline
• Input: one image, oriented with sky above• Objective: find the skyline in the image• Graph: grid• Two states: sky, ground• Unary (data) term:
– State = sky, low if blue, otherwise high– State = ground, high if blue, otherwise low
• Binary term for vertical connections:– If the state of a node is sky, the node above should also be sky (set to infinity if
not)– If the state of a node is ground, the node below should also be ground
• Solve with expansion move. This is a binary (two state) problem, and so graph cut can find the global optimum in one expansion move