Lecture 10: Multi-view geometry - Artificial Lecture 10 - Fei-Fei Li 29 25-Oct-12 Dynamic...

Click here to load reader

  • date post

    03-Jun-2020
  • Category

    Documents

  • view

    0
  • download

    0

Embed Size (px)

Transcript of Lecture 10: Multi-view geometry - Artificial Lecture 10 - Fei-Fei Li 29 25-Oct-12 Dynamic...

  • Lecture 10 -

    Fei-Fei Li

    Lecture 10: Multi-view geometry

    Professor Fei-Fei Li Stanford Vision Lab

    25-Oct-12 1

  • Lecture 10 -

    Fei-Fei Li

    What we will learn today?

    • Review for stereo vision • Correspondence problem (Problem Set 2 (Q3)) • Active stereo vision systems • Structure from motion

    25-Oct-12 2

    Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 3

    Recovering structure from a single view Pinhole perspective projection

    C

    Ow

    P p

    Calibration rig

    Scene

    Camera K

    Why is it so difficult?

    Intrinsic ambiguity of the mapping from 3D to image (2D)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 4

    Two eyes help!

    O2 O1

    p2 p1

    ?

    This is called triangulation

    K2 =known K1 =known

    R, T

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 5

    Epipolar geometry

    • Epipolar Plane • Epipoles e1, e2

    • Epipolar Lines • Baseline

    O1 O2

    p2

    P

    p1

    e1 e2

    = intersections of baseline with image planes = projections of the other camera center = vanishing points of camera motion direction

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 6

    Triangulation

    O O’

    p p’

    P

    0pFpT =′

    F = Fundamental Matrix (Faugeras and Luong, 1992)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 7

    Why is F useful?

    - Suppose F is known - No additional information about the scene and camera is given - Given a point on left image, how can I find the corresponding point on right image?

    l’ = FT p p

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 8

    Estimating F

    O O’

    p p’

    P The Eight-Point Algorithm

    (Hartley, 1995)

    (Longuet-Higgins, 1981)

    Lsq. solution by SVD!

    • Rank 8 A non-zero solution exists (unique)

    • Homogeneous system

    • If N>8 F̂

    f

    1=f

    W 0=f

    W

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 9

    Rectification

    p’

    P

    p

    O O’ • Make two camera images “parallel”

    • Correspondence problem becomes easier

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 10

    Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30

  • Lecture 10 -

    Fei-Fei Li

    What we will learn today?

    • Review for stereo vision • Correspondence problem (Problem Set 2 (Q3)) • Active stereo vision systems • Structure from motion

    25-Oct-12 11

    Reading: [HZ] Chapters: 4, 9, 11 [FP] Chapters: 10

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 12

    Stereo-view geometry

    • Correspondence: Given a point in one image, how

    can I find the corresponding point p’ in another one?

    • Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

    • Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

    Previous lecture (#9)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 13

    Stereo-view geometry

    • Correspondence: Given a point in one image, how

    can I find the corresponding point p’ in another one?

    • Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

    • Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

    This lecture (#10)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 14

    Depth estimation Pinhole perspective projection

    Subgoals: - Solve the correspondence problem - Use corresponding observations to triangulate

    p’

    P

    p

    O1 O2

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 15

    Correspondence problem Pinhole perspective projection Pinhole perspective projection

    p’

    P

    p

    O1 O2

    Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 16

    Correspondence problem Pinhole perspective projection Pinhole perspective projection

    •A Cooperative Model (Marr and Poggio, 1976)

    •Correlation Methods (1970--)

    •Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)

    [FP] Chapters: 11

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 17

    Correlation Methods Pinhole perspective projection

    p

    • Pick up a window around p(u,v)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 18

    Correlation Methods Pinhole perspective projection

    • Pick up a window around p(u,v) • Build vector W • Slide the window along v line in image 2 and compute w’ • Keep sliding until w∙w’ is maximized.

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 19

    Correlation Methods Pinhole perspective projection

    Normalized Correlation; maximize: )ww)(ww( )ww)(ww( ′−′− ′−′−

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 20

    Correlation Methods Pinhole perspective projection Left Right

    scanline

    Norm. corr Credit slide S. Lazebnik

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 21

    Correlation Methods Pinhole perspective projection

    – Smaller window - More detail - More noise

    – Larger window

    - Smoother disparity maps - Less prone to noise

    Window size = 3 Window size = 20

    Credit slide S. Lazebnik

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 22

    Issues with Correlation Methods Pinhole perspective projection

    •Fore shortening effect

    •Occlusions

    O O’

    O O’

    It is desirable to have small B/z ratio!

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 24

    Issues with Correlation Methods Pinhole perspective projection

    •Homogeneous regions

    Hard to match pixels in these regions

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 25

    Issues with Correlation Methods Pinhole perspective projection

    •Repetitive patterns

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 26

    Results with window search Pinhole perspective projection Data

    Ground truth Window-based matching

    Credit slide S. Lazebnik

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 27

    Improving correspondence: Non-local constraints

    • Uniqueness – For any point in one image, there should be at most one

    matching point in the other image

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 28

    Improving correspondence: Non-local constraints

    • Uniqueness – For any point in one image, there should be at most one

    matching point in the other image

    • Ordering – Corresponding points should be in the same order in both

    views

    Courtesy J. Ponce

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 29

    Dynamic Programming Pinhole perspective projection (Baker and Binford, 1981)

    •Nodes = matched feature points (e.g., edge points). • Arcs = matched intervals along the epipolar lines. • Arc cost = discrepancy between intervals. Find the minimum-cost path going monotonically down and right from the top-left corner of the graph to its bottom-right corner.

    [Uses ordering constraint]

    Courtesy J. Ponce

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 30

    Dynamic Programming (Baker and Binford, 1981)

    (Ohta and Kanade, 1985)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 31

    Improving correspondence: Non-local constraints

    • Uniqueness – For any point in one image, there should be at most one

    matching point in the other image

    • Ordering – Corresponding points should be in the same order in both

    views

    • Smoothness Disparity is typically a smooth function of x (except in occluding boundaries)

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 32

    Stereo matching as energy minimization Pinhole perspective projection I1 I2 D

    • Energy functions of this form can be minimized using graph cuts

    Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 01

    W1(i ) W2(i+D(i )) D(i )

    )(),,( smooth21data DEDIIEE βα +=

    ( )∑ −= ji

    jDiDE ,neighbors

    smooth )()(ρ( )221data ))(()(∑ +−= i

    iDiWiWE

  • Lecture 10 -

    Fei-Fei Li 25-Oct-12 33

    Stereo matching as energy minimization Pinhole perspective projection