LSD-SLAM: Large-Scale Direct Monocular SLAM

download LSD-SLAM: Large-Scale Direct Monocular SLAM

of 31

  • date post

    11-Feb-2017
  • Category

    Documents

  • view

    239
  • download

    5

Embed Size (px)

Transcript of LSD-SLAM: Large-Scale Direct Monocular SLAM

  • 1

    LSD-SLAM: Large-Scale Direct Monocular SLAM

    Jakob Engel, Thomas Schps, Daniel CremersTechnical University Munich

    Monocular Video Camera Motion and Scene Geometry

    Computer Vision GroupTechnical University of Munich

    Jakob Engel

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 2

    Live Operation

    real-time operation on laptop (no GPU)LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 3

    (Some) Related Work

    DTAM: Dense Tracking and Mapping in Real-Time. Newcombe, Lovegrove, Davison; ICCV 11

    MonoSLAM: Real-time single camera SLAM. Davison, Reid, Molton, Stasse; PAMI 07

    Structure from Motion Causally Integrated Over Time. Chiuso, Favaro, Jin, Soatto; PAMI 02

    SVO: Fast Semi-Direct Monocular Visual Odometry. Forster, Pizzoli, Scaramuzza; ICRA 14

    Parallel Tracking and Mapping for Small AR Workspaces. Klein, Murray; ISMAR 07

    Scalable monocular SLAM. Eade, Drummond; CVPR 06

    Visual Odometry.Nistr, Naroditsky, Bergen; CVPR 04

    Scale Drift-Aware Large Scale Monocular SLAM. Strasdat, Montiel, Davison; RSS 10

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • Extract & MatchFeatures

    (SIFT / SURF / BRIEF /...)

    Input Images

    4

    Track:min. reprojection error

    (point distances)

    Map:est. feature-parameters

    (3D points / normals)

    abstract images to feature observations

    Input Images

    Track:min. photometric error

    (intensity difference)

    Map: est. per-pixel depth

    (semi-dense depth map)

    keep full image

    LSD-SLAM: whats new?Keypoint-Based Direct (LSD-SLAM)

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 5

    ...and why do that?

    can only use & reconstruct corners can use & reconstruct whole imageLSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • Depth Estimation

    Input Video

    6

    Current KF

    Tracking

    Overview

    Add to Map

    Map Optimization

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 7

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KFAdd to Map

    Map Optimization

  • 8

    Tracking

    KF image KF depth

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 8

    Tracking

    KF image KF depth back-warped new frame

    Camera Pose in

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 8

    Tracking

    KF image KF depth back-warped new frame

    Camera Pose in

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • ( forward-compositional Lucas-Kanade)

    8

    Tracking

    minimize using Gauss-Newton Algorithm

    KF image KF depth back-warped new frame

    Camera Pose in

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 9

    Tracking multi-resolution (track large motions)

    Huber norm instead of L2 (outliers & occlusions)

    statistical normalization (respect depth- and pixel-noise)

    single core timings:320x240: 5-10ms640x480: 20-30ms

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 10

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KFAdd to Map

    Map Optimization

  • 11

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Add to Map

    Map Optimization

    Depth Estimation

    Current KF

    Take KF?

    Createnew KF

    Refine KF

  • 12

    Depth Estimation

    pixelwise filtering (exploit video)small-baseline large baseline

    information selectiononly do stereo if sufficientinformation gain

    edge-preserving smoothing distance-based KF selection

    image inverse depth inverse depth variance

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    [Engel, Sturm, Cremers; ICCV 13]

  • 13

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KF

    Take KF?

    Createnew KF

    Refine KF

    Add to Map

    Map Optimization

  • 14

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KF

    Take KF?

    Createnew KF

    Refine KF

    Add to Map

    Map Optimization

  • 15

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KF

    Take KF?

    Createnew KF

    Refine KF

    Add to MapSim(3) alignmentto all nearby KFs

    Optional: FabMap for large loops

    Map OptimizationSim(3) pose-graph

  • 16

    Global Mapping

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • with (warped point)

    16

    Global Mapping

    Direct Tracking with scale (on Sim(3)):

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • with (warped point)

    16

    Global Mapping

    Direct Tracking with scale (on Sim(3)):

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • with (warped point)

    16

    Global Mapping

    Direct Tracking with scale (on Sim(3)):

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    + GN optimization + multi-resolution + Huber norm + statistical norm.

  • with (warped point)

    16

    Global Mapping

    Direct Tracking with scale (on Sim(3)):

    Optimize pose-graph on Sim(3)

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    + GN optimization + multi-resolution + Huber norm + statistical norm.

  • 17

    Global Mapping

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 18

    Overview

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Input Video640x480 @ 30Hz

    TrackingSE(3) alignment

    to current KF

    Depth Estimation

    Current KF

    Take KF?

    Createnew KF

    Refine KF

    Add to MapSim(3) alignmentto all nearby KFs

    Optional: FabMap for large loops

    Map OptimizationSim(3) pose-graph

  • 19

    Results

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    6 minutes, 640x480@50fps: 16.000 Tracked Frames, 800 Keyframes; 11.000 Constraints; 51 Million Points

  • 20

    Results

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    12 minutes, 640x480@50fps:36.000 Tracked Frames, 1.000 Keyframes; 18.000 Constraints; 100 Million Points

  • 21

    Results

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Semi-Dense Visual Odometry for AR on a Smartphone; T. Schps, J. Engel, D. Cremers; ISMAR 14.

  • 22

    Key Ingredients

    Direct Tracking

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 22

    Key Ingredients

    Direct Tracking

    Semi-Dense Stereo filter over many small-baseline frames strict information selection

    Pose-Graph on Sim(3)

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

  • 23

    LSD-SLAM

    LSD-SLAM: Large-Scale Direct Monocular SLAMEngel, Schps, Cremers

    Large-scale direct mono-SLAM Fully direct (no keypoints / features) Real-time even on CPU Open-source code & data-sets

    Slide Number 1Slide Number 2Slide Number 3Slide Number 4Slide Number 5Slide Number 6Slide Number 7Slide Number 8Slide Number 9Slide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Slide Number 18Slide Number 19Slide Number 20Slide Number 21Slide Number 22Slide Number 23Slide Number 24Slide Number 25Slide Number 26Slide Number 27Slide Number 28Slide Number 29Slide Number 30Slide Number 31