Parallel Tracking and Mapping Parallel Tracking and Mapping for Small AR Workspacesfor Small AR Workspaces

Vision Seminar

2008. 9. 4 (Thu)

Young Ki BaikComputer Vision Lab.

References

Parallel Tracking and Mapping for Small AR Parallel Tracking and Mapping for Small AR WorkspacesWorkspacesGeorg Klein, David Murray, ISMAR 2007

Visual OdometryVisual OdometryDavid Nister et. al. , CVPR 2004

Outline

What is AR?What is AR? Previous worksPrevious works Proposed methodsProposed methods DemoDemo ConclusionConclusion

What is AR?

AR : Augment Reality ugument reality(AR) uses the real scene as the background, and makes applications with putting 3D objects to the background. Since the cost of augment reality is not as expensive as full 3D

virtual reality(VR), AR has been very popular research topic.

Full 3D? (O) Expensive? (O)

Full 3D (X)Expensive? (X)

A

Demonstration

The Aim

AR with a hand-held camera

Visual Tracking provides registration

The Aim

AR with a hand-held camera

Visual Tracking provides registration

Track without prior model of world

The Aim

AR with a hand-held camera

Visual Tracking provides registration

Track without prior model of world

ChallengesSpeedAccuracyRobustness Interaction with real world

Existing attempts : SLAM

SLAM : Simultaneous Localization and Mapping can use many different types of sensor to acquire observation

data used in building the map such as laser rangefinders, sonar sensors and cameras.

Well-established in robotics (using a rich array of sensors)

Demonstrated with a single hand-held camera by Davison at 2003 (Mono-SLAM).

Mono-SLAM was applied to AR system at 2004.

Existing attempts : Model based tracking

Model-based tracking isMore robustMore accurateProposed by Lepetit et. al. at ISMAR 2003

Frame by Frame SLAM

Why? is SLAM fundamentally harder?

Time

One frame

Find features

Draw graphics

Update camera pose and entire map

Many DOF

Frame by Frame SLAM

SLAMUpdating entire map every frame is so expensive!!!Needs “sparse map of high-quality features” - A. Davison

Proposed approachUse dense map (of low quality features)Don’t update the map every frame : KeyframesSplit the tracking and mapping into two

threads

Parallel Tracking and Mapping

Proposed method- Split the tracking and mapping into two threads

Time

One frame

Find features

Draw graphics

Update camera pose only Simple & easy

Thread #2Mapping

Thread #1Tracking

Update map

Parallel Tracking and Mapping

Tracking thread:

• Responsible estimation of camera pose and rendering augmented graphics

• Must run at 30 Hz

• Make as robust and accurate as possible

Mapping thread:

• Responsible for providing the map

• Can take lots of time per key frame

• Make as rich and accurate as possible

Tracking thread

Overall flow

Pre-process framePre-process frame

Project pointsProject points

Measure pointsMeasure points

Update Camera Pose

Update Camera Pose

Project pointsProject points

Measure pointsMeasure points

Update Camera Pose

Update Camera Pose

Draw GraphicsDraw Graphics

Coarse stage Fine stage

MapMap

Pre-process frame

Make for pyramid levels

640x480 320x240 160x120 80x60

Pre-process frame

Make for pyramid levels

Detect Fast cornersE. Rosten et al (ECCV 2006)

640x480 320x240 160x120 80x60

Project Points

Use motion model to update camera poseConstant velocity model

Vt =(Pt – Pt-1)/∇t

Estimated current Pt+1

Previous pos Pt

Previous pos Pt-1

Pt+1=Pt+∇t’(Vt)

∇t

∇t’

Project Points

Choose subset to measure~ 50 biggest features for coarse stage1000 randomly selected for fine stage

640x480 320x240 160x120 80x60

~50~5010001000

Measure Points

Generate 8x8 matching template (warped from source key-frame:map)

Search a fixed radius around projected positionUse Zero-mean SSDOnly search at Fast corner points

Update caemra pose

6-DOF problemObtain by SFM (Three-point algorithm)

?

Dray graphics

What can we draw in an unknown scene?Assume single plane visible at startRun VR simulation on the plane

Mapping thread

Overall flow

Stereo InitializationStereo Initialization

Wait for new key frameWait for new key frame

Add new map pointsAdd new map points

Optimize mapOptimize map

Map maintenanceMap maintenance

TrackerTracker

Stereo Initialization

Use five-point-pose algorithm D. Nister et. al. 2006

Requires a pair of frames and feature correspondences

Provides initial map

User input required:Two clicks for two key-framesSmooth motion for feature correspondence

Wait for new key frame

Key frames are only added if :There is a sufficient baseline to the other key frameTracking quality is goodKey frame (4 level pyramid images and its corners)

When a key frame is added :The mapping thread stops whatever it is doingAll points in the map are measured in the key frameNew map points are found and added to the map

Add new map points

Want as many map points as possible

Check all maximal FAST corners in the key frame :Check scoreCheck if already in map

Epipolar search in a neighboring key frame

Triangulate matches and add to map

Repeat in four image pyramid levels

Optimize map

Use batch SFM method: Bundle Adjustment

Adjusts map point positions and key frame poses

Minimize reprojection error of all points in all keyframes (or use only last N key frames)

Map maintenance

When camera is not exploring, mapping thread has idle time

Data association in bundle adjustment is reversible

Re-attempt outlier measurements

Try measure new map features in all old key Try measure new map features in all old key frames frames

Comparison to EKF-SLAM

More Accurate More robust Faster tracking

<SLAM based AR Proposed AR

System and Results

EnvironmentDesktop PC (Intel Core 2 Duo 2.66 GHz)OS : LinuxLanguage : C++

Tracking speed

Total 19.2 ms

Key frame preparation 2.2 ms

Feature Projection 3.5 ms

Patch search 9.8 ms

Iterative pose update 3.7 ms

System and Results

Mapping scalability and speedPractical limit

• 150 key frames• 6000 points

Bundle adjustment timing

Key frames 2-49 50-99 100-149

Local Bundle Adjustment 170 ms 270 ms 440 ms

Global Bundle Adjustment 380 ms 1.7 s 6.9 s

Demonstration

Remaining problem

Outlier management Still brittle in some scenario

Repeated texturePassive stereo initialization

Occlusion problem

Relocation problem

Conclusion

ConclusionParallel tracking and mapping process are presented

using multi-thread.

ContributionVisual odometry system was well presented.Overcome computation by multi-thread

OpinionThe proposed algorithm can be applied to our research

• Navigation system• 3D tracking system

Q & A