Parallel Tracking and Mapping Parallel Tracking and Mapping for Small AR Workspacesfor Small AR Workspaces
Vision Seminar
2008. 9. 4 (Thu)
Young Ki BaikComputer Vision Lab.
References
Parallel Tracking and Mapping for Small AR Parallel Tracking and Mapping for Small AR WorkspacesWorkspacesGeorg Klein, David Murray, ISMAR 2007
Visual OdometryVisual OdometryDavid Nister et. al. , CVPR 2004
Outline
What is AR?What is AR? Previous worksPrevious works Proposed methodsProposed methods DemoDemo ConclusionConclusion
What is AR?
AR : Augment Reality ugument reality(AR) uses the real scene as the background, and makes applications with putting 3D objects to the background. Since the cost of augment reality is not as expensive as full 3D
virtual reality(VR), AR has been very popular research topic.
Full 3D? (O) Expensive? (O)
Full 3D (X)Expensive? (X)
A
Demonstration
The Aim
AR with a hand-held camera
Visual Tracking provides registration
The Aim
AR with a hand-held camera
Visual Tracking provides registration
Track without prior model of world
The Aim
AR with a hand-held camera
Visual Tracking provides registration
Track without prior model of world
ChallengesSpeedAccuracyRobustness Interaction with real world
Existing attempts : SLAM
SLAM : Simultaneous Localization and Mapping can use many different types of sensor to acquire observation
data used in building the map such as laser rangefinders, sonar sensors and cameras.
Well-established in robotics (using a rich array of sensors)
Demonstrated with a single hand-held camera by Davison at 2003 (Mono-SLAM).
Mono-SLAM was applied to AR system at 2004.
Existing attempts : Model based tracking
Model-based tracking isMore robustMore accurateProposed by Lepetit et. al. at ISMAR 2003
Frame by Frame SLAM
Why? is SLAM fundamentally harder?
Time
One frame
Find features
Draw graphics
Update camera pose and entire map
Many DOF
Frame by Frame SLAM
SLAMUpdating entire map every frame is so expensive!!!Needs “sparse map of high-quality features” - A. Davison
Proposed approachUse dense map (of low quality features)Don’t update the map every frame : KeyframesSplit the tracking and mapping into two
threads
Parallel Tracking and Mapping
Proposed method- Split the tracking and mapping into two threads
Time
One frame
Find features
Draw graphics
Update camera pose only Simple & easy
Thread #2Mapping
Thread #1Tracking
Update map
Parallel Tracking and Mapping
Tracking thread:
• Responsible estimation of camera pose and rendering augmented graphics
• Must run at 30 Hz
• Make as robust and accurate as possible
Mapping thread:
• Responsible for providing the map
• Can take lots of time per key frame
• Make as rich and accurate as possible
Tracking thread
Overall flow
Pre-process framePre-process frame
Project pointsProject points
Measure pointsMeasure points
Update Camera Pose
Update Camera Pose
Project pointsProject points
Measure pointsMeasure points
Update Camera Pose
Update Camera Pose
Draw GraphicsDraw Graphics
Coarse stage Fine stage
MapMap
Pre-process frame
Make for pyramid levels
640x480 320x240 160x120 80x60
Pre-process frame
Make for pyramid levels
Detect Fast cornersE. Rosten et al (ECCV 2006)
640x480 320x240 160x120 80x60
Project Points
Use motion model to update camera poseConstant velocity model
Vt =(Pt – Pt-1)/∇t
Estimated current Pt+1
Previous pos Pt
Previous pos Pt-1
Pt+1=Pt+∇t’(Vt)
∇t
∇t’
Project Points
Choose subset to measure~ 50 biggest features for coarse stage1000 randomly selected for fine stage
640x480 320x240 160x120 80x60
~50~5010001000
Measure Points
Generate 8x8 matching template (warped from source key-frame:map)
Search a fixed radius around projected positionUse Zero-mean SSDOnly search at Fast corner points
Update caemra pose
6-DOF problemObtain by SFM (Three-point algorithm)
?
Dray graphics
What can we draw in an unknown scene?Assume single plane visible at startRun VR simulation on the plane
Mapping thread
Overall flow
Stereo InitializationStereo Initialization
Wait for new key frameWait for new key frame
Add new map pointsAdd new map points
Optimize mapOptimize map
Map maintenanceMap maintenance
TrackerTracker
Stereo Initialization
Use five-point-pose algorithm D. Nister et. al. 2006
Requires a pair of frames and feature correspondences
Provides initial map
User input required:Two clicks for two key-framesSmooth motion for feature correspondence
Wait for new key frame
Key frames are only added if :There is a sufficient baseline to the other key frameTracking quality is goodKey frame (4 level pyramid images and its corners)
When a key frame is added :The mapping thread stops whatever it is doingAll points in the map are measured in the key frameNew map points are found and added to the map
Add new map points
Want as many map points as possible
Check all maximal FAST corners in the key frame :Check scoreCheck if already in map
Epipolar search in a neighboring key frame
Triangulate matches and add to map
Repeat in four image pyramid levels
Optimize map
Use batch SFM method: Bundle Adjustment
Adjusts map point positions and key frame poses
Minimize reprojection error of all points in all keyframes (or use only last N key frames)
Map maintenance
When camera is not exploring, mapping thread has idle time
Data association in bundle adjustment is reversible
Re-attempt outlier measurements
Try measure new map features in all old key Try measure new map features in all old key frames frames
Comparison to EKF-SLAM
More Accurate More robust Faster tracking
<SLAM based AR Proposed AR
System and Results
EnvironmentDesktop PC (Intel Core 2 Duo 2.66 GHz)OS : LinuxLanguage : C++
Tracking speed
Total 19.2 ms
Key frame preparation 2.2 ms
Feature Projection 3.5 ms
Patch search 9.8 ms
Iterative pose update 3.7 ms
System and Results
Mapping scalability and speedPractical limit
• 150 key frames• 6000 points
Bundle adjustment timing
Key frames 2-49 50-99 100-149
Local Bundle Adjustment 170 ms 270 ms 440 ms
Global Bundle Adjustment 380 ms 1.7 s 6.9 s
Demonstration
Remaining problem
Outlier management Still brittle in some scenario
Repeated texturePassive stereo initialization
Occlusion problem
Relocation problem
Conclusion
ConclusionParallel tracking and mapping process are presented
using multi-thread.
ContributionVisual odometry system was well presented.Overcome computation by multi-thread
OpinionThe proposed algorithm can be applied to our research
• Navigation system• 3D tracking system
Q & A
Top Related