Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3...
Transcript of Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3...
![Page 1: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/1.jpg)
Visual-Inertial SimultaneousLocalization and Mapping with
Multiple Cameras Speaker: Pak Hong, Chui
Supervisors: Dr. Vladyslav Usenko, Dr. Torsten Enßlin
![Page 2: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/2.jpg)
Outline1. Motivation2. Introduction to VIO and VI-Mapping3. Flow of proposed VI-SLAM4. Extend to multi cameras
2
![Page 3: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/3.jpg)
Advantages of using IMU
3
Accurate when
IMU measurements Small time interval
Triangulation (visual) Large time interval (Large baseline)
The advantages of combining visual and IMU measurement are,
1. combine the strengths of both sensors2. keep the map gravity aligned
![Page 4: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/4.jpg)
VI-Odometry and VI-Mapping
4
Feature type Loop detection
VI-Odometry(online) Optical Flow No, tend to drift over time
VI-Mapping(offine) Orb-feature yes
VI-SLAM ~ VI-Odometry + VI-Mapping
![Page 5: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/5.jpg)
Objectives
● try to build a VI-SLAM
● capable of working with multiple cameras
Use case:
for example, multi-player VR gaming etc...
5source: https://vrscout.com/news/cant-stop-playing-pool-nation-vr/
SLAM-1SLAM-2
Map
![Page 7: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/7.jpg)
How the mapper reuse information from VIO?Turn optical flow kp and imu measurement to non-linear
factors
● just need to store a few data for each frame in VIO● smaller optimization problem in mapping
7
from non-linear factors
non-linear factor extraction
optical flow keypoints
imu measurement
roll-pitch relative pose
![Page 8: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/8.jpg)
Structure of proposed VI-SLAM
8
![Page 9: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/9.jpg)
Structure of proposed VI-SLAM (simplified)
9
Tracker
Mapper
Loop closer
- track frame by Basalt-VIO
- build and update map bases on non-linear factor (faster)
- detect and close loop by using hash-bow
Credit: V. Usenko, N. Demmel, D. Schubert, J. Stckler, and D. Cremers, 2020
![Page 10: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/10.jpg)
Flow of proposed VI-SLAM
10
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 11: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/11.jpg)
Flow of proposed VI-SLAM
11
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2 imu measurement
Set initial pose of the frame
![Page 12: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/12.jpg)
Flow of proposed VI-SLAM
12
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
Build connection bwt the map and the new flame
![Page 13: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/13.jpg)
Track optical flow keypoints by re-projection
Time t-1 Time t13
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 14: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/14.jpg)
Flow of proposed VI-SLAM
14
Perform local bundle adjustment
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 15: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/15.jpg)
Flow of proposed VI-SLAM
15
Erase one keyframe to prevent the problem size from growing
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 16: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/16.jpg)
Flow of proposed VI-SLAM
16
Turn optical flow keypoints and imu measurement to relative pose and roll pitch
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 17: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/17.jpg)
Flow of proposed VI-SLAM
17
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 18: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/18.jpg)
Flow of proposed VI-SLAM
18
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
Build connection bwt the map and the new keyflame
![Page 19: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/19.jpg)
Scaled features and Non-maximal suppression
19
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. insert kf to loop closerM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
without non-maximal suppression with non-maximal suppression
![Page 20: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/20.jpg)
Flow of proposed VI-SLAM
20
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
Tell the loop closer to search if any loop
Detect loop plz!
![Page 21: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/21.jpg)
Flow of proposed VI-SLAM
21
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 22: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/22.jpg)
Flow of proposed VI-SLAM
22
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. detect loopM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
![Page 23: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/23.jpg)
Quantitative results
23
![Page 24: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/24.jpg)
Improve lost tracking
24
![Page 25: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/25.jpg)
Drift correction
25
![Page 26: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/26.jpg)
Structure of multi camera slam
26 source: https://vrscout.com/news/cant-stop-playing-pool-nation-vr/
SLAM-1SLAM-2
SLAM-1 SLAM-2 SLAM-3
Map
SLAM-1 SLAM-2 SLAM-3
Map Map Map
Map
example:
![Page 27: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/27.jpg)
Qualitative results
274 SLAM systems with 4 maps 4 SLAM systems with 1 map
SLAM-1 SLAM-2 SLAM-3
Map Map Map
SLAM-1 SLAM-2 SLAM-3
Map
SLAM-4
Map
SLAM-4
![Page 29: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/29.jpg)
End
Thank you !!!
29
![Page 30: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/30.jpg)
Demo
1 camera Basalt-SLAM (1 camera)
2 cameras Basalt-SLAM (2 cameras)
3 cameras Basalt-SLAM (3 cameras) 4x
4 cameras Basalt-SLAM (4 cameras) 4x
30
![Page 31: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/31.jpg)
Appendix: Merge loop
Search observations
31
Use Hash-bow to find potential loop
T1. input dataT2. set ini poseT3. search opt kps correspondenceT4. local BA within windowT5a. frame to kfT5b. select a kf to marginalizeT6. extract non-linear factorT7. insert marginalized kf to mapperT8. repeat step T1-T7
M1. receive a kf from trackerM2. search orb kps correspondenceM3. insert kf to loop closerM4. repeat step M1-M3
L1. receive a kf from mapperL2. detect and merge loopsL3. repeat step L1-L2
Pose graph optimization with error function:
![Page 32: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/32.jpg)
Appendix: Bundle adjustment
=
32
H_abs =
poses mappoints
mappoints
poses
=
.
.
.
mappoint 1
mappoint 4
2
3
41
![Page 33: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/33.jpg)
Appendix: Track optical flow keypoints by re-projection
33
![Page 34: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/34.jpg)
Appendix: Optical flow
34
![Page 35: Multiple Cameras Localization and Mapping with Visual ... · M3. detect loop M4. repeat step M1-M3 L1. receive a kf from mapper L2. detect and merge loops L3. repeat step L1-L2. Flow](https://reader034.fdocuments.net/reader034/viewer/2022051810/60172a85c10239600748a762/html5/thumbnails/35.jpg)
Appendix: Steps of loop detection
35