Positional tracking for VR/AR using Stereo...
Transcript of Positional tracking for VR/AR using Stereo...
![Page 1: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/1.jpg)
1
Positional tracking for VR/AR using Stereo VisionEdwin AZZAM, CTO Stereolabs
![Page 2: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/2.jpg)
Outside-in Positional Tracking
IR LED constellation tracking system
– Oculus Rift
Lighthouse laser system – HTC Vive
©iFixit
![Page 3: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/3.jpg)
Inside-out Positional Tracking
▪ Use camera on HMD to track user position in space
▪ Eliminates the need for external tracking hardware
and allows freedom of movement
▪ Best for mobile VR/AR
▪ Many use cases outside the VR/AR industry!
![Page 4: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/4.jpg)
Our approach to depth and motion tracking
▪ 1x RGB camera
▪ 1x Fisheye camera (global shutter)
▪ 1x IR projector
▪ 1x IR camera
▪ 2x RGB camera (rolling shutter)
Tango tablet
ZED stereo camera
![Page 5: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/5.jpg)
Hard problems of stereo positional tracking
▪ Positional tracking with rolling shutter stereo is difficult
▪ Stereo calibration issues
▪ Depth reliability and computation
▪ Stereo odometry computation
▪ Position accuracy
▪ Position jitter
▪ Judder
▪ Drift
![Page 6: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/6.jpg)
What is ZED SDK?
▪ Native C++ SDK provided with ZED camera
▪ Software pipeline for depth, tracking and mapping from stereo
![Page 7: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/7.jpg)
ZED SDK pipeline modules
Stereo Images Self-calibration
Depth
Estimation
Visual
Odometry
Spatial
Mapping
Graphics
Rendering
CPU
GPU
![Page 8: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/8.jpg)
ZED SDK pipeline modules
Stereo Images Self-calibration
Depth
Estimation
Visual
Odometry
Spatial
Mapping
Graphics
Rendering
CPU
GPU
▪ Pose information is output at the frame rate of the camera
▪ sl::Pose is used to store camera position, timestamp and confidence
zed.getPosition(zed_pose, REFERENCE_FRAME_WORLD);
![Page 9: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/9.jpg)
Mobile positional tracking demo workflow
Jetson TX1(Auvidea J120)
Gear VR
ZED Stereo Camera
USB 3.0
Stereo images720p60
Wi-Fi
6DoF Pose60Hz
ZED SDK
Galaxy S7
![Page 10: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/10.jpg)
Linq - Mixed-Reality demo
![Page 11: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/11.jpg)
ZED SDK pipeline modules
Stereo Images Self-calibration
Depth
Estimation
Visual
Odometry
Spatial
Mapping
Graphics
Rendering
CPU
GPU
➢ Most problems solved, however drift and judder still need appropriate handling
![Page 12: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/12.jpg)
ZED SDK pipeline modules
Stereo Images Self-calibration
Depth
Estimation
Visual
Odometry
Spatial
Mapping
Graphics
Rendering
Spatial
Memory
CPU
GPU
➢ Most problems solved, however drift and judder still need appropriate handling
![Page 13: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/13.jpg)
▪ Small pose errors accumulate slowly over time
▪ Spatial memory module learns and recognizes its surroundings to
detect drift in absolute position
Mitigating drift with Spatial Memory
![Page 14: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/14.jpg)
▪ When a large error is detected, immediate drift
correction will cause a jump in the absolute pose
▪ Need to apply corrections without the user noticing ▪ Smooth error correction over multiple frames
▪ Correction rate still needs to be higher than drift accumulation!
▪ Smoothing strategy does not work for AR ▪ Misalignment between real and virtual too visible
▪ Jumps preferable
Drift correction strategies
![Page 15: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/15.jpg)
▪ Target tracking FPS not met
▪ 90Hz for desktop, 60Hz for mobile, 120Hz for next-gen?
▪ FPS instability
▪ 90fps = 11ms
▪ 1ms compute variability costs 7 frames: 83fps
▪ Traditional SLAM algorithms have +/- 30% variability
▪ Additional factors: OS scheduling, CPU/GPU allocated
resources, render vs compute...
▪ USB dropped frames
Judder reasons
![Page 16: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/16.jpg)
Mitigating judder with Pose prediction
No Pose prediction
Pose prediction
Render
▪ Pose prediction is different from timewarp
▪ Ensures a consistent rate of new position and orientation
tracking values
6DoF
Pose
Render
6DoF
Pose
Missed frame
Render did not finish on time
Render
6DoF
Pose
Render
Pose computation did not finish
Predict new pose
GPU
CPU
GPU
CPU
![Page 17: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/17.jpg)
Balancing vision vs rendering
▪ Maxwell: New GPU work can’t be started until compute and
graphics queues are finished
▪ Pascal: Dynamic balancing fills up the remaining time with work
from the other queues
▪ Thread and instruction level compute preemption can help
![Page 18: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/18.jpg)
Plugins and interfaces
(Q3 17)
(Q3 17)
![Page 20: Positional tracking for VR/AR using Stereo Visionon-demand.gputechconf.com/gtc/2017/presentation/s7801... · 2017-05-18 · Positional tracking for VR/AR using Stereo Vision Edwin](https://reader030.fdocuments.net/reader030/viewer/2022040511/5e5d9d6e1e894e07fd5816e8/html5/thumbnails/20.jpg)
▪ STEREOLABS provides depth and motion sensing
technology based on stereo vision
▪ 30+ employees based in San Francisco and Paris
▪ Launched the ZED in 2015, the world’s first stereo
camera for space and motion sensing
▪ Announced the Linq in 2017, the first mixed-reality HMD
prototype with stereo vision
Who we are