Efficient Perception of Proteins and Nucleic Acids from Atomic Connectivity
Efficient 360-degree Visual Perception -...
Transcript of Efficient 360-degree Visual Perception -...
![Page 1: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/1.jpg)
Efficient 360-degree Visual Perception高效360度視覺感知
Min Sun
Assistant Professor
National Tsing Hua University1
![Page 2: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/2.jpg)
The Power of 360 Camera
2
![Page 3: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/3.jpg)
The Power of 360 Camera
3
![Page 4: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/4.jpg)
The Market of 360 Cameras is Booming
4
![Page 5: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/5.jpg)
Applications: Virtual Reality
5
![Page 6: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/6.jpg)
Applications: Autonomous Systems
Indoor Robot Self-Driving Car Drone
6
![Page 7: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/7.jpg)
Cube Padding for Unsupervised Saliency Prediction in 360 Videos立方填補於360影片之非監督式學習
Hsien-Tzu Cheng, Chun-Hung Chao, Jin-Dong Dong, Hao-Kai Wen, Tyng-Luh Liu, Min Sun
7
![Page 8: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/8.jpg)
Motivation
8
![Page 9: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/9.jpg)
Motivation
9
![Page 10: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/10.jpg)
Our Goal – Predict Salient Regions
• Self-supervised training
• Computational efficiency10
![Page 11: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/11.jpg)
Our Goal – Automatic View Selection
11
![Page 12: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/12.jpg)
Overview
Challenges:• Image distortion• Image Boundary CNN on the Cube
Robust saliency• on top and bottom• across boundary
12
![Page 13: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/13.jpg)
Outline
• Our method
• Dataset
• Result
• Conclusion
13
![Page 14: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/14.jpg)
Cube Padding
14
![Page 15: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/15.jpg)
Feature Visualization
15
![Page 16: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/16.jpg)
Model
16
![Page 17: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/17.jpg)
Model – Static Model
17
![Page 18: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/18.jpg)
Learning Deep Features for Discriminative Localization, Zhou et al. CVPR16
“achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation”
specific class
Model – Static Model
18
![Page 19: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/19.jpg)
VGG-16
ResNet-50
Learning Deep Features for Discriminative Localization, Zhou et al. CVPR16
“achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation”
specific class
Model – Static Model
19
![Page 20: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/20.jpg)
VGG-16
ResNet-50
Learning Deep Features for Discriminative Localization, Zhou et al. CVPR16
“achieve 37.1% top-5 error for object localization on ILSVRC 2014 without training on any bounding box annotation”
specific class
avg.
po
ol
CA
M-c
on
v
1000 x [fc channel]
B x H x W x [fc channel]
B x H x W x 1000
B x H x W x 1
(Maximum)
(Weighted)
Model – Static Model
20
![Page 21: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/21.jpg)
Model – Temporal Model
21
![Page 22: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/22.jpg)
Model – Loss function design
𝑶 (static saliency) 𝒎 (motion)
22
![Page 23: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/23.jpg)
Model – Loss function design 𝑂 (static saliency) 𝑚 (motion)
23
![Page 24: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/24.jpg)
•Warp( ),_ԡ ԡ 2
𝑂 (static saliency) 𝑚 (motion)Model – Loss function design
24
![Page 25: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/25.jpg)
•
•
Warp( ),_ԡ ԡ 2
_ԡ ԡ 2
𝑂 (static saliency) 𝑚 (motion)Model – Loss function design
25
![Page 26: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/26.jpg)
•
•
•
Warp( ),_ԡ ԡ 2
_ԡ ԡ 2
Mask( ),ԡ ԡ 2
𝑂 (static saliency) 𝑚 (motion)Model – Loss function design
26
![Page 27: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/27.jpg)
Outline
• Our method
• Dataset
• Result
• Conclusion & Future work
27
![Page 28: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/28.jpg)
DatasetTrain/test split: 60/25 clips, totally 50k frames
Totally 30 annotators, 80 trajectories per video
28
![Page 29: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/29.jpg)
Outline
• Our method
• Dataset
• Result
• Conclusion
29
![Page 31: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/31.jpg)
Result
31
![Page 32: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/32.jpg)
Resultfps
fps
fps
fps
32
![Page 33: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/33.jpg)
Result – Human study
33
![Page 36: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/36.jpg)
Outline
• Our method
• Dataset
• Result
• Conclusion
36
![Page 37: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/37.jpg)
Conclusion
• Simple and effective Cube Padding (CP) technique
• Novel spatial-temporal network
• Unsupervised trained with designed temporal loss functions
• Wild-360 dataset with videos and saliency heatmap annotations
• Outperforms state-of-the-art methods in both speed and quality
37
![Page 38: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/38.jpg)
Self-Supervised Learning of Depth from 360◦ Videos
自主式學習360影片之深度預測
Under Submission
38
![Page 39: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/39.jpg)
Our Goal – 360 Depth Prediction
39
![Page 40: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/40.jpg)
Applications: Autonomous Systems
Indoor Robot Self-Driving Car Drone
40
![Page 41: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/41.jpg)
𝑸𝟏
𝑰𝟏
𝑰𝟐
𝑫𝟏
𝑷𝟏𝑷𝟐
R, T
𝑷𝟏
𝑷𝟐
DNet
PNet
Our Model
Zhou et al., Unsupervised Learning of Depth and Ego-Motion from Video, CVPR 2017
I: EquirectangularI: CubeD: DepthP: Camera motionQ: Point Cloud
41
![Page 42: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/42.jpg)
𝑡1
Frame Inverse Depth
𝑡2
Frame Inverse Depth Frame Inverse Depth
𝑡1
𝑡2
Dataset – PanoSUNCG
42
![Page 43: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/43.jpg)
Our Results
https://drive.google.com/open?id=1BhTwMxtPyoNcny-qyCqJN-FgvqMuCPFA43
![Page 44: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/44.jpg)
Quantitative Results – Depth
44
![Page 45: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/45.jpg)
Efficiency – Speedup Ratio
45
![Page 46: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/46.jpg)
Frame Our prediction Frame Our prediction
Qualitative Results – Real-world Videos
46
![Page 47: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/47.jpg)
Conclusion
360 saliency and depth prediction systems• Cube Padding (CP) technique is Simple and effective
• Self-training is important for both systems to scale-up
• Outperform other state-of-the-art methods in both speed and quality
47
![Page 48: Efficient 360-degree Visual Perception - NVIDIAon-demand.gputechconf.com/gtc-taiwan/2018/pdf/1-4... · Efficient 360-degree Visual Perception ... •Wild-360 dataset with videos and](https://reader034.fdocuments.net/reader034/viewer/2022042810/5f9b750b50f8ef30955b7964/html5/thumbnails/48.jpg)
ThanksQ & A
48