Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs ›...
Transcript of Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs ›...
![Page 1: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/1.jpg)
![Page 2: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/2.jpg)
•
•
•
•
•
•
![Page 3: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/3.jpg)
Image source: octomap.github.io
Image source: pirobot.org/blog/0015/
![Page 4: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/4.jpg)
• Map from first-person images to actions
• Need to learn how to reason about changing observations
![Page 5: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/5.jpg)
• Add explicit Camera Projection and Differentiable Mapping
• Reason about the instruction on a static map
• Automatically handle changing first-person observations
![Page 6: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/6.jpg)
Each pixel in the feature encodes an image neighbourhood
Input Image
Feature Map
![Page 7: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/7.jpg)
Feature MapProjected Features(Map Frame)
(Image Plane in Camera Frame)
![Page 8: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/8.jpg)
Projected features(time )
Semantic Map (time )
Semantic Map(time )
![Page 9: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/9.jpg)
Semantic Map Grounding Map Goal Map
1x1 F
ilter
9x9 F
ilter
LSTMgo to the left side of plane
Inferred goal locationRecognized airplane
![Page 10: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/10.jpg)
Grounding Map Goal Map
Perceptron
Forward velocity
• Output the velocity
command, given Grounding
and Goal maps
• Sent to quadcopter’s flight-
controller
Yaw rate
![Page 11: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/11.jpg)
Agent
Action
Image Instruction
Oracle
Ground truth action
Ground truth trajectory
Modified variant of DAgger
Trade convergence guarantees for speed and memory efficiency
![Page 12: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/12.jpg)
3500 Instructions + Environments
Ground-truth trajectories
63 Landmarks
252 Possible Tasks
Go to right side of mushroom
Total number of rollouts:3500 oracle
2000 policy
![Page 13: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/13.jpg)
83.47
28.67
87.87
0
20
40
60
80
100
GSMN
(Ours)
NN with no
Mapping
Oracle
Outperform standard NN with no mapping
Very close to oracle performance
![Page 14: Image source: octomap.github.io Image source: pirobot.org/blog/0015/ › ~valts › docs › rss18_slides.pdf · 2019-10-11 · Feature Extraction Mapping 1x1 Filter 9x9 MLP Filter](https://reader034.fdocuments.net/reader034/viewer/2022042406/5f1fd0b1c6f0d664112be6ec/html5/thumbnails/14.jpg)
Feature
ExtractionMapping
1x1 Filter
9x9
FilterMLP
LSTMGo to the left side of
plane
Action
Image Features
Instruction Embedding
Grounding Map
Semantic Map
Goal Map