Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team...
Transcript of Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team...
![Page 1: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/1.jpg)
Amazon Picking Challenge 2016:Team NimbRo of University of Bonn
Computer Science Institute VI – Autonomous Intelligent Systems
Max Schwarz Sebastian Schüller Christian Lenz Arul Selvam PeriyasamyAura Muñoz Michael Schreiber Anton Milan Sven Behnke
![Page 2: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/2.jpg)
Outline
System Control Perception Special Features
![Page 3: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/3.jpg)
3
System OverviewUR 10 Arm (6 DOF)
Line
ar a
ctua
tor
Foldablesuction finger
Central vacuum cleaner (3100 W)
2x Intel RealSense SR300 + LED light
Total: 6 + 2 DOF
Suction strength control
Air velocity sensor
![Page 4: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/4.jpg)
4
Concept and Design
● UR 10: Workspace, Payload, Cost, Safety● Single suction gripper: Avoid design
complexity– Second supporting finger planned
● Folding finger:
front, top, and side grasps
Aim for highest performance at lowest complexity!
![Page 5: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/5.jpg)
Outline
System Control Perception Special Features
![Page 6: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/6.jpg)
6
Motion Generation
● Replace complex motion planning with:– Keyframebased motion generation– Collision detection at runtime (triggered in picking run)
● Assumption: If we can see a point, we can suction it
● Selfcollision detection using MoveIt● Avoid collisions with shelf in IK solver!
![Page 7: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/7.jpg)
7
Inverse Kinematics
● Redundancy resolution by nullspace cost optimization:1.Joint limit avoidance
2.Cartesian plane avoidance (keep wrist out of shelf/tote)
3.Keep linear extension short
● Robust solution using damped least squares
● For inshelf manipulation, onlyposition + suction direction(5D IK)
![Page 8: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/8.jpg)
Outline
System Control Perception Special Features
![Page 9: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/9.jpg)
9
Sensors
2x Intel RealSense SR300
3 Depth measurements per pixel
1) Depth 12) Depth 23) RGB Stereo
Fusion: 2 out of 3
![Page 10: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/10.jpg)
10
RGB
Depth
![Page 11: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/11.jpg)
11
Data Acquisition
● ca. 100 images per setting (shelf/tote) – → 10 images per object
● Manual annotation
![Page 12: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/12.jpg)
12
Deep Features and Online Learning
[Johnson et al., CVPR 2016]
DenseCap
![Page 13: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/13.jpg)
13
Deep Features and Online Learning
[Johnson et al., CVPR 2016]
DenseCap
![Page 14: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/14.jpg)
14
Deep Features and Online Learning
[Johnson et al., CVPR 2016]
Replace Text Generation with Online SVM Training
DenseCap
![Page 15: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/15.jpg)
15
Deep Features andOnline Learning
Glue sticksGloves Sippy cup
![Page 16: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/16.jpg)
16
Semantic Segmentation
Fully Convolutional Neural Network● Pretrained OverFeat on ImageNet● Finetuned (last 3 layers) on APC Data
[Husain et al., RAL 2016]
Training: ~ 3 hours on multiple GPUs
Testing:~ 200 ms per image
![Page 17: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/17.jpg)
17
Semantic SegmentationRGB
HHA
Result
![Page 18: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/18.jpg)
18
Combined Detection and Segmentation
DenseCap Detection
CNN Segmentation
x
![Page 19: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/19.jpg)
19
Generic Grasp Pose Selection
● Center grasp for “standing” objects:– Find support area for suction close to
bounding box center
● Top grasp for “lying” objects:– Find support area for suction close to
horizontal bounding box center
● Custom rules for specific objects
(9 rules in total)
![Page 20: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/20.jpg)
20
6D Pose Estimation
V4R Library [Aldoma et al., ICRA 2013]
● Capture item on turn table● Build 3D model● Generate proposals● Register to test image
![Page 21: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/21.jpg)
21
Pick / Stow Strategy● Pick:
– order A … L
– On failure, retry at end
– Drop at 3 predefined positions in tote
● Stow:– Try to put all items into one 20 points bin
● (select the one with most free space)
– Stow “large” items into own bin ● (coffee, socks, paper towels, tissue box, curtain, pencil cup, mailer)
– If leftover object at end, retry segmentation with all classes
![Page 22: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/22.jpg)
Outline
System Control Perception Special Features
![Page 23: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/23.jpg)
23
Foldable Funnel
![Page 24: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/24.jpg)
24
Foldable Funnel
![Page 25: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/25.jpg)
25
Tricky Items to Grasp
Heavy / cylindrical Hard to suck
→ Ensure that grasp is on center of mass! → Grasp on one ball
?
![Page 26: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/26.jpg)
26
Sucking the Pencil Cup
1. Knock over 2. Suck on bottom
![Page 27: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/27.jpg)
27
Sucking the Pencil Cup1.5x
![Page 28: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/28.jpg)
28
Summary
DELFT 214
NimbRo 186
MIT 164
Stow: 2nd place
DELFT 105
PFN 105
NimbRo 97
Pick: 3rd place
Do it as simple as possible,but not simpler!
![Page 29: Amazon Picking Challenge 2016: Team NimbRo of University ... · Amazon Picking Challenge 2016: Team NimbRo of University of Bonn Computer Science Institute VI – Autonomous Intelligent](https://reader034.fdocuments.net/reader034/viewer/2022050306/5f6e4c3b268f941a8e28fc63/html5/thumbnails/29.jpg)
29
Thank you
Max Schwarz Sebastian Schüller Christian Lenz Arul Selvam PeriyasamyAura Muñoz Michael Schreiber Anton Milan Sven Behnke