ISAACAn Accelerated Robotics Framework

Hammad Mazhar

2

Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine

PR2 - HOCKEY

3

PROJECT ISAAC: HOCKEY

Teach a robot how to play hockey using ground truth

Robot: PR2 (Willow Garage)

Kinematic model

7 joints, 1 arm

Observations:

Puck and Joint positions, velocities

Goal location

3

4

PROJECT ISAAC

5

DISCO HOCKEY BOT

5

6

ISAAC LAB: DOMINOES

6

7

SYNTHETIC DATA GENERATION• Generate 50K images

• Multiple distractors, lights, etc

• Labeled:

• Location, pose, pip count

7

8

GAMEPLAY LOGIC TRAINING (SLOW MOTION)

8

9

ISAAC PLAYS DOMINOES

10

ROBOTICS AT NVIDIATransfer learning in Isaac Lab

11

ISAAC PLAYS GOLF

12

VIRTUAL DEMONSTRATION FOR ROBOTICS

13ISAAC SDK

14

ISAACSDK FOR ROBOTICS

Simulation EnginePhoto-realistic Graphics ∙ Physics ∙

∙ Procedural Generation ∙ Massive parallelism

∙ Unreal Engine 4

Isaac FrameworkCodelets ∙ Behaviors ∙ 3D Poses ∙ Distributed

∙ Messaging ∙ Synchronization ∙ Record & Replay

∙ Configuration ∙ Visualization

World modelWarehouse ∙ Office

∙ Store ∙ Home

Robot modelCarter ∙ URDF loader

Virtual Sensors Virtual Actuators Sensor Processing Actuator Control HW Sensor HW Actuator

MLTensorRT ∙ CUDA

∙ Tensorflow ∙ ...

GemsOptimizers ∙ Algebra

∙ EKFs ∙ Depth ∙ ...N

avig

atio

n

Be

ha

vio

rs

Inte

ractio

ns

with

hu

ma

ns

Manip

ula

tion

Pe

rce

ptio

n

DriversLidar ∙ Camera ∙ IMU ∙

Robot Base ∙ ...

Unified Message APIUse the same messages for simulation,

actual hardware and across all apps

Simulate DeployDevelop

JetsonFully integrated with

X2 and Xavier

15

ISAAC SDK: A ROBOTICS TOOLBOX

Jetson AGX XAVIER

Advanced GPU, 64-bit CPU, Video CODEC, VIC, ISP

CUDA, Linux4Tegra

Core libraries Navigation Reference DNN

VisionWorks, OpenCV, TensorRT, cuDNN

ToolsDrivers

16

NODE BASED PROGRAMMING INTERFACE

17

Global planner

NAVIGATION GEMS

Global planner Obstacle avoidance Localization State machine

18

STEREO DNN

19

DEVELOPER TOOLS

20

ISAAC SIMULATION

21

SENSOR SIMULATIONCameras and Lidar

22

MANIPULATIONURDF Support

23

ROSInteroperability with existing robotic stacks

24

25

DESIGNING MEANINGFUL TRAINING ENVIRONMENTS

25

26

DESIGNING MEANINGFUL TRAINING ENVIRONMENTS

26

27

PHYSX 4.0

28

CARTER

29

CARTER REFERENCE DESIGN

Velodyne VLP16 Zed stereo IMUSegway

Carter

Jetson AGX Xavier

30

Iterate at high speed on a desktop and in simulation

Deploy and test in reality

Training on DGX

CARTER REFERENCE DESIGN

Xavier and TX2

Carter app

31

CARTER WITH ISAAC SDK

32

SALAD DELIVERY

33

NEXT STEPS

34

July 2018 2019 2020

CARTER IS JUST THE BEGINNING

35

QUESTIONS?

Hammad Mazharhmazhar@nvidia.com