MotionPlanningTechnologies&Human‐RobotInteraction

Dinesh ManochaDepartment of Computer Science

University of North Carolina at Chapel Hill

dm@cs.unc.eduhttp://gamma.cs.unc.edu

1

MotionPlanningTechnologies&Human‐RobotInteraction

Dinesh ManochaDepartment of Computer Science

Department of Electrical & Computer EngineeringUniversity of Maryland at College Park (Spring 2018)

dm@cs.unc.edu

2

Robotics Collaborators• Mohit Bansal (UNC) • Andrew Best (UNC)• Sachin Chitta (Willow Garage/SRI)• Biao Jia (UNC)• Sahil Narang (UNC)• Jia Pan (UNC/Berkeley/CityU)• Jean Paul Laumond (LAAS)• Chonhyon Park (Zoox)• Jae Sun Park (UNC)• Zherong Park (UNC)• Jamie Snape (UNC/Kitware)• Fabian Rabe, Shashank Sharma, Christian Scheurer, Uwe Zimmerman (KUKA)• Dorabot (Shenzhen)

Robotics

• Has been around for almost 50 years

• Major success has been in industrial applications: welding, painting, few assembly tasks

• Simple, limited use

• Medical robotics: useful for certain procedures

• Recent trend on consumer robotics

RobotEraisComing!

HRP4C humanoid da vinci

Snake robot

Swarm robots

MEMS bugsBig dog

Robotics: Recent trends

• Autonomous vehicles (Google, Tesla, Uber, Baidu, GM, Ford, SAIC, etc.)

• Warehouse logistics (Amazon, Alibaba, JD.com, etc.)

• Large scale production (Foxconn, etc.)

• Service and social robots (Aldebaran, Sony, Midea, Facebook, Google, etc.)

• Consumer applications (startups)

Robotics & Manufacturing

• Robots have been used for manufacturing and industrial applications since 1970s

• Worldwide robotics spending was $71billion in 2015

• Robotic sales increasing at 17%; expected to be $135 billion in 2019

• Manufacturing sector purchasing hardware + services

Growth in China: 2005-2014

The Chinese suppliers are providing a larger share of robots

RobotsinChinavs.Others

China has been the largest market over the last few years!

Industrial Robot Costs

• Few years ago entry level industrial robot cost was approximately $50K+ (ABB, Kuka, Yaskawa, Fanuc, etc.)

• Over the last few years this cost has come down to approximately $25K (Baxter, Universal)

• Several Chinese manufacturers have stated that they need $5K industrial robots

• $5K appears to be realistic target if the sales volume is high─ Electronic manufacturing is a driving factor─ Sophisticated household appliances cost less than $5K─ An Indian company is able to sell cars for less than $3K

HardwareAloneisNottheSolution

• In 2011, founder Terry Guo predicted that by 2014 Foxconn would install 1M robots

• However, only tens of thousands robots have been installed on assembly lines by 2016

• Foxconn found automation ineffective for assembly lines where people are better than robots?

RobotsatFoxconn

HardwareAloneisNottheSolution

• We need to pay more attention to software and algorithmic solutions

• Exploit recent growth in computer vision, machine learning, and parallel computing

• Robots need to work next to humans

High performance and impact can only be achieved by a combination of hardware and software + interfaces

Challenge:Human‐RobotInteraction

AutonomousDriving• 1.2 billion vehicles on the roads today

• 84 million new vehicles in 2015• China: 24m          U.S.:   2.7m• India:   3.7m          S.E Asia: 3.8m

15New Delhi Bangkok

AutonomousDriving• 1.2 billion vehicles on the roads today

• 84 million new vehicles in 2015• China: 24m          U.S.:   2.7m• India:   3.7m          S.E Asia: 3.8m

• Autonomous vehicles in Challenging Conditions

16New Delhi Bangkok

Vehicle‐HumanInteractions• It will deal with situations like these

17

Vehicle‐HumanInteractions• It will deal with situations like these

18

Vehicle‐HumanInteractions• Ultimate Challenge

19

RobotSystem:Abstraction

• F:    Feedforward• C:   Control• A:    Actuator• S:    Sensor• S+:   Sensor post‐processing

F C A

S

desired task

S+

desired trajectory movementF

planning and trajectory generation

RobotMotionPlanning• Given initial setting A of the robot, find a valid or optimal trajectory for the robot to reach goal B

• Collision‐free• Other constraints (balance)• Optimal criteria (shortest path, min‐time ...)

Initial A

Goal B

Motion Planning

• Widely studied in academic for 40+ years

• Good algorithms and software tools

• Technology transfer (CAD/CAM, games, simulation)• Synthetic environments

• Limited use for industrial or service or entertainment robots

• Challenges for human‐robot systems

Motion Planning: Physical World

The real world is not so niceas virtual reality!

The perceived world!

• Robots use sensors such as cameras, Lidar, tactile, which provide only partial  information about the physical world

• Sensor and actuator error; real‐time data processing

Collision & Proximity Queries

Geometric reasoning of spatial relationships among objects (in a dynamic environment)

d

Closest Points & Separation Distance

d

Penetration Depth

Collision Detection Contact Points & Normals

Our Prior work: Proximity Computations

Multiple software systems• I‐Collide, RAPID, PQP, DEEP, SWIFT, SWIFT++, DeformCD, PIVOT, Self‐CCD, RVO, HRVO

http://gamma.cs.unc.edu/software/#collision

• More than 150,000 downloads from 1995 onwards

• Issued more than 55 commercial licenses (Kawasaki, MSC Software, Ford, Honda, Sensable, Siemens, BMW, Phillips, Intel, Boeing, etc.)

RobotSensors:DataCollection

Cameras

HandlingSensorData

• Human environments• Clutter, dynamic obstacles

• Data from 3D sensors• Large number of points (~10k for laser scans, ~20k for stereo)

• Real‐time computation important for fast online reactive grasping, motion planning

FCL:NoisyPointCloudData[UNC&WillowGarage2013]

Rigid Objects Point Clouds Deformable Objects

ArticulatedObjects

(Discrete)Collision Detection

Y Y Y Y

Continuous Collision Detection

Y Y Y Y

Self Collision Detection

Y Y Y Y

PenetrationEstimation

Y N N N

Distance Computation

Y N Y Y

Broad‐phaseCollision

Y Y Y Y

FCL:Usage• Independent code, but ROS interface is provided

• Available at http://gamma.cs.unc.edu/FCL

• Part of MoveIt: http://moveit.ros.org/wiki/MoveIt!

FCLApplication:OptimalInverseKinematicsforPathPlanning(SWRI)

TaskExecutionsofRobots• Advances in technology allow robots to perform complex tasks

<PR2: fetching a beer from the fridge> <Baxter: $22k robot needs no programming>

TaskExecutionwithMultipleComponents

• A task is decomposed into many primitive subtasks

<PR2: taking out a drink from the fridge>

(From Willow Garage)

1. Move the body to the fridge.2. Move the left arm to the handle.3. Move the body to open the fridge door.4. Move the body to in front of the fridge.5. Move the left arm to hold the door.6. Move the right arm to the beer.7. Grasp the bottle.8. Move the right arm to the basket.9. Release the bottle.

1. Move the body to the fridge.2. Move the left arm to the handle.3. Move the body to open the fridge door.4. Move the body to in front of the fridge.5. Move the left arm to hold the door.6. Move the right arm to the beer.7. Grasp the bottle.8. Move the right arm to the basket.9. Release the bottle .

Most of the subtasks are moving the robot to the next desired pose

SubtaskExecution• ‘Move the body to the fridge’ subtask

• Use sensors to recognize the objects and obstacles in the environment

• Compute a collision‐free path to the pose close to the fridge

• Control motors to execute the computed motion

RobotPlanning:Challenges• How to perform motion planning computation in realtime?• How to handle high DOF robots?• How to handle noisy sensor data?• How to enable human‐robot collaboration?

• Our solution using trajectory optimization http://gamma.cs.unc.edu/ITOMP/ITOMP_ROS

OurWork

• Realtime Planning algorithm for Robots

• End‐effector task constraints• Smooth trajectory• Avoid singularities• Dynamic obstacles• Adapt to human actions• Parallel trajetory optimization using CPU and GPU cores

http://gamma.cs.unc.edu/ITOMP/ITOMP_ROS

ParallelTrajectoryOptimization

• Parallel optimization of multiple trajectories• Use Multiple threads 

• Start from different initial trajectories• Trajectories are generated by quasi‐random sampling

• Exploits the multiple CPU cores (multi‐cores) or GPU‐based cores (many‐cores)

• Used for realtime Motion & Cartesian Planning

http://gamma.cs.unc.edu/ITOMP/ITOMP_ROS/

DynamicObstaclesfromKinect

• Depthmap

37

Reconstructed Octomap

MotionPlanningonKUKAManipulators

38

DoraPicker:AmazonChallenge

39

SafeMotionPlanningforHuman‐RobotInteraction

Motivation:UncertainHumanMotions

• Co‐robots (or cobot)• Sharing workspace with human

• Human safety• Uncertain human motions

• Noisy representation ofhuman motion

• Need future motionprediction

• Need collision handlingwith noisy data

Ourwork• Human motion prediction

• Action classification• Trajectory estimationand prediction

• Safe motion planning• Collision handling with noisy data

• Efficient task planning• Reduce task completion time• Delays when attempting to avoid collisions

Current Future

Human‐RobotPlanner

Intention & MotionEstimation

EnvironmentRobot

Motion Planner

Taskplanner

SafeMotionPlanningusingPrediction

44

SafeMotionPlanningusingPrediction

45

NaturalLanguageHuman‐RobotCommunication

46Handling complex attributes

RobotsandCrowds• Goal: Get where you are going without running into anyone

• Challenges• Each other agent is an obstacle• Dynamic obstacles• Reactive• Complex landscape

• Distributed motion planning problem!

Multi-agent Planning 47

Crowd Simulation & Prediction

“

Trajectory Prediction: Multi-Agent Planning + Bayesian Learning

Autonovi‐Sim:AutonomousDrivingSimulation

• 3 Modes in AutonoVi‐Sim• Manual• Basic Follower• AutonoVi navigation algorithm

50

Manual Drive Basic Follower AutonoVi

Vehicle‐PedestrianInteractions• Vehicle accounts for pedestrians and comes to a stop

Vehicle‐PedestrianInteractions• Modeling vehicle dynamics and pedestrian behaviors

Vehicle‐BicycleInteractions

53

Conclusions• The next challenge in robotics is to make them work with humans

• Builds on work in AI, computer vision, NLP, machine learning and simulation

• New Designs for Corobots: Robots working with humans

• Key technology for autonomous vehicles

• Need more investments in software and interfaces