Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques...
Transcript of Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques...
![Page 1: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/1.jpg)
Is Motion Planning Overrated?Jeannette Bohg - Interactive Perception and Robot Learning Lab - Stanford
![Page 2: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/2.jpg)
Is Motion Planning Overrated?Jeannette Bohg - Interactive Perception and Robot Learning Lab - Stanford
tl;dr; Yes
![Page 3: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/3.jpg)
Is Motion Planning Overrated?Jeannette Bohg - Interactive Perception and Robot Learning Lab - Stanford
![Page 4: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/4.jpg)
Is Motion Planning Overrated?Jeannette Bohg - Interactive Perception and Robot Learning Lab - Stanford
tl;dr; It depends
![Page 5: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/5.jpg)
Robot System DesignDynamic and Uncertain Environment
![Page 6: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/6.jpg)
Robot System DesignDynamic and Uncertain Environment
![Page 7: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/7.jpg)
Robot System DesignDynamic and Uncertain Environment
Real-Time Perception meets Reactive Motion Generation
![Page 8: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/8.jpg)
Inverse Dynamics Controller
RobotTorques
Joint Sensors
Robot State Acceleration Policies
Camera
World
Interaction
World Model
Object Tracker
Arm Tracker
Motion Optimizer
Kappler et al. Real-Time Perception meets Reactive Motion Generation. RA-L + ICRA’18. Finalist 2018 Amazon Best Systems Paper
![Page 9: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/9.jpg)
One time step in the system
![Page 10: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/10.jpg)
One time step in the system
Raw Sensory Data
30Hz-1kHz
![Page 11: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/11.jpg)
One time step in the system
Raw Sensory Data Processed Sensory Data
30Hz-1kHz 15-30Hz
![Page 12: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/12.jpg)
One time step in the system
Raw Sensory Data Processed Sensory Data Local and optimised policies
30Hz-1kHz 15-30Hz 30kHz 5-10Hz
![Page 13: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/13.jpg)
One time step in the system
Raw Sensory Data Processed Sensory Data Local and optimised policies Fused Policy
30Hz-1kHz 15-30Hz 30kHz 5-10Hz 30 kHz
![Page 14: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/14.jpg)
One time step in the system
Raw Sensory Data Processed Sensory Data Local and optimised policies Fused Policy
30Hz-1kHz 15-30Hz 30kHz 5-10Hz 30 kHz
Robot Controlled at 1kHz
![Page 15: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/15.jpg)
System-Level Evaluation
![Page 16: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/16.jpg)
System-Level Evaluation
Sense-Plan-Act
![Page 17: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/17.jpg)
System-Level Evaluation
Sense-Plan-Act Feedback Control
![Page 18: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/18.jpg)
System-Level Evaluation
Sense-Plan-Act Feedback Control Full System
![Page 19: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/19.jpg)
System-Level Evaluation
Static Pick and Place Dynamic Pick and Place Dynamic Grasping Dynamic Pointing
![Page 20: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/20.jpg)
System-Level Evaluation
Static Pick and Place Dynamic Pick and Place Dynamic Grasping Dynamic Pointing
![Page 21: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/21.jpg)
![Page 22: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/22.jpg)
![Page 23: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/23.jpg)
Is Motion Planning Overrated?Jeannette Bohg - Interactive Perception and Robot Learning Lab - Stanford
tl;dr; It depends
![Page 24: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/24.jpg)
![Page 25: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/25.jpg)
Environment Complexity
![Page 26: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/26.jpg)
Success of Feedback Control
Environment Complexity
![Page 27: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/27.jpg)
Success of Feedback Control
Environment Complexity
![Page 28: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/28.jpg)
Success of Feedback Control
Environment Complexity
Real-Time Perception meets Reactive Motion Generation. Kappler et al. ICRA’18 + RAL.
![Page 29: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/29.jpg)
Success of Feedback Control
Environment Complexity
Real-Time Perception meets Reactive Motion Generation. Kappler et al. ICRA’18 + RAL.
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Kalashnikov et al. To appear at CORL ’18.
![Page 30: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/30.jpg)
Success of Feedback Control
Environment Complexity
Real-Time Perception meets Reactive Motion Generation. Kappler et al. ICRA’18 + RAL.
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Kalashnikov et al. To appear at CORL ’18.
Motion Planning
![Page 31: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/31.jpg)
Success of Feedback Control
Environment Complexity
Real-Time Perception meets Reactive Motion Generation. Kappler et al. ICRA’18 + RAL.
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. Kalashnikov et al. To appear at CORL ’18.
Motion PlanningReactive
![Page 32: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/32.jpg)
Take-Home from Systems Paper
![Page 33: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/33.jpg)
Take-Home from Systems Paper
Higher complexity - More planning
![Page 34: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/34.jpg)
Take-Home from Systems Paper
Higher complexity - More planning
More uncertainty and dynamics - Online Re-Planning
![Page 35: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/35.jpg)
Controlling through contact
Object
Finger
Goal
![Page 36: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/36.jpg)
Controlling through contact
![Page 37: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/37.jpg)
Controlling through contact
![Page 38: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/38.jpg)
Controlling through contact
![Page 39: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/39.jpg)
Controlling through contact
![Page 40: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/40.jpg)
Controlling through contact
![Page 41: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/41.jpg)
Controlling through contact
Better ModelsBetter Feedback
![Page 42: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/42.jpg)
Predictive Model
![Page 43: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/43.jpg)
Model
Predictive Model
![Page 44: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/44.jpg)
Model
Sensory Observations
Predictive Model
![Page 45: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/45.jpg)
Model
Sensory Observations
Action
Predictive Model
![Page 46: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/46.jpg)
Model
Sensory Observations
Action
Predictive Model
![Page 47: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/47.jpg)
Model
Sensory Observations
Action
Predicted Effect
Predictive Model
![Page 48: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/48.jpg)
Model
Sensory Observations
Action
Predicted Effect
Model-Predictive Control
Predictive Model
![Page 49: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/49.jpg)
Predicting physical effect
Goal
![Page 50: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/50.jpg)
Predicting physical effect
Goal
![Page 51: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/51.jpg)
Predicting physical effect
Goal
![Page 52: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/52.jpg)
Keep Predicting
Goal
![Page 53: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/53.jpg)
Keep Predicting
Goal
![Page 54: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/54.jpg)
Our Hypothesis
![Page 55: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/55.jpg)
Our HypothesisPhysics Models
![Page 56: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/56.jpg)
Our HypothesisPhysics Models +
![Page 57: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/57.jpg)
Our HypothesisPhysics Models + Learning
![Page 58: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/58.jpg)
Our HypothesisPhysics Models + Learning
= Generalization
![Page 59: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/59.jpg)
Hybrid Model
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sensory Observations
![Page 60: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/60.jpg)
Hybrid Model
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sensory Observations End-to-EndLoss on Effect
![Page 61: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/61.jpg)
Hybrid Model
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sensory Observations End-to-EndLoss on Effect
Extrapolation
![Page 62: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/62.jpg)
Testing Hypothesis on a Case Study
More than a Million Ways to Be Pushed. A High-Fidelity Experimental Dataset of Planar Pushing. Yu et al. IROS 2016.
![Page 63: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/63.jpg)
Testing Hypothesis on a Case Study
More than a Million Ways to Be Pushed. A High-Fidelity Experimental Dataset of Planar Pushing. Yu et al. IROS 2016.
![Page 64: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/64.jpg)
Testing Hypothesis on a Case Study
More than a Million Ways to Be Pushed. A High-Fidelity Experimental Dataset of Planar Pushing. Yu et al. IROS 2016.
![Page 65: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/65.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
Analytic Model
![Page 66: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/66.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
Analytic Model
![Page 67: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/67.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
1. Stage
Analytic Model
![Page 68: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/68.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
1. Stage
Analytic Model
![Page 69: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/69.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
1. Stage
Analytic Model
![Page 70: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/70.jpg)
x
x
x
x
K. M. Lynch, H. Maekawa, and K. Tanie, “Manipulation and active sensing by pushing using tactile feedback,” in Proc. IEEE/RSJ Int. Conf. Intelligent Robots and Systems, vol. 1, Jul 1992, pp. 416–421
2. Stage
1. Stage
Analytic Model
![Page 71: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/71.jpg)
Learned Model
Action
Predicted Effect
Sensory Observations
Neural Network only
Compared Architectures
![Page 72: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/72.jpg)
Learned Model
Action
Predicted Effect
Sensory Observations
Neural Network only Hybrid Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Compared Architectures
![Page 73: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/73.jpg)
Learned Model
Action
Predicted Effect
Sensory Observations
Neural Network only Hybrid Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Compared ArchitecturesError Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Learned Error
+
![Page 74: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/74.jpg)
Learned Model
Action
Predicted Effect
Sensory Observations
Neural Network only Hybrid Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Compared ArchitecturesRaw Sensory Observations
Error Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Learned Error
+
![Page 75: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/75.jpg)
Learned Model
Action
Predicted Effect
Sensory Observations
Neural Network only Hybrid Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Training: End-to-End Loss: Error between Predicted and Ground Truth Effect
Compared ArchitecturesRaw Sensory Observations
Error Model
Physics-based Model
ActionPredicted
Effect
Sensory Observations
Parameters
Learned Model
Learned Error
+
![Page 76: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/76.jpg)
Testing Data Efficiency
![Page 77: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/77.jpg)
Testing Data Efficiency
![Page 78: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/78.jpg)
Testing Data Efficiency
![Page 79: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/79.jpg)
Testing Data Efficiency
![Page 80: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/80.jpg)
Testing Generalization
Alina Kloss et al, “Combining learned and analytical models for predicting action effects,” Submitted. 2018. Pre-print on arXiv.
![Page 81: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/81.jpg)
Testing Generalization
New Pushing Angles & Contact Points
Training Testing
Interpolation
Alina Kloss et al, “Combining learned and analytical models for predicting action effects,” Submitted. 2018. Pre-print on arXiv.
![Page 82: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/82.jpg)
Testing Generalization
New Pushing Angles & Contact Points
Training Testing
New Push Velocities
Training Testing
Interpolation Extrapolation
Alina Kloss et al, “Combining learned and analytical models for predicting action effects,” Submitted. 2018. Pre-print on arXiv.
![Page 83: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/83.jpg)
Testing Generalization
New Pushing Angles & Contact Points
Training Testing
New Push Velocities
Training Testing
New Object Shapes
Training Testing
Interpolation Extrapolation
Alina Kloss et al, “Combining learned and analytical models for predicting action effects,” Submitted. 2018. Pre-print on arXiv.
![Page 84: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/84.jpg)
Training Testing
Generalization to new push velocities
![Page 85: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/85.jpg)
Training Testing
Generalization to new push velocities
Extrapolation
![Page 86: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/86.jpg)
Training Testing
Generalization to new push velocities
Extrapolation
![Page 87: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/87.jpg)
Training Testing
Generalization to new push velocities
![Page 88: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/88.jpg)
Training Testing
Generalization to new push velocities
Extrapolation
![Page 89: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/89.jpg)
Training Testing
Generalization to new push velocities
![Page 90: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/90.jpg)
Training Testing
Generalization to new push velocities
Extrapolation
![Page 91: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/91.jpg)
Training Testing
Generalization to new push velocities
![Page 92: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/92.jpg)
Training Testing
Generalization to new push velocities
Extrapolation
![Page 93: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/93.jpg)
Don’t throw away structure
![Page 94: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/94.jpg)
Learn to extract given state representation from raw data
![Page 95: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/95.jpg)
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sensory Observations End-to-EndLoss on Effect
A Concrete Suggestion
![Page 96: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/96.jpg)
Interpretability
![Page 97: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/97.jpg)
Interpretability Real and Predicted Box Position after 200 identical pushes
Representation interpretable
![Page 98: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/98.jpg)
Compensation for Errors in Analytical Model Wrong Friction Parameters of Analytical Model
![Page 99: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/99.jpg)
Future Direction
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sequence of Sensory Observation
![Page 100: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/100.jpg)
Future Direction
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sequence of Sensory Observation
Multistep Prediction
![Page 101: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/101.jpg)
Future Direction
Physics-based Model
Action
Predicted EffectParameters
Learned Model
Sequence of Sensory Observation
Multistep Prediction
Backpropagation w. r. t. control
![Page 102: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/102.jpg)
Models will never be perfect
![Page 103: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/103.jpg)
Differentiable Recursive Filtering
Jonschkowski et al. RSS’18
Haarnoja et al. NIPS’16
Karkus et al. arXiv’18
Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
![Page 104: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/104.jpg)
Learning Heteroscedastic Noise
On Learning Heteroscedastic Noise Models within Differentiable Filtering. Kloss and Bohg. Submitted to ICLR’19. Picture adapted from: Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
![Page 105: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/105.jpg)
Learning Heteroscedastic Noise
On Learning Heteroscedastic Noise Models within Differentiable Filtering. Kloss and Bohg. Submitted to ICLR’19. Picture adapted from: Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
Observation Noise
![Page 106: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/106.jpg)
Learning Heteroscedastic Noise
On Learning Heteroscedastic Noise Models within Differentiable Filtering. Kloss and Bohg. Submitted to ICLR’19. Picture adapted from: Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
Observation Noise
Process Noise
![Page 107: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/107.jpg)
Learning Heteroscedastic Noise
On Learning Heteroscedastic Noise Models within Differentiable Filtering. Kloss and Bohg. Submitted to ICLR’19. Picture adapted from: Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
Observation Noise
Process Noise
![Page 108: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/108.jpg)
Learning Heteroscedastic Noise
On Learning Heteroscedastic Noise Models within Differentiable Filtering. Kloss and Bohg. Submitted to ICLR’19. Picture adapted from: Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors. Jonschkowski et al. RSS’18
Observation Noise
Process Noise
![Page 109: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/109.jpg)
Four Non-Linear Filters
![Page 110: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/110.jpg)
Four Non-Linear Filters
Extended Kalman Filter
![Page 111: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/111.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
![Page 112: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/112.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
![Page 113: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/113.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
![Page 114: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/114.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 115: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/115.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 116: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/116.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 117: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/117.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 118: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/118.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 119: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/119.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 120: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/120.jpg)
Four Non-Linear Filters
Extended Kalman Filter
Unscented Kalman Filter (UKF)
Monte-Carlo UKF
Particle Filter
Process Model Observation Model
![Page 121: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/121.jpg)
Loss Function
![Page 122: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/122.jpg)
Loss Function
Ground Truth State Sequence
![Page 123: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/123.jpg)
Loss Function
Ground Truth State Sequence
Sequence of State Estimates (Mean and Covariance)
![Page 124: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/124.jpg)
Loss Function
Ground Truth State Sequence
Sequence of State Estimates (Mean and Covariance)
Network Weights
![Page 125: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/125.jpg)
Loss Function
Ground Truth State Sequence
Sequence of State Estimates (Mean and Covariance)
Network Weights
Negative log-likelihood of true state given believe
![Page 126: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/126.jpg)
Loss Function
Ground Truth State Sequence
Sequence of State Estimates (Mean and Covariance)
Network Weights
Negative log-likelihood of true state given believe Estimation Error
![Page 127: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/127.jpg)
Loss Function
Ground Truth State Sequence
Sequence of State Estimates (Mean and Covariance)
Network Weights
Negative log-likelihood of true state given believe Estimation Error Regularization
![Page 128: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/128.jpg)
Two Application
Kitti Visual Odometry Task Planar Pushing Task
![Page 129: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/129.jpg)
Visual OdometryLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises Mixed
![Page 130: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/130.jpg)
Visual OdometryLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises Mixed
![Page 131: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/131.jpg)
Visual OdometryLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises Mixed
![Page 132: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/132.jpg)
Visual OdometryLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises Mixed
![Page 133: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/133.jpg)
Visual OdometryLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises Mixed
![Page 134: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/134.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 135: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/135.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 136: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/136.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 137: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/137.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 138: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/138.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 139: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/139.jpg)
Planar PushingLearned Constant Observation Noise
Learned Hsc. Process Noise
Learned Constant Noises MixedWell-Tuned
NoiseBadly-Tuned
Noise
![Page 140: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/140.jpg)
Take Home
![Page 141: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/141.jpg)
Take Home
Differentiable EKF most accurate and robust to inaccurate noise models
![Page 142: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/142.jpg)
Take Home
Differentiable EKF most accurate and robust to inaccurate noise models
Particle filter gains most from heteroscedastic noise model.
![Page 143: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/143.jpg)
Take Home
Differentiable EKF most accurate and robust to inaccurate noise models
Particle filter gains most from heteroscedastic noise model.
UKF works best for more complex dynamics models.
![Page 144: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/144.jpg)
Take Home
Differentiable EKF most accurate and robust to inaccurate noise models
Particle filter gains most from heteroscedastic noise model.
UKF works best for more complex dynamics models.
![Page 145: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/145.jpg)
Take Home
Differentiable EKF most accurate and robust to inaccurate noise models
Particle filter gains most from heteroscedastic noise model.
UKF works best for more complex dynamics models.
Uncertainty bounds for long-term predictions.
![Page 146: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/146.jpg)
Conclusions
![Page 147: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/147.jpg)
Conclusions
Environment Complexity
![Page 148: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/148.jpg)
ConclusionsSuccess of Feedback Control
Environment Complexity
![Page 149: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/149.jpg)
ConclusionsSuccess of Feedback Control
Environment Complexity
![Page 150: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/150.jpg)
ConclusionsSuccess of Feedback Control
Environment Complexity
Motion Planning
![Page 151: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/151.jpg)
ConclusionsSuccess of Feedback Control
Environment Complexity
Motion PlanningReactive
![Page 152: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/152.jpg)
ConclusionsSuccess of Feedback Control
Environment Complexity
Motion PlanningReactive
Better Predictive Models Better Feedback
O P
![Page 153: Is Motion Planning Overrated? - Personal Robotics Lab...Inverse Dynamics Controller Robot Torques Joint Sensors Robot State Acceleration Policies Camera World Interaction World Model](https://reader034.fdocuments.net/reader034/viewer/2022052102/603cc2da52d49e5e3a21b198/html5/thumbnails/153.jpg)
Thank you for your Attention!
IPRL @ Stanford AMD @ MPI, CLMC @ USCiprl.stanford.edu https://am.is.tuebingen.mpg.de/