Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep...

17
Deep Learning for Semantics, Geometry, and Physics in Robotics Hao Su

Transcript of Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep...

Page 1: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Deep Learning for Semantics, Geometry, and

Physics in Robotics

Hao Su

Page 2: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Course website

• http://robotmlcourse.github.io

http://geoml.github.io
Page 3: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Who are we?

Instructor: Hao Su Teaching Assistant: Jiayuan Gu

Page 4: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Goal

• A research-driven course

• Explore how to apply machine learning for object manipulation tasks

• We study the topic together!

Page 5: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Why we are interested in this topic?

Passive AI

• We know how to fit data well (by “deep learning”)• e.g., computer vision, natural language processing

Page 6: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Why we are interested in this topic?

Passive AI Active AI

• We aspire that autonomous agents can discover knowledge by itself through exploration• Need the ability to interact

Page 7: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Perception Actuation

3D Vision InteractiveLearning

World Model

Page 8: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

To interaction with the environment, we need

• Observe the environment (perception)

• Build the environment model (modeling)

• Select the actions (planning)

Three Components of Interaction

Page 9: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

• Perception: observe the status of the environment

• Classification

• Detection

• Reconstruction

Perception

Page 10: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

• Models: structured knowledge about the current environment from the observation.

• Physical laws

• 3D geometry

• Theory of mind

Environment Modeling

Page 11: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

• Planning: select the action based on the model

Planning

Shortest path Evolutionary AlgorithmGradient descent

Page 12: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Environment Modeling

Perception Planning

Modeling

Page 13: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Structure of the Course (I)

• First semi-quarter: My Ph.D. students will teach the basics of robotics• Deep reinforcement learning• Basics concepts in mechanics• Forward/inverse kinematics/dynamics• Motion planning• Control• Rigid-body simulation• Fluid simulation

• Students are asked to finish a small project to implement a robot arm for object movement (mid-term project)

Page 14: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

SAPIEN: A SimulAted Part-based Interactive Environment

https://sapien.ucsd.edu/

https://sapien.ucsd.edu/
Page 15: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Structure of the Course (II)

• Second semi-quarter: Paper presentation by students• Cover classic and latest papers

• Students are asked to work on an object manipulation challenge using SAPIEN benchmark (final project)

Page 16: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Logistics

Grading (tentative)• Class presence: 10%• Mid-term project: 30%• Paper presentation: 30%• Final project: 30%• There will not be a final exam.

Page 17: Deep Learning for Semantics, Geometry, and Physics in Robotics · basics of robotics • Deep reinforcement learning • Basics concepts in mechanics • Forward/inverse kinematics/dynamics

Pre-requisite

• Try to be as self-contained as possible

• Proficiency in Python and C

• Calculus, Linear Algebra

• Machine learning


Robotics for Beginners flyer...Robotics for Beginners! Learn the basic concepts behind robotics and computer programing We’ll use the LittleBits system to learn the basics of electronics

Robotics for Beginners flyer...Robotics for Beginners! Learn the basic concepts behind robotics and computer programing We’ll use the LittleBits system to learn the basics of electronics

Deep Reinforcement Learning for Robotics · Deep reinforcement learning . Robotics . Percepts . Hand-engineered state-estimation . Many-layer neural network with many parameters to

Deep Reinforcement Learning for Robotics · Deep reinforcement learning . Robotics . Percepts . Hand-engineered state-estimation . Many-layer neural network with many parameters to

Electronics and Controls Basics of an FRC Robot’s control system. Robotics 101.

Electronics and Controls Basics of an FRC Robot’s control system. Robotics 101.

OVN - Basics and deep dive

OVN - Basics and deep dive

Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

Deep Reinforcement Learning for Robotics Using DIANNEon-demand.gputechconf.com/gtc-eu/2017/presentation/... · PUBLIC Deep Reinforcement Learning for Robotics Using DIANNE Tim Verbelen,

(RSS 2017) Workshop New Frontiers for Deep Learning in ... · Robotics: Science and Systems (RSS 2017) Workshop New Frontiers for Deep Learning in Robotics 15th July, 2017, Boston,

(RSS 2017) Workshop New Frontiers for Deep Learning in ... · Robotics: Science and Systems (RSS 2017) Workshop New Frontiers for Deep Learning in Robotics 15th July, 2017, Boston,

Basics of Robotics

Basics of Robotics

The Basics of Deep Venous DVD Disease (DVD) Deep Venous ...pvdandme.com/wp-content/themes/dt-the7-child/... · The Basics of Deep Venous DVD Disease (DVD) Deep veins play an important

The Basics of Deep Venous DVD Disease (DVD) Deep Venous ...pvdandme.com/wp-content/themes/dt-the7-child/... · The Basics of Deep Venous DVD Disease (DVD) Deep veins play an important

Deep-Learning for Robotics & Autonomous Vehiclesperso.mines-paristech.fr/fabien.moutarde/Prez/Deep-Learning_Roboti… · Deep-Learning for Robotics & Autonomous Vehicles, Pr.Fabien

Deep-Learning for Robotics & Autonomous Vehiclesperso.mines-paristech.fr/fabien.moutarde/Prez/Deep-Learning_Roboti… · Deep-Learning for Robotics & Autonomous Vehicles, Pr.Fabien

AI Sensing for Robotics using Deep Learning based Visual ...

AI Sensing for Robotics using Deep Learning based Visual ...

Deep Learning Basics Lecture 8: Autoencoder & DBM Learning Basics Lecture 8: Autoencoder & DBM ... •Sparsity of the representation (sparse autoencoder) ... Figure from Deep Learning,

Deep Learning Basics Lecture 8: Autoencoder & DBM Learning Basics Lecture 8: Autoencoder & DBM ... •Sparsity of the representation (sparse autoencoder) ... Figure from Deep Learning,

Lecture 1: Introduction to Neural Networks and Deep Learning · Robotics and AI Music and the arts! o A brief history of Neural Networks and Deep Learning o Basics of Neural Networks

Lecture 1: Introduction to Neural Networks and Deep Learning · Robotics and AI Music and the arts! o A brief history of Neural Networks and Deep Learning o Basics of Neural Networks

Introduction to Deep Reinforcement Learning Model-free Methods · •People started to apply deep learning into robotics Deep RL for Robotics Levine, Sergey, and Chelsea Finn. "Deep

Introduction to Deep Reinforcement Learning Model-free Methods · •People started to apply deep learning into robotics Deep RL for Robotics Levine, Sergey, and Chelsea Finn. "Deep

Deep-Learning : the basics · 2018-11-29 · Deep-Learning : the basics A. Allauzen Universit e Paris-Sud / LIMSI-CNRS 9 mai 2017 Alexandre Allauzen (LIMSI-CNRS) NNet basics 9 mai

Deep-Learning : the basics · 2018-11-29 · Deep-Learning : the basics A. Allauzen Universit e Paris-Sud / LIMSI-CNRS 9 mai 2017 Alexandre Allauzen (LIMSI-CNRS) NNet basics 9 mai

Basic robotics - XLR8 basics

Basic robotics - XLR8 basics

Deep Learning in Robotics

Deep Learning in Robotics

SOLIDWORKS: Lesson 1 - Basics and Modeling Fundamentals Introduction to Robotics.

SOLIDWORKS: Lesson 1 - Basics and Modeling Fundamentals Introduction to Robotics.

Basics of Deep Learning: A Radiologist’s Guide to ... › Synapse › Data › PDFData › 0068...Basics of Deep Learning: A Radiologist’s Guide to Understanding Published Radiology

Basics of Deep Learning: A Radiologist’s Guide to ... › Synapse › Data › PDFData › 0068...Basics of Deep Learning: A Radiologist’s Guide to Understanding Published Radiology

BEST Kit Basics - Best Robotics Inc

BEST Kit Basics - Best Robotics Inc

Msc Project / Internship Deep Learning for Industrial ... de... · Msc Project / Internship Deep Learning for Industrial Robotic Applications About Delft Robotics Delft Robotics is

Msc Project / Internship Deep Learning for Industrial ... de... · Msc Project / Internship Deep Learning for Industrial Robotic Applications About Delft Robotics Delft Robotics is