Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed...
Transcript of Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed...
![Page 1: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/1.jpg)
1© 2015 The MathWorks, Inc.
Connecting MATLAB and ROS
for Autonomous Driving
Carlos Santacruz-Rosero, Ph.D.Senior Application Engineer – Robotics
Harshita BhuratRobotics System Toolbox Product Manager
![Page 2: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/2.jpg)
2
Agenda
Automated Driving at BMW
ROS for Automated Driving
Robotics Workflow at the MathWorks
![Page 3: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/3.jpg)
3
![Page 4: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/4.jpg)
4
Automated Driving with ROS at BMW
** MICHAEL AEBERHARD, BMW, ROSCON 2015
![Page 5: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/5.jpg)
5
Automated Driving with ROS at BMW
Using Robotics System Toolbox
![Page 6: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/6.jpg)
6
What is ROS (Robot Operating System)?
• An architecture for distributed inter-
process communication
• Packages for common algorithms
and drivers
• Multilanguage interface (C++,
Python, Lua, Java and MATLAB)
![Page 7: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/7.jpg)
7
ROS Trends in Robotics Development
http://rosindustrial.org/ric-americas/
• #1 middleware for robotics
applications development
• Popular in research and
gaining great momentum in
industry
![Page 8: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/8.jpg)
8
Benefits of ROS for Automated Driving
• Sensor data acquisition
• Reliable distributed architecture
• Lots of “off the shelf” algorithms for
autonomy • Path planning
• Collision avoidance
• Sensor processing
• Simplified component compatibility
through standalone interfaces
• Integration with simulation environments
![Page 9: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/9.jpg)
9
Autonomous Car as an Advanced Robotics System
LIDAR RADAR GPS/IMUCamera
Steering Actuator
Actuator ECUs
GAS/Brake Actuator
Motion Controllers
Planning
Localization Obstacle
avoidance
Global Map
Motion controlNODE
NODE
NODE
NODE
NODE
NODE
ROS: communication framework and stack of libraries
![Page 10: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/10.jpg)
10
Robotics Development Workflow with Hand Code
Implementation
- Traceability to requirements
- Analyzing logged data
- Designing algorithm
- Translating to C/C++ , Python
- Visualizing and debugging
- Changing open-source code
- Integration into production
Product
Requirements
Idea
Design, Simulation, Prototype
C++ Python
![Page 11: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/11.jpg)
11
Robotics Development Workflow with MATLAB and MBD
Implementation
Product
Requirements
Idea
Design and
Simulation
- System Level Simulation
- Leverage built-in algorithms and
Apps
- Co-simulation with ROS-Enabled
Systems
- C/C++ automatic code generation
- SIL with ROS Enabled Systems
- Debug C/C++ with MATLAB Engine
Prototype
RAPID ITERATIVE PROCESS
Design independent of
communication
framework!
![Page 12: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/12.jpg)
12
Simulate & Prototype
Robotics Development Workflow with MATLAB and ROS
Visualize
Experiment
Test
![Page 13: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/13.jpg)
13
Let’s solve a real problem: Sign Detection System
Clearpath Husky Robot
- ROS Enabled
- Kinect (RGB and Point cloud)
- Hokuyo 2D Lidar
Track and Park
Reduce Speed
Collision Avoidance
Simulation model in Gazebo
![Page 14: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/14.jpg)
14
System Level Design ROS as CommunicationFramework
Object ClassifierObstacle Avoidance
State Machine
![Page 15: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/15.jpg)
15
Sign Recognition with Collision Avoidance
![Page 16: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/16.jpg)
16
Robotics Development Workflow with MATLAB and ROS
- Import image data from ROS bags
- Threshold and detect features in images
- Train a classifier
- Test algorithm
Implementation
Product
Requirements
Idea
Design and
Simulation
Prototype
RAPID ITERATIVE PROCESS
![Page 17: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/17.jpg)
17
Importing Simulation and Experimental Data
Robotics System Toolbox™
Import ROS Data
Filter your logged field
data by topic or time
intervalExperimental data or
simulation Data
Data ready to
design algorithms
![Page 18: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/18.jpg)
18
Visualize, Analyze, and Process Data: Classifier
OutputInput
stop
membrane
speedlimit
neg
Training data Preprocessing Feature Extraction Training
Classifier
% Detect red regions
BW = createMask(videoFrame);
% Fill image regions
BW = imfill(BW,'holes');
% Get bounding boxes
stats = regionprops('table',BW,'BoundingBox','Area');
% Filter based on area size
targetIndex = stats.Area > 500;
% Get bounding boxes from detected regions
testFeatures(k,:) = extractHOGFeatures(Icr);
![Page 19: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/19.jpg)
19
Visualize, Analyze, and Process Data: Classifier
Statistics and Machine Learning Toolbox™Computer Vision System Toolbox™
OutputInput
stop
membrane
speedlimit
neg
Training data Preprocessing Feature Extraction Training
ClassifierClassifier
Preprocessing Feature Extraction
![Page 20: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/20.jpg)
20
Design and Test Algorithm
![Page 21: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/21.jpg)
21
Design and Test Algorithm
![Page 22: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/22.jpg)
22
Robotics Development Workflow with MATLAB and ROS
- Test algorithm with an external simulator
- Tune your algorithm
- Integrate with other algorithms
Implementation
Product
Requirements
Idea
Design and
Simulation
Prototype
RAPID ITERATIVE PROCESS
![Page 23: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/23.jpg)
23
MATLAB and Simulink connect to the ROS network
- Multiple master support
- ROS publishers/subscribers
- ROS services
- ROS TF tree
- ROS Parameter server
?
Vehicle
![Page 24: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/24.jpg)
24
Co-simulation with ROS
![Page 25: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/25.jpg)
25
Co-simulation with ROS
![Page 26: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/26.jpg)
26
Leverage Built-in Algorithms in MATLAB
0.2 𝑠 0.2 𝑠 0.2 𝑠
time
% Sensor fusion and pose estimation
>> pf = robotics.ParticleFilter;
% Robot localization
>> mcl = robotics.MonterCarloLocalization;
% Loops at system time or ROS time
>> r = robotics.Rate(5);
% Obstacle Avoidance
>> vfh = robotics.VectorFieldHistogram;
![Page 27: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/27.jpg)
27
Optional Products for Robotics Applications Development
Image Processing
Toolbox™
Contrast adjustment
Geometric transformations
Various filters
Segmentation
Object analysis
Computer Vision System Toolbox™
High-speed video I/O
Point Cloud processing
Tracking
Stereovision
Image Acquisition Toolbox™
Image capture from standard H/W
Analog, Camera Link, DCAM,
GigE Vision, USB camera, etc
Microsoft Kinect Support
Statistics and Machine Learning
Toolbox™
Multivariate statistics
Probability distribution
Machine learning
Experimental design
Statistical process control
![Page 28: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/28.jpg)
28
Optional Products for Robotics Applications Development
Control System Toolbox™
Linear analysis
Classical
control design
Modern
control design
Simulink Design Optimization™
Model parameter estimation from
test data
Optimization of
parameters
Response
optimization
Simulink Control Design™
Automatic tuning of PID
Controller blocks
Linearization of
Simulink models
Continuous-
Discrete time
conversions
Robust Control Toolbox™
Robust control design
Automatic tuning
of gain-scheduled
controllers
![Page 29: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/29.jpg)
29
Final Design
![Page 30: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/30.jpg)
30
Robotics Development Workflow with MATLAB and ROS
System level design to target a different
middleware or framework
Implementation
Product
Requirements
Idea
Design and
Simulation
Prototype
RAPID ITERATIVE PROCESS
![Page 31: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/31.jpg)
31
Deployment
Generate ROS
Node with
Simulink and
Embedded
Coder™
Create a
Stand Alone
Executable
with MATLAB
Compiler™
Generate a
shared library
with MATLAB
Coder™
Determine deployment methods based on application
![Page 32: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/32.jpg)
32
Implementation
![Page 33: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/33.jpg)
34
Takeaways
1. Robotics development workflow with MATLAB and Simulink for autonomous driving with ROS.
2. Rapid iterative process based on powerful visualization tools, interactive apps, built-in algorithms,
debugging capabilities.
3. Design is independent of communication framework.
Call to Action! Check Demo
![Page 34: Connecting MATLAB and ROS for Autonomous Driving · Computer Vision System Toolbox™ High-speed video I/O Point Cloud processing Tracking Stereovision Image Acquisition Toolbox™](https://reader033.fdocuments.net/reader033/viewer/2022042121/5e9aa81efe38272b1245d586/html5/thumbnails/34.jpg)
35
% Thank you