Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings –...
Transcript of Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings –...
![Page 1: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/1.jpg)
1© 2020 The MathWorks, Inc.
Design 3D Scenes for Automated Driving Simulation
Shashank Sharma
Peter Fryscak
![Page 2: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/2.jpg)
2
Some common questions from automated driving engineers
How can I
analyze & synthesize
scenarios?
How can I
design & deploy
algorithms?
How can I
integrate & test
systems?
Control
Planning
Perception
![Page 3: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/3.jpg)
3
Some common questions from automated driving engineers
How can I
analyze & synthesize
scenarios?
How can I
design & deploy
algorithms?
How can I
integrate & test
systems?
Control
Planning
Perception
![Page 4: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/4.jpg)
4
Synthetic scenario workflows
Real-world data workflows
Analyze and synthesize scenarios
Connect Visualize Label
Model actors Model sensors
Enables
open loop
workflows
Enables
open loop and
closed loop
workflowsCreate scenes
![Page 5: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/5.jpg)
5
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
Update 1
![Page 6: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/6.jpg)
6
Update 1
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
![Page 7: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/7.jpg)
7
Design scenes with road, marking, and prop assets
▪ Roads and markings
▪ Traffic signals
▪ Guard rails
▪ Trees
▪ Signs
▪ Elevation data
Assets
RoadRunnerTM
Update 1
![Page 8: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/8.jpg)
8
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Edit roads
▪ Edit road materials
▪ Add road markings
Exporting to CARLA
RoadRunnerTM
Update 1
![Page 9: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/9.jpg)
9
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Install plugin
▪ Export from RoadRunner
▪ Import into CARLA/Unreal
Exporting to CARLA
RoadRunnerTM
Update 1
![Page 10: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/10.jpg)
10
Design scenes and export to driving simulator
Design
scenes
Export
meshes
Import to
simulatorSimulate
▪ Move vehicle in automated driving
simulation
▪ Visualize pixels IDs for semantic
segmentation
Exporting to CARLA
RoadRunnerTM
Update 1
![Page 11: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/11.jpg)
11
Design scenes with hundreds of premade assets
▪ Road and highway signs
▪ Traffic signals
▪ Road surface markings
▪ Trees
▪ Barriers
▪ Road damage textures
– Cracks, oil spills
Asset Library
RoadRunnerTM Asset Library
Update 1
![Page 12: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/12.jpg)
12
Export scenes to file formats and driving simulators
▪ Export to common file formats for
use in third-party applications
– Filmbox (.fbx), OpenDRIVE (.xodr)
– Unreal Engine®, CARLA
– Unity®, LGSVL
– VIRES Virtual Test Drive, Metamoto
– IPG Carmaker, Cognata, Baidu Apollo
– Tesis Dynaware, TaSS PreScan
– Universal Scene Description (USD)
Exporting
RoadRunnerTM
FBX
(meshes)
OpenDRIVE
(semantics)
Update 1
![Page 13: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/13.jpg)
13
Import, visualize, and edit OpenDRIVE files
Import
OpenDRIVEVisualize Edit Export
▪ Validate OpenDRIVE file
▪ Import and visualize
▪ Edit roads and scene
▪ Export to common driving simulator
formats (including OpenDRIVE)
Importing OpenDRIVE Files
RoadRunnerTM
Update 1
![Page 14: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/14.jpg)
14
Get started designing scenes by watching tutorial videos
Update 1
▪ Add roads and junctions
▪ Add lane markings
▪ Add traffic signals
▪ Add traffic signs
https://www.mathworks.com/videos/sear
ch.html?q=roadrunner
RoadRunnerTM
![Page 15: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/15.jpg)
15
Update 1
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
![Page 16: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/16.jpg)
16
Synthesize scenarios to test algorithms and systems
Scenes Cuboid
Testing Controls, sensor fusion, planning
Sensing Probabilistic vision (detection list)
Probabilistic lane (detection list)
Probabilistic radar (detection list)
Lidar (point cloud)
![Page 17: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/17.jpg)
17
Synthesize scenarios to test algorithms and systems
Scenes Cuboid Unreal Engine
Testing Controls, sensor fusion, planning Controls, sensor fusion, planning, perception
Sensing Probabilistic vision (detection list)
Probabilistic lane (detection list)
Probabilistic radar (detection list)
Lidar (point cloud)
Monocular camera (image, labels, depth)
Fisheye camera (image)
Probabilistic radar (detection list)
Lidar (point cloud)
![Page 18: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/18.jpg)
18
Graphically author scenarios with Driving Scenario Designer
▪ Design scenes
– Roads, Lane markings
– Pre-built scenes (Euro NCAP)
▪ Import roads
– OpenDRIVE, HERE HD Live Map
▪ Add actors
– Size, Radar cross-section (RCS)
– Trajectories
▪ Export scenarios
– MATLAB code, Simulink model
Driving Scenario Designer
Automated Driving ToolboxTM
Updated
![Page 19: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/19.jpg)
19
Synthesize driving scenarios from recorded data
▪ Import roads from OpenDRIVE
▪ Create ego trajectory from GPS
▪ Create target trajectories object lists
Scenario Generation from Recorded
Vehicle Data
Automated Driving ToolboxTM
Visualize
video
Import
roads
Create ego
trajectory
Create target
trajectories
Simulate
scenario
![Page 20: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/20.jpg)
20
Design with cuboid and Unreal Engine driving scenarios
Trajectories
Specify Vehicle Trajectories
for 3D Simulation
Automated Driving ToolboxTM
Scenes
Cuboid Versions of 3D Simulation
Scenes in Driving Scenario Designer
Automated Driving ToolboxTM
Customize scenes
Customize 3D Scenes for
Automated Driving
Automated Driving ToolboxTM
![Page 21: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/21.jpg)
21
Integrate RoadRunner with MATLAB and Simulink workflows
Unreal EngineRoadRunner
RoadRunner
scene
Export to
Unreal Engine(.FBX, .XML)
Import and
configure game
Export to
OpenDRIVE(.XODR)
MATLAB & Simulink
Simulink
model
Connect to
game
Add
vehicles
Import to
driving scenario
![Page 22: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/22.jpg)
22
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
Export to
Unreal Engine(.FBX, .XML)
Simulink
model
Import and
configure game
Connect to
game
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
RoadRunner
scene
![Page 23: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/23.jpg)
23
Create and explore scene in RoadRunner
Create
Roads
Add
guard rails
Add
trees
Change
lane markings
Change
road textures
▪ Design scene to test behavior of
lane following algorithm under
different road conditions
– Shadows, guard rails
– Lane markings, road textures
– Target vehicles
![Page 24: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/24.jpg)
24
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
RoadRunner
scene
Export to
Unreal Engine(.FBX, .XML)
Simulink
model
Import and
configure game
Connect to
game
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
![Page 25: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/25.jpg)
25
Integrate RoadRunner scene with driving scenario
Export scene to
OpenDRIVE
Import to
driving scenario
Add
vehiclesSimulate
▪ Export scene from RoadRunner to
OpenDRIVE file (.XODR)
![Page 26: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/26.jpg)
26
Integrate RoadRunner scene with driving scenario
Export scene to
OpenDRIVE
Import to
driving scenario
Add
vehiclesSimulate
▪ Create scenario with Driving
Scenario Designer App
▪ Import road network from
OpenDRIVE
▪ Add vehicle
▪ Specify trajectory waypoints and
speed
![Page 27: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/27.jpg)
27
Unreal EngineRoadRunner MATLAB & Simulink
Integrate RoadRunner with MATLAB and Simulink workflows
RoadRunner
scene
Add
vehicles
Export to
OpenDRIVE(.XODR)
Import to
driving scenario
Simulink
model
Export to
Unreal Engine(.FBX, .XML)
Import and
configure game
Connect to
game
![Page 28: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/28.jpg)
28
Integrate RoadRunner scene with Unreal Engine scenario
▪ Export scene from RoadRunner to Unreal
Engine game (.FBX & .XML)
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
![Page 29: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/29.jpg)
29
Integrate RoadRunner scene with Unreal Engine scenario
▪ Import game into Unreal Engine
▪ Add plugin to communicate with Simulink
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
![Page 30: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/30.jpg)
30
Integrate RoadRunner scene with Unreal Engine scenario
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
▪ Create Simulink model
▪ Configure connection to Unreal Engine
▪ Read driving scenario
▪ Add vehicle
▪ Add sensors
Scenario
reader
Sensors
VehicleGame
configuration
![Page 31: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/31.jpg)
31
Integrate RoadRunner scene with Unreal Engine scenario
Export to
Unreal Engine
Configure
Unreal Engine
Connect to Unreal Engine
and driving scenario Simulate
▪ Create Simulink model
▪ Configure connection to Unreal Engine
▪ Read driving scenario
▪ Add vehicle
▪ Add sensors
![Page 32: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/32.jpg)
32
Model sensors in Unreal Engine driving scenarios
▪ Monocular camera
– Image
– Depth
– Labels
▪ Fisheye camera image
▪ Lidar point cloud
▪ Radar detections
3D Simulation for Automated Driving
Automated Driving ToolboxTM
Updated
Monocular image Lidar
Radar
FisheyeLabels
Depth
![Page 33: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/33.jpg)
33
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Integrate algorithms
– Vision detection
– Sensor fusion
– Decision logic and controls
– Use MathWorks solutions or your ownScenario
&
sensors
Vehicle
dynamics
Vision
detection
Sensor
fusion
Decision
logic
& controls
![Page 34: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/34.jpg)
34
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Create driving scenario from scene
▪ Read driving scenario
▪ Configure Unreal Engine game
▪ Add ego and target vehicles
▪ Connect sensors to ego vehicle
Scenario
Reader
Game
configuration
Vehicles
Sensors
![Page 35: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/35.jpg)
35
Integrate RoadRunner scene with lane following scenario
Explore highway lane
following system
Integrate with driving scenario
and Unreal EngineSimulate
▪ Explore system behavior under different
road conditions
– Shadows, guard rails
– Lane markings, road textures
– Target vehicles
![Page 36: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/36.jpg)
36
External Simulators MATLAB & Simulink
Design 3D scenes for automated driving simulation
Update 1
![Page 37: Design 3D Scenes for Automated Driving Simulation · Design scenes – Roads, Lane markings – Pre-built scenes (Euro NCAP) Import roads – OpenDRIVE, HERE HD Live Map Add actors](https://reader036.fdocuments.net/reader036/viewer/2022071210/602214faa547ad557f6de7b0/html5/thumbnails/37.jpg)
37
Let us know how we can help you…
analyze & synthesize
scenarios
design & deploy
algorithms
integrate & test
systems
Control
Planning
Perception
Contact us : [email protected]