Smart Cars for Safe Driving - intranet.ceautomatica.es
Transcript of Smart Cars for Safe Driving - intranet.ceautomatica.es
![Page 1: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/1.jpg)
Smart Cars for Safe Driving
Prof. Dr. Dariu M. GavrilaEnvironment Perception
Group Research and Advanced Engineering
XXXII Jornadas de Automática, Sevilla, 9-9-2011
![Page 2: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/2.jpg)
2
We originally thought Machine Intelligence would look like
1956 "Forbidden Planet"
Robby the Robot (Flickr)
![Page 3: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/3.jpg)
3
Then more recently, some suggested it would be more like
2004 iRobot 1983-2009 Terminator
![Page 4: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/4.jpg)
4
when in fact, Machine Intelligence is already with us, and has a familiar embodiement
![Page 5: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/5.jpg)
5
Driver Assistance in the current Mercedes Benz E-Class
Speed Limit
Adaptive
High Beam
PRE-SAFE
Lane Keeping
Blind Spot
Nightview
Plus
Attention
![Page 6: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/6.jpg)
6
•Technology is rapidly expanding the capabilities of modern vehicles.
•One breakthrough development over the past few years is the emergence of
driver assistance systems.
•Use of sensor systems which continuously monitor vehicle surroundings and
interior, provide information to the driver, and even perform vehicle control.
•Help drivers operate their vehicles in a safe, comfortable, and energy-
efficient manner.
• Enables market differentiation for vehicle manufacturers
Driver Assistance
![Page 7: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/7.jpg)
7
What got us here: Sensors
Better and cheaper.
Radars Cameras Laser Scanners
![Page 8: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/8.jpg)
8
What got us here: Computational Power
CPU performance over time
MFlo
ps
in m
yve
hciles
10
Processing Power over Time
time
GFLOPS/MIPS
1990 2000 2010
100
200
300
400
500
ASIC
FPGA (Xilinx)
GPU (NVidia)
CPU (Intel)
Transputer/x86 P4
Core2Duo
G92
G80
G70
NV40
Virtex 5
Spartan3
Virtex 4
Tyzx
IQ2
Standford engine
3DIP
Processing Power over Time
time
GFLOPS/MIPS
1990 2000 2010
100
200
300
400
500
ASIC
FPGA (Xilinx)
GPU (NVidia)
CPU (Intel)
Transputer/x86 P4
Core2Duo
G92
G80
G70
NV40
Virtex 5
Spartan3
Virtex 4
Tyzx
IQ2
Standford engine
3DIP
*1,78/a
106
Prognosis 2030:
optimistic (1.78/a): 100 PFlops
pessimistic (1.41/a): 1 PFlops
(Still) exponentially increasing.
![Page 9: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/9.jpg)
9
Next Challenge: Active Pedestrian Safety
Pedestrian are the most vulnerable traffic
participants. Children are particularly at risk.
Driver inattention and/or bad visibility are
important accident causes.
Worldwide fatalities of
pedestrians, bicyclists,
and motorcyclists (2006)
Source: Bosch Accident Research
![Page 10: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/10.jpg)
10
Why is it difficult?
Large variation in pedestrian appearance (viewpoint, pose, clothes).
Dynamic and cluttered backgrounds.
Pedestrians can exhibit highly irregular motion.
Real-time processing required.
Stringent performance requirements (especially for emergency maneuvres).
![Page 11: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/11.jpg)
11
Pedestrian System Architecture
Object
Classification
Object
Classification TrackingTracking
Driver Warning /
Vehicle Control
Driver Warning /
Vehicle Control
Path Prediction
&
Risk Assessment
Path Prediction
&
Risk Assessment
Obstacle Detection(Stereo, Flow, Radar)
Obstacle Detection(Stereo, Flow, Radar)
The benefit of object classification:
•improved detection reliability vs. obstacle detection only
•better path prediction: taking advantage of prior knowledge of
object class motion and additional object class-specific cues
•allows object class-specific driver warning and vehicle control strategies
D. M. Gavrila and S. Munder. Multi-Cue Pedestrian Detection and Tracking from a Moving Vehicle. IJCV 73(1), 2007.
S. Munder, C. Schnörr and D.M. Gavrila. Pedestrian Detection and Tracking Using a Mixture of View-Based Shape-Texture Models.
IEEE Trans. on Intelligent Transportation Systems, vol.9, nr.2, pp.333-343, 2008.
C. Keller, T. Dang, A. Joos, C. Rabe, H. Fritz, and D.M. Gavrila. Active Pedestrian Safety by Automatic Braking and Evasive Steering,
IEEE Trans. on Intelligent Transportation Systems, 2011
![Page 12: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/12.jpg)
12
A. Wedel, C.Rabe, T. Vaudrey, T. Brox, U.Franke, D.Cremers.
“Efficient Dense Scene Flow from Sparse or Dense Stereo Data”. ECCV 2008.
stereo
stereo
optical flowoptical flow
ltyxI )1,,( −
ltvyuxI ),,( ++
ltvyuxI ),,( ++=
rtvyddduxI ),,( ++++=
rtvyddduxI ),,( ++++=
rtydxI )1,,( −+
3D Position and Motion for Every Pixel (Scene Flow)
Joint Optimization
Motion
time
t
time
t-1
![Page 13: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/13.jpg)
13
Scene Flow
![Page 14: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/14.jpg)
14
Pedestrian Classification – Experimental Studies
What features? E.g. Chamfer, Haar wavelets, HOG, and Local Receptive Field
What pattern classifier? E.g. SVM, Neural Networks
How to combine pattern classifiers? E.g. Cascading, Parallel (Sum/Max/Mixture)
How to deal with occlusion?
Haar wavelets + AdaBoost cascade
[Viola & Jones, 2005]
HOG features + linear SVM
[Dalal & Triggs, 2005]
Local receptive fields + NN
[Wöhler & Anlauf, 1999]
![Page 15: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/15.jpg)
15
Daimler Pedestrian Benchmark Data Sets
1. Mono Pedestrian ClassificationS. Munder and D. M. Gavrila. An Experimental Study on Pedestrian Classification.
IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.28, no.11, pp. 1863-1868, 2006.
2. Multi-Modal / Occluded Pedestrian ClassificationM. Enzweiler, A. Eigenstetter, B. Schiele and D. M. Gavrila. Multi-Cue Pedestrian Classification with
Partial Occlusion Handling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
3. Mono/Stereo Pedestrian DetectionM. Enzweiler and D. M. Gavrila. Monocular Pedestrian Detection: Survey and Experiments.
IEEE Trans. on Pattern Analysis and Machine Intelligence, vol.31, no.12, pp.2179-2195, 2009.
C. Keller, M. Enzweiler, and D. M. Gavrila. A New Benchmark for Stereo-based Pedestrian Detection.
Proc. of the IEEE Intelligent Vehicles Symposium, Baden-Baden, Germany, 2011.
Training: 14400 peds. / 15000 non-peds.
Test: 9600 peds. / 10000 non-peds. All 18x36 pixel.
Available for download (Google)
Training: 15660 peds. / 6744 non-ped images
Test: 21790 images with 259 ped. trajectories
>130.000 samples (intensity, dense stereo,
dense flow), 48x96 pixel
1. 2. 3.
![Page 16: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/16.jpg)
16
An intriguing question …
ROC performance improves with
enlarged training set. No saturation
effects (even) for N = 12.800
In fact, doubling training size
matters more than selecting
the best feature-classifier
combination.
How many image examples are needed to learn pedestrian appearance?
Manually labeling humans in images is time-consuming and tedious!Can we do better?
![Page 17: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/17.jpg)
17
Generating Virtual Pedestrians
Shape
variation
Texture
variation
M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.
![Page 18: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/18.jpg)
18
Mixed Generative-Discriminative Classification Framework
Enlarged training set significantly improved classification
performance (30% less false positives at equal true positive rate)
Meanwhile, current pedestrian classifier on-board vehicle uses
more than 1.5 million samples (“real” and “virtual”)
M. Enzweiler and D. M. Gavrila. A Mixed Generative-Discriminative Framework for Pedestrian Classification. CVPR 2008.
![Page 19: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/19.jpg)
19
Pedestrian Detection - Daytime (Videoclip)
![Page 20: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/20.jpg)
20
Pedestrian Detection – Nighttime (Videoclip)
![Page 21: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/21.jpg)
21
Now with dense stereo …
![Page 22: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/22.jpg)
22
EU WATCH-OVER (2008)85%
50 km/h
10
Pedestrian Recognition Performance (Historical Perspective)
Correctly recognizedpedestrians
Number of falsely recognized pedestrian trajectories per hour
100%
50%
10 10000 100100
EU SAVE-U (2005)
65%40 km/h
EU PROTECTOR (2003)
40%
600
30 km/h
N.B. # False alarms per hour << # Falsely recognized trajectories per hour
We need to get
somewhere here Source: EU Final Review WATCH-OVER
![Page 23: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/23.jpg)
23
lateral position
(m)
longitudinal position
(m)
featu
re
dim
ensi
on
lateral position
longitudin
al posi
tion
(m)
Pedestrian Path Prediction by Trajectory Matching
Our approach
Use higher order model; match learned
trajectory “snippets” (segment of fixed length).
QRLCS (Hermes et al. IV’09) metric computes
similarity after alignment (translation/rotation).
Use of additional motion features.
Path prediction by extrapolation of matched
trajectory snippets (non-param. regression).
Use of particle filter representation.
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
main mode
traj.
prediction
aligned snippet distribution
system trajectory
State-of-the-art path prediction: Kalman filter-
ing based on position detected bounding box.
Problem: first-order model does not capture
non-linearities well during sudden motion
changes.
![Page 24: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/24.jpg)
24
Path Prediction
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
![Page 25: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/25.jpg)
25
Action Classification (Crossing or not)
Predicting the correct pedestrian’s
action with accuracy 80% is
reached:
•570 ms before a possible standstill
by the human (cyan).
•180 ms before a possible standstill
by the proposed system (black).
•only after the possible standstill by
the IMM-KF (pink).
Motion features help.
1 Frame ≈ 45ms
C. Keller, C. Hermes and D. M. Gavrila. Will the pedestrian cross? DAGM 2011 Prize.
![Page 26: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/26.jpg)
26
World Premiere (2009): Automatic Evasion on Pedestrians
![Page 27: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/27.jpg)
27
Automated Test Driving (Videoclip)
Source: Daimler Testing Department
![Page 28: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/28.jpg)
28
Understand What Is What
Localize and classify objects in the environment
Background
Street
Pedestrian
Sky
Vision
Moving vehicle
Source: U. Franke
![Page 29: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/29.jpg)
29
Driver Monitoring
Head / Face / Gaze Tracking Mindlab
Head / Face tracking using stereo vision
and Active Appearance Models
Driver intention estimation based on
head motion, gaze, and vehicle trajectory
Online EEG analysis of driver mental
state (work load, fatigue)
Use to objectively evaluate driver
assistance systems (Attention, IHC)
![Page 30: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/30.jpg)
30
Automation Systems: Gradually Getting There
Not certifiable todayTraditional
driving
All on
Today‘s
ACC
Assisted 1st
Feet off
Short
takeover times
Assisted 2nd
Hands off
Moderate
takeover times
Autonomous 1st
Eyes off
Ability to
drive empty
Autonomous 2nd
Body out
Source: R. G. Herrtwich
![Page 31: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/31.jpg)
31
Final Remarks
Driver assistance is experiencing a breakthrough: a first major deployment of
machine intelligence technology (sensing, reasoning, acting in physical environment).
Computer vision and machine learning play a central role.
Trend towards increased actuation of safety systemsDriver information � driver warning � “soft” vehicle actuation / driver-initiated “hard”
vehicle actuation � automatic “hard” vehicle actuation
Environment Perception is still the bottleneck. Need to
• recognize a wider set of traffic objects classes with better classification performance
• localize objects more accurately in 3D (perform segmentation and classification jointly).
• handle adverse visibility conditions
Future systems will fuse data from lots of sensors and build a precise 3D-
representation of the 360° car surrounding.
The progress in environment perception, driver monitoring, communication as
well as in precise 3D map data will bring us close to our vision of
accident free driving.
![Page 32: Smart Cars for Safe Driving - intranet.ceautomatica.es](https://reader031.fdocuments.net/reader031/viewer/2022020911/6201931c7c21144a512835da/html5/thumbnails/32.jpg)
32
The best is yet to come!
Questions ?