Post on 11-Jan-2016
Automated Drowsiness Detection For Improved
Driving Safety
Aytül ErçilNovember 13, 2008
Outline
Problem Background and Description Technological Background Action Unit Detection Drowsiness Prediction
Objectives/Overview•Statistical Inference of fatigue Using Machine Learning Techniques
In over 500.000 accidents in 2005 (in Turkey): Injured: 123,985 people
Deceased: 3,215 people
Financial loss: 651,166,236 USD
Number of accidentsı
0
100.000
200.000
300.000
400.000
500.000
600.000
19
80
19
82
19
84
19
86
19
88
19
90
19
92
19
94
19
96
19
98
20
00
20
02
20
04
(*)
A. Kaza B. Ölü C. Yaralı D.Maddi Hasar Miktarı (ABD $) A
Driver error has been blamed as the primary cause for approximately 80% of these traffic accidents.
The US National Highway Traffic Safety Administration estimates that in the US alone approximately 100,000 crashes each year are caused primarily by driver drowsiness or fatigue
Growing Interest In Intelligent Vehicles US Department of Transportation Initiative European Transport Policy for 2010: set a
target to halve road fatalities by 2010.
Problem Background
The Drivesafe ProjectThe Drivesafe Project
Current Funding Status:
• Turkish Development Agency funding of Drive-Safe (August 2005-July. 2009)
• Japanese New Energy and Industrial Technology Development Organization (NEDO) (October 2005 -December 2008)
• FP6 SPICE Project at Sabancı University (May 2005- October 2008)
• FP6 AUTOCOM Project at ITU Mekar (May 2005- April 2008).
Readiness-to-perform Mathematical models of alertness dynamics Vehicle-based performance technologies (Vehicle
Speed, Lateral Position, Pedal Movement) In-vehicle, on-line, operator status monitoring
technologies
Fatigue Detection and Prediction TechnologiesFatigue Detection and Prediction Technologies
Physiological Signals (heart rate, pulse rate and Electroencephalography (EEG))
Computer Vision Systems (detect and recognize the facial motion and appearance changes occurring during drowsiness)
In-vehicle, on-line, operator status monitoring In-vehicle, on-line, operator status monitoring technologiestechnologies
Computer Vision SystemComputer Vision Systemss
Visual BehaviorsVisual Behaviors• ExamplesExamples
Gaze DirectionGaze Direction Head MovementHead Movement YawningYawning
• No requirement for physical contactNo requirement for physical contact
Facial Actions
Ekman & Friesen, 1978
Background Information- Background Information- Action UnitsAction Units
Proposed WorkProposed Work
Detection Of Driver Fatigue From A Recorded Video Using Facial Appearance Changes
The framework will be based on graphical models and machine learning approaches
Proposed ArchitectureProposed Architecture
Sensing ChannelsEye Tracker
AU 61
Pupil Motion
AU 62
Gaze Tracker
Gaze
AU 51 AU 52
Eye Tracker
AU 61
Pupil Motion
AU 62
GazeTracker
Gaze
AU 51 AU 52
Features
Time n-1 Time n
Inattentive Falling Asleep
Fatigue
Inattentive Falling Asleep
Fatigue
Entire Face Behavior
Partial Face Behavior
Single AU
Action Unit TrackingAction Unit Tracking
Previous techniques Previous techniques Do not employ a spatially and temporally Do not employ a spatially and temporally
dependent structure for Action Unit Trackingdependent structure for Action Unit Tracking Contextual information is not exploitedContextual information is not exploited Temporal information is not exploitedTemporal information is not exploited
Classification- ChallengesClassification- Challenges
Which action units or combinations is a cue for fatigue?
Learning from real examples
Posed Drowsiness
Actual Drowsiness
Different Neural pathways for posed/spontaneous expressions
Initial Experimental Setup
Subjects played a driving video game on a windows machine using a steering wheel and an open source multi-platform video game. At random times, a wind effect was applied that dragged the car to the right or left, forcing the subject to correct the position of the car.
Head movement measures
Head movement was measured using an accelerometer that has 3 degrees of freedom. This three dimensional accelerometer has three one dimensional accelerometers mounted at right angles measuring accelerations in the range of 5g to +5g
The one minute preceding a sleep episode or a crash was identified as a non-alert state. There was a mean of 24 non-alert episodes with a minimum of 9 and a maximum of 35.
Fourteen alert segments for each subject were collected from the first 20 minutes of the driving task.
Crash
Overcorrection
Seconds0 20
Steering
Distance from center
Eye openingEyes closed
Histograms for Eye Closure and Eye Brow Up
Eye Closure: AU45 Brow Raise:AU2 Area under the ROC
Pattern Recognition(Adaboost)
(SVM)
FeatureSelection
Machine Learning
Facial Action Unit Detection
AU1
AU2
AU4
….
….
AU46
++
Drowsiness Prediction
The facial action outputs were passed to a classifier for predicting drowsiness based on the automatically detected facial behavior.
Two learning-based classifiers,
Adaboost and multinomial logistic regression are compared.
Within-subject prediction of drowsiness and across-subject (subject independent) prediction of drowsiness were both tested.
Classification Task
Multinomial Logistic
Regression (MLR)
Frame
Alert
60 secBeforecrash
:
• 31 Facial Action Channels• Continuous output for each frame
AU1
AU2
AU4
AU31
Testing: MLR Weighted Temporal Windows
Within subject drowsiness prediction
For the within-subject prediction, 80% of the alert and non-alert episodes were used for training and the other 20% were reserved for testing.
This resulted in a mean of 19 non-alert and 11 alert episodes for training, and 5 non-alert and 3 alert episodes for testing per subject.
Across Subject Drowsiness Prediction
Training : 31 actions -> MLR Classifier Framewise training
Cross validation: 3 subjects –> training 1 subject –> testing
Crash prediction:• choose 5 best features by sequential feature selection• Sum MLR weighted features over 12 second time interval• .98 across subjects (Area under the ROC)
More when critically drowsy
Eye Closure Brow Raise Chin Raise Frown Nose Jaw Wrinkle Sideways
Predictive Performance of Individual Facial Actions
Predictive Performance of Individual Facial Actions
Less when critically drowsy
Smile Squint Nostril Brow Lower Jaw Drop
CompressorA’ > .75
We observed during this study that many subjects raised their eyebrows in an attempt to keep their eyes open, and the strong association of the AU 2 detector is consistent with that observation.
Also of note is that action 26, jaw drop, which occurs during yawning, actually occurred less often in the critical 60 seconds prior to a crash. This is consistent with the prediction that yawning does not tend to occur in the final moments before falling asleep.
Drowsiness detection performance, using an MLR classifier with different feature combinations.
Effect of Temporal Window Length
* 12 secondsA’
Seconds
Coupling of Facial Movements
ALERT DROWSY
Eye Openness
Brow Raises
Brow RaisesBrow Raise
Eye ClosureBrow Raise
Eye Closure
r=0.87
0 Seconds 10 10Seconds0
Coupling of Steering and Head Motion
ALERT DROWSY
r=0.27
r=0.65
Steering
Head Acceleration
Head Acceleration
SteeringSeconds 60 600 0 Seconds
Coupling of Steering and Head Motion
New associations between facial behavior and drowsiness
• Brow raise• Chin raise• More head roll• Possibly less yawning just before crash
• Coupling of behaviors– Head movement and steering– Brow raise and eye opening
Future WorkFuture Work
Extend the graphical model so that it captures the Extend the graphical model so that it captures the temporal relationships using a discriminative temporal relationships using a discriminative approachapproach
Future Work: More Data Collection in Future Work: More Data Collection in Simulator Environment Simulator Environment
Uykucu (Sleepy)
Thank you