Emotional awareness in autonomous driving...Emotional awareness in autonomous driving Challenges,...

Emotional awareness in autonomous driving

Challenges, Approaches & Vision

Luis Gressenbuch, Sebastian Bergemann

Technical University of Munich

Department of Informatics

[email protected], [email protected]

Seminar Emotional awareness in autonomous driving SS2019

Munich, Jun. 28th 2019

Outline

2Luis Gressenbuch (TUM), Sebastian Bergemann (TUM) | Emotional Awareness in Autonomous Driving – Challenges, Approaches and Vision

IntroductionEmotion

recognition

Emotion-aware

applicationsin AD

Challenges Conclusion

Memory

Attention

Problem solving

Drivingpleasure

Decisionmaking

Motivation


Impacts of emotions on the driving task

Source: Affectiva

Safety Comfort

Motivation


What are the goals of emotional awareness in autonomous driving?

Emotion classification


Basic Emotions (Ekman) Circumplex Model (Russell & Barrett)

[1] [2]

• Relationship between arousal and cognitive

performance

• Low arousal results in sleepiness

• High arousal results in stress

• Optimal level depends on the difficulty of the

task

The Yerkes-Dodson law


[3]

IntroductionEmotion

recognition

Emotion-aware

applications in AD


Outline


A. Facial expressions

B. Thermal imaging

C. Physiological signals

D. Driving behavior

E. Speech

F. Gesture, head pose & eye gaze

G. Multimodal fusion

Approaches


Emotion recognition – A pattern recognition task


ConventionalPre-

processingSegmentation

Feature extraction

Classification Tracking

Deep Learning

• Six basic emotions

• Facial Action Coding System (FACS)

▪ 28 Action Units

▪ Based on the anatomy of facial muscle groups

▪ Action Units can be present with different

intensity

▪ Emotions correlate with combinations of Action

Units

• Measurement:

▪ Electromyography

▪ Camera

Approaches – A) Facial expressions


[4]

Features

Geometricfeatures

Spatial

Temporal

Appearancefeatures

Global

Local



[5]

[6] [7]

[7]

• Deep learning derives features from training data

• Visualization of learned features showed

similarities to Action Units

+ Features do not need to be constructed manually

+ Better accuracy

- Time consuming optimization of hyperparameters

- Needs higher amount of computational resources



Deep Learning

[8]

• Illumination

➢ Near-infrared cameras

• Head pose

➢ Stereo cameras

• Occlusion

• Mock expressions

• Emotions sharing the same expression



Challenges

[9]

[9][9]

• Local temperature from arousal or muscle activity

• Similar methods as in visual imaging

• Facial Thermal Feature Points (FTFPs)

+ Robust towards illumination

+ Facilitated segmentation

+ Resistant against manipulation

- Thermodynamic effects

- High cost

Approaches – B) Thermal imaging


[10]

Source: Flir

• Signals:

▪ Electrocardiography (ECG) /

Photoplethysmogram (PPG)

▪ Electrodermal activity (EDA)

▪ Skin temperature (SKT)

• Single signal can indicate arousal

• Multiple differential signals can indicate basic

emotions

Approaches – C) Physiological signals


Autonomic Nervous System

[11]

• Electroencephalography (EEG)

• Measures local electrical activity of the brain

• Most common feature: Power spectral density

at different frequency bands

• Circumplex model



Central Nervous System

[12]

Theta Alpha Beta Gamma

Arousal

Valence

• Need contact-based sensor equipment

➢ Smart wearables

➢ Highly integrated sensors

• Signal quality



Challenges

[13]

• Distracted drivers

▪ drive slower

▪ steer less frequently

▪ steer with higher angles

• Signals

▪ Vehicle speed

▪ Steering wheel angle

▪ Throttle position

▪ Lateral deviation from the lane

▪ Relative vehicle kinematics

+ Availability of signals from CAN-Bus

- Indirect measurement of emotions

- Driving behaviour not available when control is automated

Approaches – D) Driving behavior


[14]

Approaches – E) Speech


[15]

ASR systems

Linguistic way:

• Speech recognition systems

• Databases with keywords linked to emotions

▪ Match all words to hyper-classes

▪ Match most beneficial words to specific emotions

• Challenges:

▪ False word detection

▪ Context complexity



[15]

Non-Linguistic way:

• Features:



Feature categories

Low-level descriptors (LLDs)

Prosodic

speed rate

pitch

pause

Spectral

amplitude

MFCCs

short energy

Functionals

extreme values

means

offset

[15]

Non-Linguistic way:

• Challenges:

▪ Differences (language, culture, age, gender, etc.)

▪ Acted comparison material

▪ Signal-Noise-Ratio of audio recordings

▪ Not available all the time



[15]

Approaches – F) Gesture, head pose & eye gaze


[16]

• Detection:

▪ Wearable sensors (accelerometers or body markers)

▪ Cameras and computer vision

• Features:

▪ Raw features (3D data points)

▪ Velocity, acceleration and fluidity

▪ Quantity of motions (QoM)

▪ Contraction index (CI)

▪ Specific movements/positions like PERCLOS, etc.



[16]

• Challenges:

▪ Car interior limitations (movement space)

▪ Few body expressions – broader emotion dimensions

▪ Either intrusive (markers) or more noisy (without them)

➢ Not a stand-alone but a support modality



[16]

Approaches – G) Multimodal fusion


[17]

One modality:

• Sensitive to noisy conditions, occlusion, etc.

• Can fail completely (recording problem)

Multiple modalities:

• Redundancy

• Diversity

➢ Increasing accuracy and robustness

Fusion variants:

• On feature-level

• On decision-level

Approaches – G) Multimodal fusion


[17]

IntroductionEmotion

recognition

Emotion-aware

applications in AD


Outline


A. Adapting driving related behavior

B. Improving the driver state

C. Adapting the HMI

• Distractions are a major reason for accidents

• Level 2: Driver constantly needs to monitor the vehicle

• Level 3: Distractions can lead to a loss of

situational awareness

➢ Increased takeover times

• Modalities:

▪ Eye gaze

▪ Head pose

▪ Facial expressions

▪ Driving behavior

• Proposed application: Lane Departure Warning (LDW)

Applications – A) Adapting driving related systems


Distraction detection

Distraction

Cognitive

Visual Manual

• Driver state adaptive forward collision warning

(FCW) system

• System only warns if 𝑑𝑤 < 𝑑𝑟𝑒𝑙• Result: +10% accuracy, +40% precision

• Benefits:

+ Less false positives

+ Increased safety



Reaction-time estimation on cognitive workload

[18]

[17]

• In a study drivers preferred second to the fastest,

routes that impose the least stress

• How to identify stressing routes?

➢ Database of geo-tagged heart rate variability

recordings of drivers

➢ Creates a heat map



Routing

[19]

• “Happy drivers are better drivers”

• Recognizing negative emotions:

➢ Influencing the driver‘s emotional state so that

it becomes more positive

Applications – B) Improving the driver state


In general

[20]

1. Driver state display

Emotional feedback

2. Voice-based HMI

Calming down and objective reasoning

3. Subliminal influencing

Overall atmosphere (temperature, light, music)



Proposed strategies

[20]

• 1,940 accidents with personal injury due to drowsy

driving in 2017 in Germany

• Drowsiness can be induced by driving autonomously

for a long time period

• Detection can support takeover management and

notification / suggestion system



Drowsiness detection

Source: Daimler Global Media Site

• Driver assistance to increase comfort and safety

• Natural communication (contextual)

➢ Driver recognizes digital emotions

• Inconsistent pairing of emotions can be a safety

hazard

➢ VICO has to be empathic

Applications – C) Adapting HMI


Virtual co-driver (VICO)

Source: BMW

• Notifications can critically increase driver‘s stress

level in complex situations

• Suppressing not immediately relevant notifications

based on current stress level

• Supported by improvement predictions

➢ Reduce annoyance and additional stress

Applications – C) Adapting HMI


Stress-adaptive notifications

Source: BMW

IntroductionEmotion

recognition

Emotion-aware

applications in AD


Outline


Research

• Limited comparison possibility

• Many applications in pilot status

• Insufficient field studies (regarding reliability)

Emotions

• Emotion definition and recognition is difficult even for humans

• Emotion differences based on context, culture, age, gender, etc.

Applications

• User acceptance unclear

• Constant monitoring

• Privacy concerns

• Ethical concerns

General challenges


IntroductionEmotion

recognition

Emotion-aware

applications in AD


Outline


• Many approaches for emotion recognition

➢ specific limitations for autonomous vehicle field

• Applications for autonomous driving are available

➢ mainly in early stage

➢ currently refer more to ADAS than to autonomous vehicles

• Challenges exist

➢ first solutions have been published

Conclusion


Source: Affectiva

More autonomous

vehicle research

More emotion awareness

research in this field

[1] O. Spindler, “Affective space interfaces,” Diplomarbeit, TU Wien, Wien, 2009.

[2] J. A. Russell and L. F. Barrett, “Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant,” J. Pers. Soc. Psychol., vol. 76, no. 5, pp. 805–819, May 1999.

[3] A. Saeed, S. Trajanovski, M. van Keulen, and J. van Erp, “Deep Physiological Arousal Detection in a Driving Simulator Using Wearable Sensors,” in 2017 IEEE International Conference on Data Mining Workshops

(ICDMW), New Orleans, LA, 2017, pp. 486–493.

[4] E. Vural, M. Çetin, A. Erçil, G. Littlewort, M. Bartlett, and J. Movellan, “Machine Learning Systems for Detecting Driver Drowsiness,” in In-Vehicle Corpus and Signal Processing for Driver Behavior, K. Takeda, H.

Erdogan, J. H. L. Hansen, and H. Abut, Eds. Boston, MA: Springer US, 2009, pp. 97–110.

[5] C. F. Benitez-Quiroz, R. Srinivasan, and A. M. Martinez, “EmotioNet: An Accurate, Real-Time Algorithm for the Automatic Annotation of a Million Facial Expressions in the Wild,” in 2016 IEEE Conference on

Computer Vision and Pattern Recognition (CVPR), 2016, pp. 5562–5570.

[6] D. Ghimire and J. Lee, “Geometric Feature-Based Facial Expression Recognition in Image Sequences Using Multi-Class AdaBoost and Support Vector Machines,” Sensors, vol. 13, no. 6, pp. 7714–7734, Jun. 2013.

[7] C. Shan, S. Gong, and P. W. McOwan, “Facial expression recognition based on Local Binary Patterns: A comprehensive study,” Image Vis. Comput., vol. 27, no. 6, pp. 803–816, May 2009.

[8] R. Breuer and R. Kimmel, “A Deep Learning Perspective on the Origin of Facial Expressions,” ArXiv170501842 Cs, May 2017.

[9] A. Yüce, H. Gao, G. L. Cuendet, and J. Thiran, “Action Units and Their Cross-Correlations for Prediction of Cognitive Load during Driving,” IEEE Trans. Affect. Comput., vol. 8, no. 2, pp. 161–175, Apr. 2017.

[10] C. Puri, L. Olson, I. Pavlidis, J. Levine, and J. Starren, “StressCam: Non-contact Measurement of Users’ Emotional States Through Thermal Imaging,” in CHI ’05 Extended Abstracts on Human Factors in Computing

Systems, New York, NY, USA, 2005, pp. 1725–1728.

[11] T. K. L. Hui and R. S. Sherratt, “Coverage of Emotion Recognition for Common Wearable Biosensors,” Biosensors, vol. 8, no. 2, Mar. 2018.

[12] S. Koelstra et al., “DEAP: A Database for Emotion Analysis ;Using Physiological Signals,” IEEE Trans. Affect. Comput., vol. 3, no. 1, pp. 18–31, Jan. 2012.

[13] A. Riener, M. Jeon, I. Alvarez, and A. K. Frison, “Driver in the Loop: Best Practices in Automotive Sensing and Feedback Mechanisms,” in Automotive User Interfaces: Creating Interactive Experiences in the Car, G.

Meixner and C. Müller, Eds. Cham: Springer International Publishing, 2017, pp. 295–323.

[14] S. Choi, J. Kim, D. Kwak, P. Angkititrakul, and J. Hansen, “Analysis and Classification of Driver Behavior Using in-Vehicle CAN-BUS Information,” Bienn Workshop DSP -Veh Mob Syst, Jan. 2007.

[15] C.-N. Anagnostopoulos, T. Iliou, and I. Giannoukos, “Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011,” Artif. Intell. Rev., vol. 43, no. 2, pp. 155–177, Feb. 2015.

[16] H. A. Vu, Y. Yamazaki, F. Dong, and K. Hirota, “Emotion recognition based on human gesture and speech information using RT middleware,” in 2011 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE

2011), 2011, pp. 787–791.

[17] L. Kessous, G. Castellano, and G. Caridakis, “Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis,” J. Multimodal User Interfaces, vol. 3, no. 1,

pp. 33–48, Mar. 2010.

[18] V. Govindarajan, K. Driggs-Campbell, and R. Bajcsy, “Affective Driver State Monitoring for Personalized, Adaptive ADAS,” in 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018,

pp. 1017–1022.

[19] S. D. Nadai et al., “Enhancing safety of transport by road by on-line monitoring of driver emotions,” in 2016 11th System of Systems Engineering Conference (SoSE), 2016, pp. 1–4.

[20] M. Braun, J. Schubert, B. Pfleging, and F. Alt, “Improving Driver Emotions with Affective Strategies,” Multimodal Technol. Interact., vol. 3, no. 1, p. 21, Mar. 2019

References


Emotional awareness in autonomous driving...Emotional awareness in autonomous driving Challenges,...

Documents

Transcript of Emotional awareness in autonomous driving...Emotional awareness in autonomous driving Challenges,...