SAVE-IT

SAfety VEhicles using adaptive Interface TechnologyPhase 1 Research Program

Quarterly Program Review

Task 5: Cognitive Distraction

John D. LeeUniversity of Iowa

April 19, 2023

SAVE-IT Task Description

Team Members – John Lee (lead)– Daniel McGehee– Michelle Reyes– Tara Smyser

– Yulan Liang

Objectives– Develop an equation relating driving performance, driver state, and vehicle

system state variables associated with cognitive distraction to reaction time

Deliverables– A report based on the literature review, the identification of potential measures of

cognitive distraction, and an updated task definition document (for Task 5A)– A final report that describes the design, research methodology, data analysis,

resultant data, and recommendations, including diagnostic measures and preliminary algorithms expressing the relationship between RTC and oculometric and performance measures (for Task 5B)

SAVE-IT Task Description

Schedule Task 5.A Literature review– 5.A.1 Review the relationship between eye movements, ocular response, scan

patterns, and other psycho-physiological measures and cognitive load (complete)

– 5.A.2 Review range of driving tasks and develop a theory-based taxonomy of different subtypes of cognitive distraction (complete)

– 5.A.3 Identify neuropsychological tests that tap into the various cognitive demands identified in Task 5.A.2 (complete)

– 5.A.4 Develop a conceptual, and potentially, a computational model of driver response to the cognitive demands of driving and in-vehicle device interactions (in progress)

SAVE-IT Task description

Schedule Task 5.B Data collection and algorithm development

– 5.B.1 Identify a small set of neuropsychological tasks for implementation in a driving scenario (in progress)

– 5.B.2 Develop scenarios and driving performance measures (in progress)

– 5.B.3 Collect data and evaluate the relationship between driver performance and behavior and degree of cognitive distraction (in progress)

SAVE-IT Literature Review: Task 5A

Research Areas of Interest– Empirical and theoretical descriptions of driver distraction– Theoretical basis for multitask performance and task management– Measures that might predict distraction– Algorithms for combining data to predict distraction

Key Source Material

– Boer, E. R. (2001). Behavioral entropy as a measure of driving performance. Proceedings of the First Biannual Driving Assessment Conference, Aspen, CO.

– Hart, S. G. and C. D. Wickens (1990). Workload assessment and prediction. MANPRINT: An emerging technology. Advanced concepts for integrating people, machines and organizations. H. R. Booher. New York, Van Nostrand Reinhold: 257-300.

– Lee, J. D. and T. F. Sanquist (2000). "Augmenting the operator function model with cognitive operations: Assessing the cognitive demands of technological innovation in ship navigation." IEEE Transactions on Systems, Man, and Cybernetics- Part A: Systems and Humans 30(3): 273-285.

– Tijerina, L., S. Kiger, et al. (1995). Final report - Workload assessment of in-cab text message system and cellular phone use by heavy vehicle drivers on the road. (Contract No. DTNH22-91-07003). DOT HS 808 467 (7A). Washington, DC.

– Wickens, C. D. (2002). "Multiple resources and performance prediction." Theoretical Issues in Ergonomics Science 3(2): 159-177

– Wierwille, W. W. (1993). Visual and manual demands of in-car controls and displays. Automotive Ergonomics. B. Peacock and W. Karwowski. Washington, DC, Taylor and Francis: 299-320.

– Young, M. S. and N. A. Stanton (2002). "Malleable attentional resources theory: A new explanation for the effects of mental underload on performance." Human Factors 44(3): 365-37

SAVE-IT Literature Review: Task 5A

Major findings: General approach– Convergent estimate of distraction induced reaction time decrements

» Driver state» Driver performance» IVIS and roadway state and associated cognitive demand

– Models augment a data-driven prediction of distraction» Multiple resource theory» OFM-COG

– Potential algorithms to relate variables to distraction estimate» Factor analysis and linear regression» Hidden Markov model» Vector Support Machines

– Theory-based considerations of distraction» Multiple resource overload » Malleable resources—underload and workload transitions» Strategic task management, decision to engage and switching efficiency» Multiple levels of behavior (control, strategic, and tactical)

SAVE-IT Literature Review: Task 5A

● = Very appropriate⊙ = Somewhat appropriate

○ = Inappropriate

Measure Timely Diagnostic Sensitive Practical

Eye movements and scan patterns ● ● ● ⊙

Eye blinks ● ⊙ ⊙ ⊙

Pupil dilation ● ⊙ ○ ⊙

Cardiac measure—0.1 Hz variability ⊙ ● ● ⊙

Cardiac measure—Vagal tone ● ● ● ○Respiration rate ● ⊙ ○ ○Blood pressure ⊙ ● ● ○Electrodermal activity ○ ⊙ ○ ○Electroencelphalogram (EEG) ● ● ● ○Event related potential (ERP) ● ● ● ○Hormone levels ○ ⊙ ○ ○Facial electromyography (EMG) ● ⊙ ⊙ ○

SAVE-IT Literature Review: Task 5A

Measure Timely Diagnostic Sensitive Practical

Driving performance—Lane position ⊙ ⊙ ⊙ ⊙

Driving performance—Speed control ⊙ ○ ⊙ ●Driving performance—Headway maintenance ⊙ ⊙ ● ●

IVIS interactions—MRT-based estimation ● ⊙ ⊙ ⊙

IVIS interactions—ACT_R-based estimation ● ⊙ ⊙ ⊙

IVIS interactions—OFM-COG-based estimation ● ⊙ ⊙ ⊙

● = Very appropriate⊙ = Somewhat appropriate

○ = Inappropriate

SAVE-ITMultiple resources describing cognitive

demands

SAVE-IT Cognitive transactions from OFM-COG

Cognitive Task

General Category of Information Processing

Human Information Processing Resources

2.Filter. Straining out what does not matter. Acquisition Selective attention

4.Search. Looking for something. Acquisition Sustained attention,Perceptual sensitivity

19.Decide/Select. Choosing a response to fit the situation.

Interpret Long-term memory, Processing strategy

20.Plan. Matching resources in time to expectations.

Interpret Working memory, Processing strategy

21.Test. Is it what is should be? Interpret Perceptual sensitivity, Working memory,Long-term memory

22.Interpret. What does it mean? Interpretation Long-term memory, Sustained attention

23.Categorize. Defining and naming a group of things.

Interpretation Long-term memory, Perceptual sensitivity

SAVE-IT Literature Review: Task 5A

Major Findings: Algorithms to predict distraction– Hidden Markov Models (HMMs)

» Represent the stochastic sequences as Markov chains where the states are not directly observed but are associated with a probability density function

» HMMs are used to build a signal model that explains and characterizes the occurrence of observed signal outputs. These models can then be used at a later time to identify or recognize the signal output patterns of other sequences.

– Support Vector Machines (SVMs)» The objective is to find an optimal hyperplane which correctly classifies as

many data points as possible and to separate the points of two classes as far as possible. It is a constrained OPTIMIZATION problem.

» SVMs map training data into the high-dimensional feature space, and construct a separating hyperplane with maximum margin. This yields a nonlinear decision boundary in input space.

SAVE-IT Literature Review: Task 5A

Input space Feature space

Φ Optimal hyperplane

Maximum margin

The red dots indicate distraction and the blue dots indicates no distraction.

SVM is useful to classify the different patterns of eye movements which indicate potential periods of distraction

15-minute period eye movement data are used as the training data. Each training point is presented as a 2-dimension vector: (fixation frequency, standard deviation of saccade length) in 3 seconds prior.

SAVE-IT Literature Review: Task 5A

Summary/Hypotheses– Driver-specific models may be more effective than generic models

Multiple types of distraction are important

– Information overload and task conflict as described by multiple resource demand vectors and conflict matrices

– Malleable resource theory predicts low workload can diminish resources and impair performance during transitions to high workload periods

– Task preemption and attentional withdrawal

– Multi-task dynamics» Task inertia leads to switching costs» Poor strategic task management—decision to engage during high demand periods

– Failures at the control and tactical levels of driving performance

SAVE-IT Literature Review: Task 5A

Summary/Hypotheses– Distraction is best predicted by estimating task demands on the driver, driver state, and

driver performance – IVIS and roadway demands can be characterized by

» an eight-value vector that describes the type and amount of resources demanded» a conflict matrix that identifies the degree to which common use of each of the eight

resources will degrade performance» OFM-COG provides a useful structure to capture higher level aspects of task

complexity– Driver state can be characterized by:

» Eye movements» Eye blinks» 0.1 Hz variability of heart rate

– Driving performance can be characterized by time based measures of » Lane position (time to lane crossing)» Speed control» Headway maintenance (Time to contact and time headway)

– Interactions between variables may be particularly good predictors of distraction (e.g., breakdowns in scanning patterns during passing or curve approaches)

SAVE-IT Research: Task 5 B

Objectives – Assess how well IVIS interactions, physiological, and driving performance

measures, alone and in combination, can predict reaction time decrements

– Investigate whether distraction affecting tactical performance displays a different signature compared to distraction affecting control performance

– Assess how individual and generic models affect the precision of the predicted distraction

– Assess the robustness, reliability, and specificity of traditional linear regression, Hidden Markov Models, and Support Vector Machines in predicting driver distraction

SAVE-IT Research: Task 5 B

Independent variables:– OFM-COG task type and MRT task demand (24 IVIS tasks total)– Driving events (control response, tactical response)

Dependent variables– Eye blink, Eye fixation and gaze history, Heart rate– Steering, speed, and headway control– Reaction time to safety-relevant driving events

Apparatus/Subjects– 24 participants between 35-55 years of age

Design– Fully within subjects design (IVIS task X Driving Event)

Issues/Concerns (include potential impact on schedule and solutions)– Accuracy and integration issues with Seeing Machines system

SAVE-IT Research: Task 5 B

Expected Results

– Equation predicting distraction-related decrements in driver reaction time

– Assessment of the viability of three-component convergent approach to estimating driver distraction

– Assessment of prediction algorithms (Linear regression, HMM, and SVM)

– Different tasks degrade tactical and control performance

– Different predictors for degraded tactical and control performance