Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation...
Transcript of Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation...
![Page 1: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/1.jpg)
From imagination to impact
Using Information to Drive Decisions
Exploring Visual Modalities Exploring Visual Modalities Exploring Visual Modalities Exploring Visual Modalities
Dr. Fang Chen
NICTA Copyright 2010 1
Dr. Fang [email protected]
![Page 2: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/2.jpg)
High cognitive load and human responses
• Disturbance in responses caused by high cognitive load may not be perceptible to humans, but may be machine measurable
• Teaching Athletes Cognitive Skills:Detecting Cognitive Load in Computer-based Training
LongLongLongLong----term memoryterm memoryterm memoryterm memory
ShortShortShortShort----term memoryterm memoryterm memoryterm memory Shared spaceShared spaceShared spaceShared space (7±2)
E.g. Baddeley’s Modal Model of Working Memory
Visual Visual Visual Visual
processingprocessingprocessingprocessing
(Visuospatial (Visuospatial (Visuospatial (Visuospatial
sketchpad)sketchpad)sketchpad)sketchpad)
Linguistic Linguistic Linguistic Linguistic
processingprocessingprocessingprocessing
(Phonological loop)(Phonological loop)(Phonological loop)(Phonological loop)
Central
executive
PerceptionPerceptionPerceptionPerception
ResponseResponseResponseResponse
Muscular action
Excitation + vocal
tract configuration
Gesture…
Multi-sensory
perception
. . .. . .. . .. . .
. . .. . .. . .. . .
Disturbance
e.g. latency, pitch,
jittering
![Page 3: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/3.jpg)
Experiment design and data set
![Page 4: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/4.jpg)
Cognitive Load and Athletic Performance
• Cognitive load is important in complex tasks– Working memory is limited in capacity and duration
• Shared by cognitive processes e.g. perception and production of responses
– When load is too high or too low
• Problematic for skill acquisition + optimal performance
– Performance is sometimes too coarse, too subjective
– Need for cognitive load measurement/assessment– Need for cognitive load measurement/assessment
• Cognitive load in elite athletes @ AIS– Perceptual cognitive/motor skills
– Decision Making
– Strategy/ Plays
Cognitive Load
Pe
rfo
rma
nce
![Page 5: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/5.jpg)
Aim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computerAim: Elicit response behaviours, using a computer----based based based based training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive training tool (AISReact), to detect differences in cognitive load.load.load.load.
• Participants: – 12 male recreational basketball players (within-subjects)
• Longitudinal Study – Two Factors: CL and Expertise
– Pre and Post test + 6 training sessions, over 6 weeks
User Study: Basketball Cognitive Skill Learning
– Pre and Post test + 6 training sessions, over 6 weeks
• Task Design: – View a short basketball clip (10 seconds each)
– Recall the positions of players (player formations)
– Draw locations on blank court
– Think aloud responses while drawing
• Task difficulty:– Report on increasing numbers of players
– Low (3), Medium (6), High (10)
– 6 trials at each level
![Page 6: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/6.jpg)
Data Collected
• Performance Scores– Scores were given for each mark within a radius of 8% screen distance (in pixels)
from the correct player position, as recommended by basketball experts at the AIS.
• Subjective Ratings– Likert Scale (1 extremely easy - 9 extremely difficult)
• Speech Responses– Think-aloud speech as they completed the tasks– Think-aloud speech as they completed the tasks
– Continuous, multiple word phrases.
• Pen gesture – Trajectory duration, velocity and length
– Three types of markings
• Eye-Gaze– Gaze trajectory, pupil dilation, and blinks
• Galvanic Skin Response – Baseline, rest and active readings
![Page 7: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/7.jpg)
Performance Scores and Subjective Ratings
50%
60%
70%
80%
90%
Perf
orm
an
ce S
co
res
5
6
7
8
9
Su
bje
cti
ve R
ati
ng
s
Scores (%)
0%
10%
20%
30%
40%
LOW MED HIGH
CL Levels
Perf
orm
an
ce S
co
res
0
1
2
3
4
Su
bje
cti
ve R
ati
ng
s
Scores (%)
Ratings
![Page 8: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/8.jpg)
Speech Responses: Classification Results
• Pre-Test: 3 Cognitive Load Levels
– 3 classes
– Each representing a single load level (low, med or high)
– 82% of samples correctly classified Classified as
Low Medium High
Testingsamplesfrom
Low 100% 0% 0%
Medium 40% 6% 54%
• Pre-Test: 2 Cognitive Load Levels
– 2 classes
– Each representing a single load level (low and high)
– 93.2% of samples correctly classified
fromMedium 40% 6% 54%
High 15% 3% 82%
Classified as
Low High
Testing samplesfrom
Low100% 0%
High15% 85%
![Page 9: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/9.jpg)
Pen Input Features
![Page 10: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/10.jpg)
Pen Input Features
• High cognitive load can be reflected in communicative signals (production)
• Symptoms of cognitive load, – depending on the application (e.g. blackberry, tablet etc)
NICTA Copyright 2010 10
– depending on the application (e.g. blackberry, tablet etc)
– Geometric and temporal features (shape and trajectory)
– Interactive features (when it is used and for what)
– Content analysis (what is being drawn)
![Page 11: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/11.jpg)
Basketball User Study Design
• Recalling basketball player formations from 10s video clip
– Mark the position of the players on the court
• Subjective ratings (1-9 scale) and performance scores
• Longitudinal: Pre-Test, 6 Training Sessions, Post-Test
Cognitive Load Levels
NICTA Copyright 2010 11
Attacker Defender Ball Carrier
Cognitive Load Levels
Low (Easy): 3 players
Medium (Med): 6 players
High (Hard): 10 players
![Page 12: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/12.jpg)
Basketball User Study
NICTA Copyright 2010 12
![Page 13: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/13.jpg)
Feature Analysis: Duration and Velocity
• Changes from Low load to High loadHypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond Hypothesis: Duration will decrease and velocity will increase as subjects try to respond
faster, before recall is lost.faster, before recall is lost.faster, before recall is lost.faster, before recall is lost.
Velocity increased as the tasks become
more complex in 78% of gestures
tested
Significant increase from Low to High (t-
test, p<0.05) in 44% gestures tested
Gesture duration decreased as tasks
become more complex in 82% of
gestures
Significant decrease from Low to High (t-
test, p<0.05) in 50% gestures tested
NICTA Copyright 2010 13
0
100
200
300
400
500
600
cross circle ball carrier
Subject 9 Pre-test Gesture Duration
easy
medium
hard
test, p<0.05) in 44% gestures tested
0
0.05
0.1
0.15
0.2
0.25
0.3
cross circle ball carrier
Subject 12 Pre-test Velocity
test, p<0.05) in 50% gestures tested
![Page 14: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/14.jpg)
Feature Analysis: Duration and Velocity
50
100
150
200
250
300
350
400
Subject 12 Gesture Duration
Hypothesis: Duration will decrease even further, and velocity will increase further as subjects master the task.
Gesture duration decreased from pre-test to post test.
• Changes from Pre-test to Post-test
NICTA Copyright 2010 14
0
50
pre cross post cross pre ball
carrier
post ball
carrier
test to post test.
Repeated measures ANOVA shows significant effect of load (decreasing duration from low to high), and a significant effect of expertise (decreasing duration from pre to post).
No significant differences were found in the velocity feature.
Defenders (Circle) - Duration
0
100
200
300
400
500
600
700
800
LOW MED HIGH
Cognitive Load
Du
rati
on
(m
s)
PRE
POST
![Page 15: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/15.jpg)
Pen-Input Results – Trajectory Durations
• Circles and Cross shapes
200
250
300
350
400
450
500
550
600
650
Du
rati
on
(m
s)
Pre-Test Cross
Pre-Test Circle
Pre-Test Ball
• Trajectory analysis
– Significant trends of decrease in
trajectory duration as CL ↑
– Significant trends of decrease in
trajectory velocity as CL ↑,
• Except in Ball Carrier
0.2
0.25
0.3
0.35
0.4
0.45
Low Med High
Cognitive Load
Sp
eed
(p
ixels
/ms)
Pre-Test Ball
Pre-Test Circle
Pre-Test Cross
150
Low Med High
Cognitive Load
![Page 16: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/16.jpg)
Traffic Incident Study: Design
• Creating traffic detours and green light corridors
– Using pen and speech interaction on a tablet
– Scratchpad for ‘working out’
• Subjective ratings (1-9 scale) and performance scores
Cognitive Load Levels
NICTA Copyright 2010 16
Cognitive Load Levels
Low (Easy): 6 streets
Medium (Med): 10 streets
High (Hard): 16 streets
Selection Examples Shape Examples
![Page 17: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/17.jpg)
Degeneration of Interactive Shapes
• Geometric analysis of trajectoryGeometric analysis of trajectoryGeometric analysis of trajectoryGeometric analysis of trajectory– 12 features from Rubine[1991] paper on single stroke pen-gesture
recognition e.g. angle at start stroke, angle and end stroke, duration, length, sharpness etc
• MalahanobisMalahanobisMalahanobisMalahanobis distance (MDISTdistance (MDISTdistance (MDISTdistance (MDIST---- a weighted Euclidean distance))))– The number of standard deviations a pen-gesture is away from the mean of
its “standard/baseline” form, captured during training.
– As load increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant).
NICTA Copyright 2010 17
– As load increases, the curve moves away from 0, indicating a greater degree of degeneration (statistically significant).
![Page 18: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/18.jpg)
Use of the Scratchpad
• Scratchpad as a cognitive tool
– Use of note-taking as an external memory aid
• High usage expected during high cognitive load
– Organisational marks for understanding, clarification, planning.
– Diagramming as a strategy for
• Results
– Significantly increased usage ad CL increases (manual freq)
– Automated trajectory frequency count and rate per second significantly increasing
– Use of diagramming doubles between low load tasks and high load tasks (manual freq)
NICTA Copyright 2010 18
– Diagramming as a strategy for generating and discarding hypotheses
– Content Analysis:
Alphanumeric -> Symbolic, Organisational -> Diagrammatic
load tasks (manual freq)
– Increased evidence of symbolic and organisational marks, as well as spatial representations when cognitive load is high
![Page 19: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/19.jpg)
Cognitive Load Evaluation
Using Stroke-level Features Using Stroke-level Features
![Page 20: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/20.jpg)
From Writing to Cognitive Load
The writing process*:The writing process*:The writing process*:The writing process*:
What is writing? What is writing? What is writing? What is writing?
~More than pure stoke creation:~More than pure stoke creation:~More than pure stoke creation:~More than pure stoke creation:Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the Writing is a comprehensive activity integrating the close cooperation between the
human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.human brain, the eyes, the limb and both hands.
Low Low Low Low mediummediummediummedium highhighhighhigh
Cognitive Load Levels
![Page 21: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/21.jpg)
Dataset
It includes the writing samples of 20 subjects, each writing 30 sentences (10 low CL, 10 med CL and 10 High CL). The sentence is composed based on a set of given words.
![Page 22: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/22.jpg)
Dataset (cont.)
A writing sample:A writing sample:A writing sample:A writing sample:
![Page 23: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/23.jpg)
Research Method
DecisionDecisionDecisionDecision
Pattern Set & Decision Rules
Workflow of the method:Workflow of the method:Workflow of the method:Workflow of the method:
Handwriting Tasks Pre-processing Feature Extraction Pattern Matching
DecisionDecisionDecisionDecision
•Sentence compositionSentence compositionSentence compositionSentence composition •Stroke quality evaluationStroke quality evaluationStroke quality evaluationStroke quality evaluation•Strokes/pointsStrokes/pointsStrokes/pointsStrokes/points
•Feature SelectionFeature SelectionFeature SelectionFeature Selection
•GMMGMMGMMGMM
The adoption of stroke quality evaluation is dependent on the
method of feature extraction and classification.
![Page 24: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/24.jpg)
Stroke Quality Evaluation
� To selectively abandon some strokes from further
process based on the selection criteria.
Why not use all the strokes?
• Strokes with insufficient information
• Strokes with redundant information• Strokes with redundant information
• Complex strokes
• Writing noise
How the quality of the strokes are evaluated?
• Time criteria
• Length criteria
• Combined criteria during feature extraction
![Page 25: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/25.jpg)
Feature Evaluation: ANOVA Test
Pressure:Pressure:Pressure:Pressure:
Velocity:Velocity:Velocity:Velocity:
Low CLLow CLLow CLLow CL High CLHigh CLHigh CLHigh CL
ANOVA test on the extracted features:ANOVA test on the extracted features:ANOVA test on the extracted features:ANOVA test on the extracted features:
Max P Min P Avg P Max V Min V Avg V
F-ratio 7.7 4.8 6.8 1.8 10.1 4.5
p Value 0.002 0.016 0.004 0.191 0.001 0.020
Velocity:Velocity:Velocity:Velocity:
![Page 26: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/26.jpg)
Cognitive Load Classification:
Subject-Dependent GMM
Classification with leave-one-out cross-validation
9 writing tasks from each subject were used to train the GMM, and the remaining one for test; this process is repeated for every single written task. Decision of the CL level is based on the maximum probability of the test task on the three CL models.
Feature Pressure Velocity Azimuth Altitude Length Time
Accuracy 46% 39.8% 40% 39% 37.3% 37.3%
Feature Azimuth + Altitude Pressure + Velocity All
Accuracy 47.2% 52.7% 50.3%
![Page 27: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/27.jpg)
Cognitive Load Classification (cont.)
Feature Pressure Velocity P+V Azimuth Altitude Angle All
Accuracy 61.8% 57.3% 62.8% 58.5% 53.8% 65.5% 69.8%
Two CL Level Classification:
Low vs Mid
Mid vs HighMid vs High
Feature Pressure Velocity P+V Azimuth Altitude Angle All
Accuracy 57.5% 58.3% 61.3% 56.8% 57.3% 57.8% 62.3%
Feature Pressure Velocity P+V Azimuth Altitude Angle All
Accuracy 64% 58.5% 68% 55% 56.8% 58.8% 64%
Low vs High
![Page 28: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/28.jpg)
Result Analysis
• Pressure is relatively the most reliable feature for CL classification, followed by velocity.
• Combined features perform better than single feature, except for the time and stroke length feature.
• In the classification of two CL levels, the accuracy between Low CL and Mid CL is better than other pairs, and this may result from the unstable performance of the subjects in the High CL level tasks.performance of the subjects in the High CL level tasks.
![Page 29: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/29.jpg)
Future Work
• GMM analysis for inter-stroke features;
• Subject-independent models for 3 CL levels;
• Other classifiers, e.g. SVM for framed stroke sequence;
• Decision level fusion for CL classification.• Decision level fusion for CL classification.
![Page 30: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/30.jpg)
Video Based Workload
Measurement
Video Based Workload Measures
![Page 31: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/31.jpg)
Content
� Imaging sensors
�Cognitive load theory and measurable elements
� Visual perception and cognition
� Eye activities affected by cognition
NICTA confidential
� Eye activities affected by cognition
�Results
� Summary
![Page 32: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/32.jpg)
Imaging sensors
• Video based eye tracker
– Head-mount/remote
• Thermal infrared camera
– Skin temperature
• Brain imaging
– Magnetic resonance imaging– Magnetic resonance imaging
– Near-infrared neuroimaging
![Page 33: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/33.jpg)
Video based measures
• Eye activity
– Eye blink
– Eye movement
– Pupillary response
• Skin temperature
• Physical behaviour• Physical behaviour
![Page 34: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/34.jpg)
Cognitive tasks
• Single task
– Visual
– Auditory
– Arithmetic
– Memory
– Executive– Executive
– Driving
– Flight
– Traffic control
• Dual task
• Multi-tasking
![Page 35: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/35.jpg)
Cognitive load theory and measurable elements
� Cognitive load theoryCognitive load theoryCognitive load theoryCognitive load theory (Sweller, 1994)
� Schema—store, acquisition, automation
� Total cognitive load—intrinsic, extraneous
� Element interactivity in high and low load � Element interactivity in high and low load
![Page 36: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/36.jpg)
Cognitive load theory and measurable
elements
� Assessment dimensionAssessment dimensionAssessment dimensionAssessment dimension (Pass et. al, 2003)
� Mental load – knowledge, subject characteristics
� Mental effort – cognitive capacity allocation
� Performance – achievement� Performance – achievement
� Source of limitations affect visual bufferSource of limitations affect visual bufferSource of limitations affect visual bufferSource of limitations affect visual buffer (Kirby et al,1992)
� The amount of entities that can be retained
� The amount of entities that can be represented
� Objects’ scope and size
![Page 37: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/37.jpg)
Visual perception and cognition
� Visual attention modelVisual attention modelVisual attention modelVisual attention model (Duchowski, 2007)
� Consists of low and high level function
� Examined by psychological basis
� Justified by neurological substrate� Justified by neurological substrate
� Measured by psychophysics
![Page 38: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/38.jpg)
Visual perception and cognition
� Psychological basis for eye movementPsychological basis for eye movementPsychological basis for eye movementPsychological basis for eye movement (Duchowski, 2007)
� To know attention behaviour, factual information
Image stimuli (entire view)
Parallel through foveal and
peripheral vision
Sensory input
Low level
Bottom-up or
Feature -driven
Interesting features pop up to
attract attention
Current foveal location is
disengaged, eye repositioned.
Fast eye movement.
Foveal location is directed to
perceive the feature. Attention
is engaged. Identify meaning or
expectation.
Detection
Context-sensitive
Feature
attribution
Top-up or
goal-driven
High level
To “where“ reflects
the will. Saccade
Identification
Gaze
![Page 39: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/39.jpg)
Visual perception and cognition
� Implication for visual display (Duchowski, 2007)
• Foveal vision �identify “what”, sequential pattern, “scan
path”;
• Parafovea or peripheral vision �detect “where”, parallel • Parafovea or peripheral vision �detect “where”, parallel
pattern, “spot light”;
• A distinction of sensitivity between them in spatial vision,
temporal vision, luminance and contrast.
• Salience map � distinctiveness of an object
![Page 40: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/40.jpg)
Eye activities affected by cognition
� Basic eye movement Basic eye movement Basic eye movement Basic eye movement –––– saccade and fixationsaccade and fixationsaccade and fixationsaccade and fixation
� No systematic pattern found with large changes in
visual stimuli
• Comparing ROI determined by human and detected by • Comparing ROI determined by human and detected by
10 algorithms (Privitera et al ,2000)
•No best matching algorithms found with high accuracy to
predict the next points of fixation
� Cognitive factor affects eye movement programming
• targets viewed are more task-related than “intrinsic
salience” of objects
![Page 41: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/41.jpg)
Eye movement
• Workload measures
– Fixation frequency
– Fixation duration
– Saccade extent
– Saccade speed
– Scan path– Scan path
– Vergence
– Spectrum
• Questionable sensitivity
– Fixation vs. saccade
– Inconsistent results
![Page 42: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/42.jpg)
Eye activities affected by cognition
� Basic eye movement Basic eye movement Basic eye movement Basic eye movement –––– task dependenttask dependenttask dependenttask dependent
� Reading �conceptually difficult text increases fixation
duration and decrease saccade length (Rayner, 1998)
� art work � complex causes short fixation and fast � art work � complex causes short fixation and fast
saccade speed (Molnar, 1981)
� driving � shorter fixation duration and more visual
resource happened to experienced driver than novice
(Chapman, 1998)
![Page 43: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/43.jpg)
Eye activities affected by cognition
�VisuospatialVisuospatialVisuospatialVisuospatial task experimenttask experimenttask experimenttask experiment
• Fixation frequency distinguishes target density by three levels while fixation time fails (Van Orden et al. 2001)
• Fixation time significantly differs from each workload Greef et al.(2009)
� Scan path: strategies shiftScan path: strategies shiftScan path: strategies shiftScan path: strategies shift (Marshall et al., 2003)
NICTA confidential
� Scan path: strategies shiftScan path: strategies shiftScan path: strategies shiftScan path: strategies shift (Marshall et al., 2003)
• Successful strategies � less effort;
• Unsuccessful strategies � more effort, poor performance
• Combining performance score or other measures
![Page 44: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/44.jpg)
Eye blink
• Workload measures
– Blink rate
– Blink interval
– Blink duration
– Startle eye blink
• Sensitive to visual demands• Sensitive to visual demands
– Blink interval: positive correlation
– Blink rate/duration: negative correlation
• Opposite effects in dual task
– Driving/listening/visual
– Memory/flight
![Page 45: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/45.jpg)
Eye activities affected by cognition
� Eye blinksEye blinksEye blinksEye blinks
� Types �reflective, voluntary, spontaneous (dominating)
� Suppress vision � tend to be inhibited in difficult part
� Thinking affects blinks � blink less as task difficulty increased (Irwin et al. 2010)et al. 2010)
� Blink tied to motor response (Baumstimler et al. 1971)
� Indicating cognition resource released or reflecting the adequate preparation (Siegle et al., 2008)
![Page 46: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/46.jpg)
Eye activities affected by cognition
� Eye blinksEye blinksEye blinksEye blinks
� blink rate � lower as task difficulty increases in both visual and non-visual tasks (Irwin et al.2010)
� blink rate � higher in more difficult arithmetic task (Tanaka et al. 1993)1993)
� Blink rate pattern � (Bentivoglio et al. 1997)
conversation > rest > reading in 101 subjects (67.3%), rest > conversation > reading in 34 subjects (22.7%), conversation > reading > resting in 12 subjects (8.0%).
![Page 47: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/47.jpg)
Eye activities affected by cognition
� Eye blinksEye blinksEye blinksEye blinks
� blink latency � increased when short-term memory capacity reaches limit (Irwin et al.2010)
� blink duration � tend to decrease with increased visual attention demands demands
� Visuospatial task experiment � (Van Orden et al. 2001))
blink duration, blink rate have significant difference when target density increased by three levels
![Page 48: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/48.jpg)
Pupillary response
• Workload measures
– Pupil dilation
– Percentage change in pupil size
– Index of cognitive activity
– Power spectrum
• Effective physiological index• Effective physiological index
![Page 49: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/49.jpg)
Eye activities affected by cognition
� PupillometryPupillometryPupillometryPupillometry
� reliable pattern � linear relationship with cognitive processing
� task-invoke or emotion-trigger? � reflect cognitive load or arousal or both?
� task-driven event � Peavler (1974)� task-driven event � Peavler (1974)
![Page 50: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/50.jpg)
Eye activities affected by cognition
� PupillometryPupillometryPupillometryPupillometry
�by emotional arousal � Bradley et al. (2008)
�Controlling cognitive demands and arousal factors � Stanners et. al (1979)
• cognitive processing affects pupil dilation more than arousal components
![Page 51: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/51.jpg)
Eye activities affected by cognition
� Combination of eye activitiesCombination of eye activitiesCombination of eye activitiesCombination of eye activities
�Visuospatial task experiment (Van Orden et al, 2001)
![Page 52: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/52.jpg)
Skin temperature/physical behaviour
• Facial skin temperature
– Nose
– Forehead
• Physical behaviour
– Head
– Hand/mouse– Hand/mouse
– Mouth
• Low sensitivity
– Composite measure
![Page 53: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/53.jpg)
Eye Activity Results
• Video-based measure
• Low to Medium load• Low to Medium load
• As cognitive load
increases,
– Blink latency ↑
– Mean pupil size ↑
– Fixation duration ↑
– Saccade size ↓
– All significantly
![Page 54: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/54.jpg)
Eye Activity Results (2)
• As cognitive load
increases,
– Blink rate ↓
– Fixation rate ↓
– Saccade speed ↓– Saccade speed ↓
– All significantly
![Page 55: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/55.jpg)
Noisy factors
• Person
– Engagement
– Fatigue
– Stress
– Physical
• Environment• Environment
– Illumination
• Sensor
– Distortion
![Page 56: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/56.jpg)
Multiple measures vs. single measure
• Different effects on individual measures
• Single measures may be insufficient
• Multiple measures could be inconsistent
![Page 57: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/57.jpg)
Multimodal data fusion
• Fusion techniques
– Linear weighting
– Neural network
– Bayesian network
– Decision tree
– Kernel learning– Kernel learning
![Page 58: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/58.jpg)
Comprehensive modelling
• Multiple-multiple mapping
Workload Pupillary response
Environment Eye blink
Fatigue Eye movement
Stress Facial temperature
Engagement Physical behaviourEngagement Physical behaviour
![Page 59: Using Information to Drive Decisions · 2017. 4. 26. · Schema—store, acquisition, automation Total cognitive load—intrinsic, extraneous Element interactivity in high and low](https://reader036.fdocuments.net/reader036/viewer/2022071110/5fe528bb73e064402f405ad4/html5/thumbnails/59.jpg)
Summary
� Different eye activities have different ability to
classify cognitive load levels;
� Look into critical points or specific boundary
might enhance the ability to distinguish cognitive
load levelsload levels
� The relationship between those eye activities and
consistency between different task categories