EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM

Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Sienawww.evalab.unisi.it – [email protected]

EYE TRACKING & VISION APPLICATIONS LAB (EVA LAB) – DEPARTMENT OF NEUROLOGICAL

NEUROSURGICAL AND BEHAVIORAL SCIENCE – UNIVERSITY OF SIENA

EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE AUTOMATIC CASE

BASE SYSTEMBASE SYSTEM

Giacomo Veneri, Pamela Federighi, Elena Pretegiani, Francesca Rosini, Nicola R Polizzotto, Francesco Fragnoli,Emiliano Santarnecchi, Giacomo DeMurtas, Antonio Federico & Alessandra Rufa

Eye tracking & Vision Applications Lab (EVA Lab) – Department of Neurological Neurosurgical and Behavioral Science – University of Siena

EVA Light SystemEVA Light System

EVA EVA

EYE TRACKING - EYE TRACKING - STIMULUS STIMULUS

INTEGRATED SEMI INTEGRATED SEMI AUTOMATIC CASE AUTOMATIC CASE

BASESYSTEM BASESYSTEM To test the cognitive

system by multi-stimulusDiagnosis by Case Base

04/10/23EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM

2


SummarySummary


3

Background (2m) Testing the cognitive

functionalities by gazeStimulus Architecture (5m)

Saccadic Behavior Visual Search

Gaze Contingent TMS Interference Sound integration and

joystick

Analysis (5m) Saccade: Velocity,Amplitude

Duration, Latency, Error Fixation: dispersion Task: sequencing Decision Making

Results (5m)Case Base (1m)


“BY TESTING THE RESPONSE TO VISUAL STIMULI WE CAN TEST MORE THAN 60% OF BRAIN ACTIVITY”


4

BackgroundBackground


VisionVision


5

“More 60% of our Barin works on vision …”:

Image Processing Spatial Attention Working memory Memory Visual-Spatial exploration Eye Movement Command Decision Making The focus of attention is jointly determined by stimulus

salience and top-down control signals. IOR is implemented by a mechanism that reduces the salience of stimuli at recently explored spatial locations.


Eye Movement & AttentionEye Movement & Attention


6

Eye movements Sequences of fixationfixation and

saccadesaccade The sequences of saccade-

and-fixation enable humans to receive a continuous flow of information from the world.

Attention Attention is the process of

selecting a few relevant perceptive stimuli by filtering of negligible aspect of the environment

Overlapped region

Peripheral attentionSaccadic eye movements


Working Memory & Visual SearchWorking Memory & Visual Search


7

Visual working memory : is a theoretical construct

that refers to the structures and processes used for temporarily for temporarily storing and manipulating storing and manipulating visual informationvisual information.

Visual Search memory maintains the

information in order to allow past experience to past experience to guide future behaviorsguide future behaviors


Decision makingDecision making


8

Decision making: can be regarded as an regarded as an

outcome of mental outcome of mental processes processes (cognitive process) leading to the selection of a course of action among several alternatives.

Basic connectivity framework. Dotted arrows represent external inputs to the model. Abbreviations: 5-HT, dorsal raphe serotonergic neurons; ACC, anterior cingulate cortex; AMYG, amygdala; DA, midbrain dopaminergic neurons; DLPFC, dorsolateral prefrontal cortex; OFC, orbitofrontal cortex; VS, ventral striatum.


The Main Idea: 3 stepsThe Main Idea: 3 steps


9

1. To develop a multistimlus multistimlus system able to stress system able to stress some part and functionalities of brain in order to make a low cost pre-diagnosis of patient.

1. Interference: TMS, sound2. Sensory stimulus: video, audio 3. Output: Eyetracking, joystick

2. To develop indicatorsindicators from from specificspecific task in order to test the brain response.

3. To develop a database of database of case case which can suggest dignosis avoiding any euristic or expert system logic.


Preview: clinical case & research casePreview: clinical case & research case


10

Clinical case Patients with movement disorder including cerebellar patients. Neo Cortex injury.

Neglect patients (parietal abnormalities). Frontal prefrontal lesions

Pre and post pharmacological assumption. Deficit on attention & dyslexic patients.

Research case Temporary interference by TMS Decision Making (NeuroEconomy – Dept. Economic)


SACCADES, MEMORY GUIDED Saccade, TMT, SOUND GUIDED visual exploration or saccades, GAZE CONTINGENT, FACE

ANALYSIS, GAME THEORY.


11

TaskTask


SaccadeSaccade Visual Search: time lineVisual Search: time line


12

Stimulus SystemStimulus System


Saccade / Anti saccade / Memory guidedSaccade / Anti saccade / Memory guided


13


Trail Making TaskTrail Making Task


14

Visual Search depends on sensory, perceptual and cognitive processes

TMTB is used to measure the cognitive domains of processing speed, sequencing, visual search abilities and visual-motor skills.


Gaze ContingentGaze Contingent


15Reactive gaze-

contingent display (GCD), which uses a uses a simple hole simple hole forcing forcing only foveal vision during exploration by blinding peripheral vision, in order to test and stress overt mechanisms of visual search in normal subjects and clinical contexts


Decision MakingDecision Making

Game TheoryThe subject was asked

to choose between two sign according to previous chooses Overconfident Bayesian


16


INDICATORS

AnalysisAnalysis


17


CommonCommon NewNew

04/10/23

Saccades Velocity Peak Mean Velocity

Fixations N° fixations in ROI Fixations Duration

Time Task duration Time in ROI

Saccades Going from/in ROI

Time Delta time among ROI

(step to step time)Errors

ROI re-exploration Sequencing

18

FeaturesFeatures

1 2

EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM


AlgorithmsAlgorithms

04/10/23

19

Signal processing Spurious data purging Saccade detection Fixations identification

Visual Search indicator Best Squence [BestSequenceAlgorithm] Fixations distance 8PS Model

Decision Indicator Likelihood function



Best Sequence AlgorithmBest Sequence Algorithm


20

Is a tree exploration algorithm that extract the Sequencing Index as an indicator of the ability in gazing correctly a logic sequence Subject 1:

O1A2B3C4BD5EE5O5A1AB52

Patient 1: O13C4BD5E2ABE5OAE1A12B


SCA2 (subject 1) vs CTRL (subject 2)SCA2 (subject 1) vs CTRL (subject 2)

04/10/23

21



Fixation Distance From ROIFixation Distance From ROI


22

The distance from nearest ROI It’s an indicator of fixation

dispersion during visual exploration

The distance from target It’s an indicator of goal

directed visual exploration

fs (xfs (xff,y,yff) * min) * minROIROI ( δ((x ( δ((xff,y,yff), (x), (xrr,y,yrr))))))fs (xfs (xff,y,yff) * min) * minROIROI ( δ((x ( δ((xff,y,yff), (x), (xrr,y,yrr))))))

fs (xfs (xff,y,yff) * ( δ((x) * ( δ((xff,y,yff), (x), (xr r (t+1)(t+1),y,yr r (t+1)(t+1)))))))fs (xfs (xff,y,yff) * ( δ((x) * ( δ((xff,y,yff), (x), (xr r (t+1)(t+1),y,yr r (t+1)(t+1)))))))


8Pointed S8Pointed Startar -Model -Model


23

Direction Error (ε*) = (direction observed (ο*) – expected (η*) )Direction Error (ε*) = (direction observed (ο*) – expected (η*) )


Ongoing Search abilitiesOngoing Search abilities

04/10/23Using Simplified Trial Making Test B to Evaluate Ongoing Learning Capabilities on Visual Search

24

Direction error may provide a quantitative assessment of the visual search strategy

provides an estimation of the ongoing visual search capabilities.

ΔΔεε** = = εε**t t - - ε ε** t-1t-1 ΔΔεε** = = εε**t t - - ε ε** t-1t-1


Upcoming direction versus previous fixationsUpcoming direction versus previous fixations


25

The forthcoming direction minus the minimum/maximum distribution of fixation

It’s an expression to plan the saccade far from visited location (fixated)

(ο*) – min(direction(fixations))(ο*) – min(direction(fixations))


Decision: likelihood functionDecision: likelihood function

Likelihood function was expressed as the probability that a subject spends more time looking the object which will be choose during the last 3 seconds preceding the explicit decision.


26

Time on sign / Time total elapsedTime on sign / Time total elapsed


PATIENTS & SUBJECTS


27

ResultsResults


SaccadeSaccade Cerebellar patientsCerebellar patients


0

100

200

300

400

500

600

10° 18° 20° 10° 18° 20° 10° 18° 20°

Peak Velocity (°/s) Mean Velocity (°/s) Duration (s)

CTRL

pCA

SCA2

28

Movement Disorder Saccade AnalysisMovement Disorder Saccade Analysis

Three group were analyzed Control (healthy subject) SCA2 (Autosomal Dominant

Cerebellar Atrophiy) pCA (Autosomal Recessive

Pure Cerebellar Atrophy) Significant differences were

found on Peak Velocity, Duration. Amplitude Gain Error


FixationFixation Cerebellar patientsCerebellar patients

Three group were analyzed Control (healthy subject) SCA2 (Autosomal Dominant

Cerebellar Atrophiy) pCA (Autosomal Recessive

Pure Cerebellar Atrophy)

Significative differences were found on Sequencing ability Fixation distances

Visual Search TMT AnalysisVisual Search TMT Analysis


29

Fixation dispersion


TMTTMT FixationFixation

Visual Search: gaze contingent analysisVisual Search: gaze contingent analysis


30

Two group were analyzed Control (healthy subject) FV (foveate vision inhibited by

gaze contingent)

Significant differences were found on Fixations dinstance Revisited ROI


Decision Making AnalysisDecision Making Analysis


31

20 subjects were asked to take a decision in a game theory context

Overconfident subjects confirmed the theory that “gaze is a renforcement of decision and vice-versa”

overconfidentbayesiani


direction versus previous fixationsdirection versus previous fixations


32

40 subjectsDistance from previous fixations increase on the last

800msDistance from unexplored region decreases on the last

800ms



33

Case Based SystemCase Based System


System AnalysisSystem Analysis


34

Case

TreeCase

Tree


Analysis SystemAnalysis System


35

Time Saccade Analysis Main Sequence

Visual Search Analysis


Case BaseCase Base

The case base look for the N cases nearest to the current case according to the previous indicators Euclidean distance Mahalanobis distance

Case Base suggests diagnosis The case base actually works on 82 subjects


36

DiagnosisDiagnosis

DiagnosisDiagnosis

Diagnosis

New Patient

Diagnosis

indicators

indicators

Case Base



37

STIMULUS INTEGRATED SEMI STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEMAUTOMATIC CASE BASE SYSTEM

Vision Cognitive Task

Analysis(Indicators)

Case Base

Diagnosis

Rehabilitation


Further WorkFurther Work

To extend the case base with: Degenerative brain diseases Movement disorders Attention deficit

Developing new indicators in order to find saccadic intrusion and micro-saccade

Developing new indicators (markov chain) in order to evaluate decision

Improve the search algorithm using Genetic algorithm


38


ALESSANDRA RUFA (MD PhD), ALESSANDRA RUFA (MD PhD), GIACOMO VENERI (MSC),GIACOMO VENERI (MSC),

PAMELA FEDERIGHI (MSC), ELENA PRETEGIANI(MD), FRANCESCA ROSINI (MD),

NICOLA R. POLIZZOTTO (MD),

EMILIANO SANTARNECCHI (PhD), FRANCESCO FRAGNOLI (MD)

GIACOMO DE MURTAS (MSC),

ANTONIO FEDERICO (MD)

EVA LabEVA Lab

EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM

Education

Transcript of EVA – EYE TRACKING - STIMULUS INTEGRATED SEMI AUTOMATIC CASE BASE SYSTEM