Physiological sensors and EEGA short introduction to (neuro-)physiological measurements

Outline

1. Physiological sensors and EEG 2. Classification of affective states 2.1 Background & experiment design 2.2 Features, Classification and Resumee

The sensorsElectroencephalogram (EEG)Electroocculogram (EOG)

Electromyogram (EMG)Electrocardiogram (ECG)Galvanic skin response (GSR)Blood volume pulse (BVP)RespirationTemperaturecentral nervoussystemperipheral nervoussystem

Electro-----gram..occulo....myo....cardio..

Blood volume pulseinfrared photoelectric sensor detects changes in tissue blood volume 60 80 pulses a minute> heart rate and inter-beat-interval can be derivedfaster when exercised or aroused

Galvanic skin responseimpedance of the skin is measuredeccrine glands at hands and food solesincreases linear with arousal

Respirationcircumference of chest is measured via strain gaugenormal breath rate: 12 - 16 breath per minute (~500ml)faster and more shallow with higher arousal

Electroencephalogramnormal signaltheta bandalpha bandbeta bandgamma band0 1000 Hz4 8 Hz13 30 Hz8 12 Hz30 100 Hz

The 10 20 system

The lobes of the brain

Time- vs frequency domainnormal signaltheta bandalpha bandbeta bandgamma band0 1000 Hz4 8 Hz13 30 Hz8 12 Hz30 100 Hz

Event-related potentials (ERPs)The raw signal with stimuli presentations

The averaging of the raw signal epochsleading to the ERP

ERPs from affective pictures

Time-frequency analysisevoked frequency analysisinduced frequency analysisTallon-Baudry & Bertrand, 1999

RecapitulationEEG measures the electrical potentials from synchroneously active pyramidal cells in the cortexhigh temporal resolution vs low spatial resolutiononly weak signals from deeper structures & closed fields blindnessERP analysis in time domain shows mainly low frequency componentsanalysis in frequency domain (evoked vs induced) also pics up high frequency components

Classification of affective statesBackground & experiment design

MotivationAffective Computing aimes at the enrichment of human-computer interactionPhysiological and neurophysiological signals give access to (non-observable) affective and cognitive states

Emotion Models2 dimensional model of emotionvalencearousalposneglowhighboredomjoyfearrelaxationAn emotion, or an affective state, is a reaction to an internal (e.g. thought)or external event (e.g. visual or auditory stimuli), and a behavioural disposition. frustration

RegulationAffective StateAppraisalStimulus presentation+/-+/-Hippocampus (HC)dorsal Anterior Cingulate Gyrus (dACG)dorsal prefrontal cortex (PFC)Amygdala Insula ventral Anterior Cingulate Gyrus (vACG)ventral prefrontal cortex (PFC)InsideOutsideHCInsulaAmygdaladACGvACGPFCNeurobiology of Emotional PerceptionPhillips et al. 2003

Overview Issues aBCIElicitation of emotionsGround truth constructionSensor modalitiesModality fusionFeature selection and reductionClassification

Example Study

Elicitation of emotionsnot easy to establish a natural context of emotion elicitation:subject- vs event-elicited lab setting vs. real worldexpression vs feelingopen vs hidden recordingemotion vs other purpose (for subject)

Affective Stimulivalencearousalposneglowhighboredomjoyfearrelaxationfrustration

Ground truth construction normed stimulus sets (e.g. IAPS, IADS) versus self-assessment after each trial

Sensor modalitiesphysiological modalities:slow response to stimulus (seconds)long inter-trial intervals (seconds)few (but long presented) stimuli per subjectneurophysiological modality:fast response to stimulus (miliseconds)short inter-trial intervals (miliseconds)many (but short presented) stimuli per subject

Experiment: Design Resting Stimulus SAM rating3 4 s 6 s not limitedemotion induction with visual affective stimuliself-assessment of (induced) affective state after each stimulus with Self-Assessment Maneken (SAM)

Classification of affective statesFeatures, Classification and Resumee

Modality fusiondata-level fusionfeature-level fusiondecision-level fusion

Feature selection and reductionSelection:manual selection (e.g. literature research)

automatical selection / wrapper methodReduction:mapping from high into low dimensional space (e.g. PCA,ICA) > filter method

Feature extractionEEG: 6 features (from 6 frequency bands over 6 brain regions)physiological Sensors: mean, variance, and(minimum & maximum)from GSR, blood pressure,heart rate, respiration, andtemperature

Classificationmany possible classification approaches: decision trees, linear discriminant analysis, support vector machine, neural networksoptimal method depends on structure of data!keep training trials appart from test data, to avoid the contamination of the classifier.. !

Results2 classes: high vs low arousal3 classes: high, medium and low arousal

Classificationphysiological studies:4 class: up to 95%8 class: 81% (!)generalization of classifiers over subjects and time

neurophysiological studies:3 class: up to 67%2 class: up to 79% ? generalization of classifiers over subjects and time ?

Recapitulationnatural context of emotion elicitationcharacteristics of sensors are importantmany different approaches for data fusion, feature selection/reduction and classification no optimal method per sestill a long way toward an affect classification in an natural and multimodal real-world setting.. but we are getting there!

Thanks

and neutrality is just assumed from normalized values..solution: taking neutral sounds from another sound database ..