Affective Choice: A Learning Approach Toward Intelligent ... · This thesis presents the relevant...
92
Affective Choice: A Learning Approach Toward Intelligent Emotional Behaviour For Ubiquitous Computing Applications A thesis submitted to the University of Dublin, Trinity College, in fulfilment of the requirements for the degree of M.Sc. in Computer Science Michael Morrissey Ubiquitous Computing, Department of Computer Science, Trinity College, University of Dublin May 2006
Transcript of Affective Choice: A Learning Approach Toward Intelligent ... · This thesis presents the relevant...
Affective Choice: A Learning Approach Toward
Intelligent Emotional Behaviour For Ubiquitous
Computing Applications
A thesis submitted to the
University of Dublin, Trinity College,
in fulfilment of the requirements for the degree of
M.Sc. in Computer Science
Michael Morrissey
Ubiquitous Computing,
Department of Computer Science,
Trinity College, University of Dublin
May 2006
ii
DECLARATION
I, the undersigned, declare that this work has not previously been submitted as an
exercise for a degree at this or any other University, and that, unless otherwise stated,
it is entirely my own work.
________________________________
Name: Michael Morrissey
Date: 30/05/06
iii
PERMISSION TO LEND AND/OR COPY
I, the undersigned, agree that the Trinity College Library may lend and/or copy this
thesis upon request.
________________________________
Name: Michael Morrissey
Date: 30/05/06
iv
Acknowledgements
I would like to thank my project supervisor Mads Haahr for his interest and advice
during the development and demonstration of this thesis.
I would also like to thank my excellent parents, for the love and support they have
provided throughout my last 20 years of education.
I especially want to thank Máire, for all her love and support.
v
Abstract
This thesis presents the relevant research undertaken in the design, development and
implementation of an EEG based reusable affective computing interface,
concomitant with its implementation, evaluation and use.
The "Affective Choice" system is an affective computing system using a learning
based approach to derive intelligent emotional behaviour. The system will attempt to
detect and describe a user�s affective patterns (emotional cues), and learn to equate
these emotional patterns with the user�s desired actions through a reinforcement-
learning framework.
Detectable affective patterns include obvious expressions of emotion, such as a
smile, a frown, a relaxed or angry voice, but are also evident in physiological
changes in autonomic nervous system activity, such as brain wave patterns,
accelerated heart rate or increasing skin conductivity. The "Affective Choice" system
will attempt to detect and model, a user�s physiological patterns, together with
feedback of their use of an application, in order to learn to predict desirable choice,
on behalf of the users. The aim of this project is to provide a reusable affective
interface which, when utilised in suitable applications, will provide an additional
�emotional channel� for human computer interaction. This will allow an application
to learn to provide the user with a desirable choice without explicit interaction,
therefore increasing ubiquity.
vi
Table of Contents
ACKNOWLEDGEMENTS.......................................................................................................... IV
ABSTRACT ..............................................................................................................................V
TABLE OF CONTENTS.............................................................................................................. VI
LIST OF FIGURES................................................................................................................... VIII
LIST OF TABLES..........................................................................................................................X
LIST OF ABBREVIATIONS....................................................................................................... XI
CHAPTER 1: INTRODUCTION............................................................................................... 1
1.1 Background...................................................................................................................... 2
1.2 Motivation........................................................................................................................ 3
1.3 Project goals and Outcomes .............................................................................................. 4
1.4 Outline ............................................................................................................................. 4
CHAPTER 2: STATE OF THE ART......................................................................................... 6
2.1 The Social-Emotional Prosthetic for Autism Spectrum Disorders ...................................... 6
2.2 The Platform for Affective Agent Research....................................................................... 7
2.3 The HandWave Bluetooth Skin Conductance Sensor......................................................... 8
2.4 Affective Choice, in relation to state of the art................................................................... 9
CHAPTER 3: EMOTION......................................................................................................... 10
3.1 Defining Emotion ........................................................................................................... 10
3.2 Models of emotion.......................................................................................................... 12
3.2.1 Basic Emotions ...................................................................................................... 13
3.2.2 Vocal model of emotion ......................................................................................... 15
3.2.3 Emotion Models based on Facial Expressions ......................................................... 18
3.2.4 Multi-Dimentional/Continuous Model of Emotion .................................................. 20
3.3 Conclusion ..................................................................................................................... 23
CHAPTER 4: AFFECTIVE SIGNALS.................................................................................... 26
4.1 Sources of Affective Signals........................................................................................... 26
4.2 Affect from Peripheral Nervous System (physiological sensors) ...................................... 27
4.2.1 Electromyogram (EMG)......................................................................................... 27
4.2.2 Galvanic Skin Response (GSR) or Skin Conductivity Response (SCR).................... 28
4.2.3 Heart Rate and Blood Volume Pulse (BVP) ............................................................ 29
4.3 Affect from Central nervous system (The Brain) ............................................................. 30
vii
4.3.1 Electroencephalography EEG ................................................................................. 30
4.3.2 MEG, fMRI, PET � Other Advanced Brain Imaging Techniques ............................. 31
4.4 Conclusion ..................................................................................................................... 32
CHAPTER 5: THE BRAIN, EEG & EMOTION .................................................................... 34
5.1 How the Brain Processes Emotions................................................................................. 35
5.2 The Lateralisation of Emotion......................................................................................... 39
5.2.1 Right Hemisphere hypothesis vs The Valence Hypothesis ....................................... 39
5.3 The recording of EEG..................................................................................................... 41
5.3.1 Types of electrode .................................................................................................. 41
5.3.2 Electrode placement ............................................................................................... 42
5.4 Conclusion ..................................................................................................................... 43
CHAPTER 6: THE AFFECTIVE INTERFACE SYSTEM..................................................... 44
6.1 Overview of the Affective Interface System .................................................................... 44
6.2 System Components ....................................................................................................... 45
6.2.1 Brainmaster Module ............................................................................................... 45
6.2.2 Support Vector Machine for Classification and Regression...................................... 47
6.3 System Architecture ....................................................................................................... 49
6.4 Overview of the Brainmaster Interface ............................................................................ 50
6.5 Overview of SVM Machine Learning Interface ............................................................... 52
6.5.1 SVMInterface Component Classes.......................................................................... 53
6.5.2 SVMScaler............................................................................................................. 53
6.5.3 SVMTrainer ........................................................................................................... 55
6.5.4 SVMPredictor ........................................................................................................ 56
6.6 Overview of AffectiveInterface....................................................................................... 57
6.7 Using the Affective Interface System .............................................................................. 60
CHAPTER 7: EVALUATION & CONCLUSION................................................................... 63
7.1 Evaluation and Effectivness of Demo Application (Affective MP3 player) ...................... 63
7.1.1 Evaluation .............................................................................................................. 64
7.1.2 Demonstration conclusion ...................................................................................... 65
7.2 User tests........................................................................................................................ 65
7.3 Conclusion ..................................................................................................................... 66
7.4 Future Work................................................................................................................... 66
REFERENCES ............................................................................................................................ 68
APPENDIX: TABLES.................................................................................................................. 72
APPENDIX: FIGURES................................................................................................................ 75
APPENDIX: SOURCE CODE..................................................................................................... 76
viii
List of Figures
Figure 3-1: 2D Valence / Arousal Space and the distance between emotions (Russel and Bullock 1985 [48]) ......................................................................................... 21
Figure 3-2: Simplified version of emotion hierarchy (P.Shaver, et al. 1987 [43]) ................................................................................................ 23
Figure 4-1: Signals from EMG on the masseter muscle in micro-volts, indicating periods during which a subject was asked to express, no emotion, anger, hate, grief, love, romantic love, joy and reverence (J. Healey and R. W. Picard 1997 [25]). ............................................................. 28
Figure 4-2: Signals from SCR in micro-Siemens, indicating periods during which a subject was asked to express, no emotion, anger, hate, grief, love, romantic love, joy and reverence (J. Healey and R. W. Picard 1997 [25]). ............................................................................................. 29
Figure 4-3: Signals from Heart Rate in bpm, indicating periods during which a subject was asked to express, no emotion, anger, hate, grief, love, romantic love, joy and reverence (J. Healey and R. W. Picard 1997 [25])..................................................................................................... 29
Figure 4-4: Brain wave samples indicating the four most dominant frequencies. ............................................................................................... 31
Figure 5-1: (left) Cerebral Cortex, showing the major lobes (frontal, parietal, temporal and occipital) and the brain stem structures (pons, medulla oblongata, and cerebellum). (Right) Limbic System inside the brain. Consists of the fornix, hippocampus, cingulate gyrus, amygdala, parahippocampal gyrus and thalamus................................................................................. 36
Figure 5-2: MacLean's Triune Brain................................................................................................ 37 Figure 5-3: 10-20 Electrode Placement System................................................................................ 42 Figure 6-1: Use Case Diagram of Affective Choice System ............................................................. 44 Figure 6-2: Picture of the Brainmaster EEG Module used in the development
of the Affective Choice system, (Thomas F. Collura [12]).................................................. 46 Figure 6-3: Graphical Depiction of how SVM learns to classify or group reading into categories.
(LEFT) This depicts a 2D set of training data, containing 3 distinct categories. (RIGHT) Depicts the SVM attempt to build a classification model for this distribution. This Model can then be used to determine the category of any future readings. ........................................... 47
Figure 6-4: LIBSVM vector data format.......................................................................................... 48 Figure 6-5: Shows Class Diagram of Affective Interface System, including ( system components ),
Brainmaster EEG Module, and LIBSVM base class. .......................................................... 49 Figure 6-6: BrainmasterInterface Class Diagram ............................................................................. 51 Figure 6-7: SVMInterface Class Diagram........................................................................................ 52 Figure 6-8: SVMScaler Class Diagram............................................................................................ 54 Figure 6-9: SVMTrainer Class Diagram.......................................................................................... 55 Figure 6-10: SVMPredictor Class Diagram ..................................................................................... 57 Figure 6-11: AffectiveInterface Class Diagram................................................................................ 58 Figure 6-12 : Effective EEG Sensor Placements .............................................................................. 61 Figure 6-13: 2D training file data showing labels and sample values, this data is then shown scaled
from �1 to 1. ..................................................................................................................... 62
ix
Figure 0-1: Ekman & Friesen's Action Units (Ekman and Friesen 1978 - http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm [23]) ......................................... 75
x
List of Tables
Table 3-1: Components of Emotion (P. Kleinginna and A. Kleinginna 1981 [35])............................ 11 Table 3-2: T.D. Kemper � 5 Criteria for Basic Emotions(T. D. Kemper 1978 [34]) ..................... 14 Table 0-1: Basic emotions according to different authors................................................................. 72 Table 0-2: Average differences evident in speech components my emotion
(Walter Sendlmeier Felix Burkhardt [22]) (See section 3.2.2)............................................. 73 Table 0-3: Demonstration Questionnaire ......................................................................................... 74
xi
List of Abbreviations
EEG Electroencephalogram
MEG Magnetoencephalogram
EMG Electromyogram
GSR Galvonic Skin Response
fMRI Functional Magnetic Resonance Imaging
PET Positron Emission Tomography
BVP Blood Volume Pulse
ECG Electrocardiogram
ERP Event Related Potentials
RHH Right Hemisphere Hypotheisis
VH Valence Hypothesis
Prosody Term used to refer to speech elements such as intonation, pitch etc.
1
Chapter 1: Introduction
An Affect is the external expression of emotion attached to ideas or mental
representations.
The "Affective Choice" system is an Affective Computing system using a Learning
based approach to derive intelligent emotional behaviour. The system will attempt to
detect and describe a user�s affective patterns (emotional cues), and learn to equate
these emotional patterns with the user�s desired actions through a reinforcement-
learning framework.
Detectable affective patterns include obvious expressions of emotion, which we, as
humans, are very effective at recognising, such as a smile, a frown, a relaxed or
angry voice. The expression of these emotional affects plays a large part in how
humans interact and communicate. For instance, it is impossible to be certain that a
statement is meant to be sarcastic without the accompaniment of a sarcastic tone of
voice or facial expression. Statements such as �It�s not what he said but the way he
said it� also attest to the importance of emotion expression in human communication.
Affective emotional patterns are also evident in physiological changes in autonomic
nervous system activity, such as accelerated heart rate or increased skin conductivity
and in the central nervous system, the brain. The "Affective Choice" system will
attempt to provide application developers with a prototype affective user interface.
This interface can be used within any application to detect and model its user�s
physiological patterns, in order to provide awareness of a user�s state of mind, or to
allow affective signals to be used to directly control some functionality of the
system.
2
1.1 Background
The vision and idea behind affective computing was first introduced by Rosalind
Picard in a technical report from MIT�s Perceptual Computing Department in 1995.
The interest that this idea sparked lead to the formation of the Affective Computing
Group, at MIT Media Lab, lead of course by Rosalind Picard , who further outlined
her ideas and thinking behind affective computing in her acclaimed book �Affective
Computing� (Rosalind W. Picard 1997 [45]). Since then the Affective Computing
Group have continued to pioneer the research in this area, producing a steady stream
of innovative and novel projects over the past ten years.
The area of affective computing relates to human computer interactions where a
device or system has the ability to react appropriately to its user�s emotional state.
Such a device must have the ability to sense at least one of the many affective signals
exhibited by an emotional human user and, most importantly, correctly categorise or
identify the particular mood or emotion that that affective signal might represent.
Sources of these affective or emotional signals include, facial expression, tone of
voice or speech patterns, gestures, body language, etc. Other possible sources of such
information can be derived from a subjects use of established computer interfaces,
such as mouse movements, number of mouse clicks, patterns and indeed the force
with which keys are pressed. This type of information would for instance exhibit
whether a user was frustrated with, or very tentative and unsure of their control of a
computing system. These sources of affective information, in fact directly represent
our unrelenting failed human attempts to communicate our frustration to our standard
computer systems. It seems that no matter how aware we are that these expressions
of affect are falling on deaf ears, this does not diminish our attempts to communicate
to computers in this way. This provides an excellent illustration of how essential and
fundamental this method of communication and interaction is to us. It also provides a
very strong argument for the development of affective user interfaces, so much so
that it seems strange that these obvious manifestations of attempted interaction have
to date been almost completely ignored or overlooked.
However, most of these signals are in some ways unsuitable as sources of affect in a
ubiquitous, mobile or wearable system, either by not providing enough information
to derive effect, by being difficult to measure and process or simply by not reflecting
3
the emotions of a user. For this reason it may be an advantage to use some less
obvious affective signals that are not so generally used for communication, such as
those emanating from our central and peripheral nervous systems. The most
accessible of these effective signals include ECG, EEG, EMG, GSR, BVP and
others. All these signals are examined and explained further in Chapter 4: .
In order to recognise patterns within these affective signals and to correctly
categorise such patterns as emotions, moods etc. it would be helpful to have some
knowledge and understanding of what exactly emotions are, what their purpose is
and where they come from. Unfortunately, as discussed in Chapter 2: there is no
clear definition of emotion, nor can emotions be objectively observed. Instead we are
left deciding between models of emotion derived from behavioural, psychological, or
physiological characteristics, so called �components� of emotion, coupled with their
subjective descriptions. Despite the difficulties of finding textbook definitions for
what exactly emotions are and where they come from, there is no argument about
their existence and importance. It is better that we seek to enhance our understanding
of emotions, rather than struggle to define them. Using this approach allows us to
develop affective computing systems, able to provide applications with an insight
into their user�s emotional state, without having to get to bogged down in the detail
and philosophical debate that surrounds the study of human emotions.
1.2 Motivation
Coming from a background in computer applications and software engineering into
the exciting area of ubiquitous computing with its fresh new ideas, interfaces and
visions for the future of human computer interaction, it is immediately clear how
many of these visions are still without answers. The implementation of the majority
of such ubiquitous systems very much involves starting from the ground up, as, to
date, there is a severe lack of suitable middleware and API�s in the area. The focus of
my research is in developing a ubiquitous user interface, which addresses some of
the visions of ambient control set out in the ubiquitous computing paradigm,
whereby, intelligent devices in our environments can act in a desirable manner on
our behalf without requiring explicit user interaction or physical control. Clearly, this
ideal presents many challenges which require attention.
4
I believe that the area of affective computing (through the use of emotionally
intelligent user interfaces) is set to offer solutions to a possibly limitless number of
computing applications, and in particular has potential as an alternative user interface
in the domain of ubiquitous computing. There are huge problems inherent in
ensuring that ubiquitous computing systems (especially agent systems acting on our
behalf behind the scenes) act only in desirable and efficient ways. I believe that,
through the use of intelligent affective interfaces these problems can be much more
easily and effectively addressed while preserving the defining ambient nature of such
systems.
1.3 Project goals and Outcomes
The aim of this project was to research, design and build a simple, reusable and easy
to implement, affective interface system. This thesis presents the outcomes of this
project whose main goal was to develop a proven affective interface which can be
easily implemented in order to provide an additional �emotional channel� for human
computer interaction. This affective interface will be shown to enable the
development of applications that can make desirable choices on behalf of its user
(based on the user�s affective state, or mood) without the need for physical actions or
explicit interaction. It is the intention that this affective interface can be easily reused
as a component of future ubiquitous applications.
1.4 Outline
When attempting to define a new human computer interface it is crucially important
to understand the background and research related to both the human and technology
aspects of the desired interaction. This thesis, therefore, explores the relevant
research relating to human emotion and affective signal processing that is required to
competently undertake decisions regarding the design and development of this
affective user interface, concomitantly with details of its implementation, evaluation
and use. The following paragraphs provide a short overview of each chapter.
5
In Chapter 2: I briefly describe some of the current projects being undertaken in the
field of affective computing, in order to provide comparison to the work presented in
this thesis.
In Chapter 3: I set out to define and understand what an emotion is, and the
limitations and problems inherent in attempting to measure, understand or even
categorise an emotion. I examine various models of emotion derived using different
mechanisms. This chapter also examines the main research in the area of human
emotion, outlining the conflicting models, opinions and viewpoints expressed in this
difficult area.
In Chapter 4: I describe and contrast the different sources of affective signal,
concentrating on sources evident in the central and peripheral nervous systems. I
conclude this study with the reasons for my choice of EEG alone to be used for the
measurement and classification of a user�s affective state.
In Chapter 5: I further examine the suitability of EEG in the study of emotions, and
examine the relevant research related to brain function, emotion and the processes
and components of EEG recording. I conclude with justification for the use of EEG
as the basic component of the affective interface system.
In Chapter 6: I outline the final design, implementation and use of the affective
interface system. Class diagrams and functional description of the components of the
interface are also presented in this chapter.
Finally in Chapter 7: I describe an evaluation of the Affective Choice System, and
conclude by outlining the successful outcomes of the project
6
Chapter 2: State of the Art
Research in the area of affective computing was pioneered in the late 1990�s by
MIT�s Affective Computing Group, headed up by (Rosalind W. Picard 1997 [45]).
This group still lies at the forefront of research and work in this area. In order to
demonstrate the highest degree of development in this field I will outline three of the
most current research projects being undertaken at MIT�s media lab. The three
projects outlined below include, The Social-Emotional Prosthetic, The Platform for
Affective Agent Research and The HandWave Bluetooth Skin Conductance Sensor.
2.1 The Social-Emotional Prosthetic for Autism Spectrum Disorders
(R. el Kaliouby, et al. 2006 [32])
The Social-Emotional Prosthetic is a novel wearable device that perceives
and reports on so called social-emotional information in human-human
interaction. The system is designed to annotate real-time human interaction
in order to assist communication in the natural environment. The intended
application of this system will be to help individuals, diagnosed with
Autism Spectrum Disorder (whose traits include social impairment), to be
able to increase their awareness and perception of communication in a
natural environment in order to maximise their ability to learn and develop
in social settings.
According to the research this project brings together and expands upon
recent advances and methodologies in affective computing, wearable
computing and real time machine perception. The system gathers this
social-emotional information using a small wearable camera and other
sensors, such as skin conductance sensors. These sensors are combined
with machine vision and emotion perception algorithms that provide real-
time emotion annotation of a human-to-human interaction. The system
analyses the facial expressions and head movements of the person with
7
whom the wearer is interacting, using mechanisms based on Kaliouby�s
model for the inference of affective-cognitive signals from head and facial
movements (R. el Kaliouby July 2005 [31]). According to the authors, the
wearable platform records natural face-to-face interactions, applying
machine perception algorithms, which help identify the features of a social
interaction and predict how that interaction should be perceived.
2.2 The Platform for Affective Agent Research
(W. Burleson, et al. 2004 [7])
This platform was built for the sensing and interpretation of several aspects
of a user�s nonverbal affective information. It is also designed to enable
appropriate responses to be simulated through an expressive agent (a
graphical 3d expressive humanoid agent). The platform consists of several
multi-modal affective sensors, a real time inference engine, a behavior
engine, and an expressive agent which operated inside a graphical virtual
environment.
The platforms architecture treats its user�s affective signals as inputs which
are mapped using probabilistic and rule-based modeling to one of several
outputs. The inference mechanism incorporates machine learning to tailor
itself to a particular user�s affective signals, this is however an offline
process that requires designer intervention. The aim of the platform is to
enable multiple channels of affect to be monitored and analysed in order to
better understand how affect is communicated. Their multi-modal sensor
system consists of a pressure mouse, a wireless skin conductivity sensor, a
posture analysis chair (uses a TekScan pressure sensor), and a facial action
unit analysis module hooked up to IBM�s blue eyes infra-red sensitive
camera system, to provide face and head tracking. This system expands
upon the earlier work (A. Kapoor, et al. 2001 [33]) which used just facial
and postural information.
This platform is seen as a general-purpose tool that can be used for
research in several areas, including how to design an affective learning
companion, and how to increase our basic understanding of empathy and
emotion in order to promote human-agent interaction leading to affective
8
machine expression. See also work on The Affective Learning Companion
(Winslow Burleson and Rosalind Picard 2006 [8]).
2.3 The HandWave Bluetooth Skin Conductance Sensor
(M. Strauss, et al. Oct 2005 [51])
The HandWave is a small, wireless networked, skin conductance or
galvanic skin response (GSR) sensor, especially designed for use by the
Affective Computing Group at MIT in their affective applications. It is
used to detect information related to emotional, cognitive, and physical
arousal of mobile users. According to the group, GSR is the most reliable
measure of arousal (see valance/arousal model in section 3.2.4) in human
emotion. As noted by the author, the majority of existing affective
computing systems have made use of inflexible sensors that are physically
attached to supporting computers, in contrast to this HandWave allows
truly mobile interaction. Its use of Bluetooth affords the device an
additional degree of flexibility by providing ad-hoc wireless networking
capabilities to a wide variety of Bluetooth devices, including PDA�s and
mobile phones. I have highlighted this device to show how affective
computing systems can be easily integrated into, and used to provide
affective user info to applications of ubiquitous computing, and as a
direction for future work involving the system presented by this thesis.
The HandWave device is currently being used in a variety of different
applications at MIT, including:
Learning Companion. A relational agent that supports different
meta-cognitive strategies to help students overcome frustration. Using
affective mirroring: the agent subtly mimics the user�s various aspects
of the user�s affective expressions (Winslow Burleson and Rosalind
Picard 2006 [8]).
Collective Calm. A multiplayer, biofeedback, tug of war video game
that teaches groups of players how to relax within a competitive
9
environment and learning to cooperate as part of a team. The team that
collectively relaxes the most wins the game.
2.4 Affective Choice, in relation to state of the art
The affective choice system is inspired by the work outlined above, but has been
designed uniquely as an interface to be used to provide ubiquitous applications with
emotional information describing the affective state of their users. The system will
allow ubiquitous applications to act and react appropriately according to their user�s
affective state, without explicit physical interaction. It is also the intention that this
interface can be used to provide more than one type of affective signal, and that it
can be configured for a wide range of applications. In order to facilitate this the
Affective Choice system uses EEG to measure its user�s affective state. This is an
approach which is not taken in any of the affective computing applications described
above, even though the area of cognitive science attests to the importance of the
brain in controlling emotions. The use of EEG to derive affective state is, in my
opinion, a much more flexible arrangement, as it allows the same system to be used
to sample the array of heterogeneous emotional and physiological artefacts evident in
the brains ERP�s. Using EEG signals also allows the system to be configured to
recognise conscious and unconscious brain wave patterns, enabling the interface to
act either as an affective choice system (where events are triggered on behalf of a
user, based on their affective signals), or as a brain controlled interface (where a user
can consciously trigger events just by �thinking about it�).
10
Chapter 3: Emotion
�Everyone knows what an emotion is, until asked to give a definition�
(Beverly Fehr and James Russell 1984 [21])
We are emotional beings, so much so that our emotional state plays an important role
in how we experience and interact with our environment. Emotion directly affects
our decision-making, perception, cognition, creativity, attention, reasoning, memory,
and how we learn. It has been shown that the ability to express our emotion is
integral to our ability to communicate with each other, through facial expression,
tone of voice or body language. (F. D. Ross [47])) found that stroke patients who
cannot properly identify or generate the affective emotion that accompanies speech,
experience severe interpersonal difficulties and problems expressing themselves. For
most, the detection of emotions is intuitive during interaction. Strange, then, that
there is as yet no accepted scientific method of detecting when or what type of
emotion a subject is experiencing. There is even disagreement concerning the
classification of emotional states into types. As (Beverly Fehr and James Russell
1984 [21]) succinctly put it �Everyone knows what an emotion is, until asked to give
a definition�
3.1 Defining Emotion
Emotions cannot be objectively observed, directly measured or wilfully elicited.
Only subjective definitions of emotions are possible. We can describe what it feels
like to have an emotion, and we do, using words like happy, sad, anger, jealousy,
love and fear. So when we attempt to describe or define emotion we find that we
cannot without returning to these subjective definitions.
What we can measure and describe are some of the behavioural & physiological
components or affects of emotion, such as facial expressions or heart rate and brain
11
responses. A broad definition of emotion was emphasised by (P. Kleinginna and A.
Kleinginna 1981 [35]) once they had completed reviewing and classifying over 100
varying and inconsistent definitions.
"Emotion is a complex set of interactions among subjective and objective
factors, mediated by neural/hormonal systems, which can: (a) give rise to
affective experiences such as feelings of arousal, pleasure/displeasure; (b)
generate cognitive processes such as emotionally relevant perceptual effects,
appraisals, labelling processes; (c) activate widespread physiological
adjustments to the arousing conditions; and (d) lead to behaviour that is often,
but not always, expressive, goal directed, and adaptive"
(P. Kleinginna and A. Kleinginna 1981 [35])
This definition cautiously outlines four important components of emotion outlined in
able 3-1. This consensual definition is in fact so consensual that some authors (Frank
Enos 2003 [20]) have dismissed it as lacking a point of view and therefore think it
irrelevant. Others believe that a definition of emotion is impossible as emotions are
hypothetical constructs, being neither observable nor measurable. Despite this
Kleinginna & Kleinginna�s model is still widely accepted as a definition of the main
components of emotion.
A. Kleinginna & P. Kleinginna Components of Emotion
Affective Components Subjective Experiences such as Arousal, Pleasure,
Displeasure.
Cognitive Components Emotionally Relevant Perceptual Effects such as
Appraisals, Labelling Processes etc.
Physiological Components Physiological Functions, changes in peripheral and
central nervous system
Behavioural Components Behavioural functions such as facial expression,
verbal and nonverbal communications and actions
Table 3-1: Components of Emotion (P. Kleinginna and A. Kleinginna 1981 [35])
12
However difficult it may be to find a suitable definition for what exactly an emotion
is, nobody can deny the existence and importance of emotions. It is then, perhaps,
more important that we seek to understand our emotions, how they work, how they
affect us and their purpose, rather then simply trying to wrap them up in a suitable
textbook definition.
3.2 Models of emotion
How do we know when a subject is experiencing a particular emotion? What are the
basic emotions that the subject may be feeling? How do we model these basic
emotions? These are all difficult questions without clear or definite answers. What
we do know is that it is more likely that an emotion is present when there are
simultaneous, complex changes in our subjective feelings, physiological states and
behaviour. Lucky for us that these are things which can be measured, regardless of
whether you consider them components of emotion or simply side effects of
cognitive processing. Therefore we can assume that an emotion is being felt
whenever recognisable changes occur simultaneously in a subject�s affective state.
How do we then categorise what emotion or type of emotion that our subject may be
experiencing? To do this we first require a set of basic emotions to be defined.
Armed with these basic emotions we can produce a model of our subject�s emotional
state.
A useful model would draw distinctive and reliable relationships between discrete
emotions and their affective components. So that, for instance, when your heart rate
rapidly jumps and you begin to shake, you are experiencing fear. Coming up with a
good model of emotion is difficult as there are many affective channels through
which it might be possible to model emotion. The main channels used to model
emotion include: facial expression, vocal expression or voice modelling, signals
present in the peripheral nervous system such as heart rate, temperature, GSR etc.
and signals present in the central nervous system usually measured by EEG or MEG.
13
3.2.1 Basic Emotions
The definition of a set of basic emotions, also known as primary or fundamental
emotions, is another area where there is no agreement. (T Turner A Ortony 1990 [1])
in their paper entitled �What�s basic about basic emotions?� questions the concept of
basic emotions altogether, stating that there are no satisfactory criterion regarding
what makes an emotion basic, and that the concept of basic emotion can not account
for the large diversity of emotions.
Support for the concept of basic emotions comes from two differing schools of
thought and relates to the origin and purpose of emotion. However, their conflicting
ideas both lend credibility to the idea that there exists some set of basic emotions.
These are the Social Constructionalists or Cognitive Theorists and the Darwinian
Evolutionary Theorists. The social constructionist theory states that adult human
emotions depend upon social concepts and are learned throughout ones lifetime.
Here basic emotions are attributed to the constant social learning which they believe
will occur for all members of a particular species, regardless of culture. The
Darwinian theorists suggest that basic emotions are selected by nature through a
process of natural selection through their survival advantages. Here basic emotions
are those, which are passed down by our ancestors, i.e. fear protects us from danger
etc. Darwinian theorists also leave room for social learning to occur throughout the
lifetime of a subject, but they believe it is control of the basic emotions that is being
learned. (C Izard and S Buechler 1980 [28]) state that adult human emotions are the
same as those found in animals and human infants with only rudimentary powers of
consciousness, proving that cognition is not required for these emotions, that they are
inherent in our makeup and that they don�t need to be learnt.
In order to describe the criteria for the definition of a set of basic emotions (T. D.
Kemper 1978 [34]) outlined five criteria which he felt must be met in order for an
emotion to be considered a basic emotion.(see Table 3-2 below), He also suggested
four basic emotions: fear, anger, depression and satisfaction.
14
T.D. Kemper � 5 Criteria for Basic Emotions
Evolutionary Basis Must be evident or at least inferable in most animals
Cross Cultural Must be universally evident in all cultures, independent of environment.
Ontogenetic Must be evident in early development
Social Relational Basis Must be a result of social interactions
Physiological Must be associated with specific autonomic physiological changes
Table 3-2: T.D. Kemper � 5 Criteria for Basic Emotions(T. D. Kemper 1978 [34])
As outlined in (Michaela Esslen Pascual Marqui 2002 [40]) Kemper also classified
the different methodologies employed in the definition of basic emotions. He
highlights seven of the more important ones as: evolutionary, neural, psychoanalytic,
autonomic, facial expression, empirical classification and developmental.
Unfortunately for Kemper his criteria for basic emotions were not widely accepted
and others kept defining their own set of basic emotions. As shown in (Michaela
Esslen Pascual Marqui 2002 [40]) (see Appendix Table 0-1) between all of these
diverse sets of basic components, we are able to extract a set of the most popular
basic emotions, over which there seems to be agreement. These are:
Anger /Rage Fear/Anxiety Happiness/joy Sadness/Sorrow/Grief
Disgust Surprise/Astonishment
(Paul Ekman 1999 [19]) who is a strong Darwinian theorist describes six basic
emotions reflected in human facial expressions across several cultures. All six are in
line with those above. Ekman et al, also support their findings with signals coming
from the autonomic nervous system. These signals show that the basic emotions of
fear, anger, sadness and disgust exhibit different and specific patterns in a subjects
affective cues of heart rate and GSR (Skin Conductivity). This is very important for
me as it lends support to the idea that by observing the affective patterns in GSR, heart
15
rate and other autonomic nervous system signals, a computer system is in effect
gaining an insight into the emotional state of the user.
Although there is no consensus on the issue, basic emotions are said by some
researchers to act as building blocks for more complex emotions. (C Izard 1991 [27,
R Plutchik 1994 [46]) describe complex emotions (non-basic emotions) as being
made up of mixtures or differing combinations of basic emotions, this view is also
expressed by early work by (W.V. Friesen P Ekman 1975 [42]) where they suggested
that �for example, smugness might be considered to be a blend of the two elemental
emotions, happiness and contempt.� These emotions are generally referred to as
compound emotions.
Building on this come the secondary emotions described by (A Damasio 1994 [14, J.
LeDoux 1996 [36]). Secondary emotions or emotional memories are evident where
orderly associations have been made between objects, situations and basic emotions.
These are learned emotional responses, such that in some situations the retrieval of
emotional memories of similar experiences are used to effect our actions in that
situation. E.g. the memory of your emotional state after being involved in a car
accident would cause a secondary emotion to be felt every time you sat into a car.
Secondary emotions seem to be the basis of most irrational fears and phobias, where
strong emotional responses are associated inconsistently with certain stimuli.
3.2.2 Vocal model of emotion
Models of emotion through vocal expression have been achieved using methods such
as Prosodic modelling and Formant Synthesis. Prosody is the term used to describe
elements of speech such as intonation, pitch, rate, loudness, rhythm, etc. Formants on
the other hand refer to variable components evident in the frequency spectrum
produced by an instrument or human voice. Formants are what give vowels their
characteristic sound.
(Walter Sendlmeier Felix Burkhardt [22]) created a model for synthesising emotional
speech. Burkhardt defines a number of modifiable parameters of speech, which when
manipulated in the appropriate ways, can transform neutral speech (devoid of affect)
into emotional speech.
16
The modifiable parameters are outlined as follows:
Pitch Height, Range: Variation of pitch values.
Pitch Contour: Changes to the pitch contours of syllables and whole phrases.
Flutter: Flutter or jitter applied to all vowels.
Intensity: Varying intensity applied to syllables.
Speech Rate: Varying speech rate.
Phonation Type Modal, falsetto, breathy, creaky, or tense voice .
Vowel Precision: Formant target overshoot or undershoot. Lip-Spreading: Raising of frequencies of formants by a given rate.
Using this set of modifiable voice parameters Burkhardt set about conducting two
experiments. The first experiment involved the systematic variation of selected
acoustical features of human speech. The aim of this first experiment was to gauge
the importance of certain acoustical features for vocal emotional expression. The
second experiment was designed to investigate a greater feature set than was
involved in the first, and also to distinguish between emotions that differ only by the
extent of arousal present. The approach to the second experiment differs from the
first, here a prototype spoken with an emotionally neutral condition is generated for
each emotion and is then manipulated to further investigate the effect of modifying
certain parameter features.
The results of the (Walter Sendlmeier Felix Burkhardt [22]) experiments outline a
system that is capable of generating recognisable emotional expression through the
measured variation of its modifiable parameters. From our point of view what
Burkhardt has produced is a model of the emotions of fear, joy, boredom, sadness
and anger which can be measured by identifying the existence of certain variations in
the signals formants.
17
The results model of Burkhardt�s first test taken from (Walter Sendlmeier Felix
Burkhardt [22]) is outlined below, also see Table 0-2:
Fear: Utterances were judged as fearful when they had a high pitch, a
broad range, falsetto voice and a fast speech rate or a speech rate according
to the mixed model.
Joy: Utterances with a broader pitch range and a faster speech rate as well
as modal or tense phonation are more often judged as joyful than the other
characteristics. A lowered pitch sounds less joyful. Striking is the
noticeable effect of vowel precision: A precise articulation enhances a
joyful impression and an imprecise one reduces it. Joy produces the least
dependable recognition rates.
Boredom: A lowered mean pitch and a narrow pitch range as well as a
breathy or creaky voice results significantly often in an assessment of the
stimuli as bored. Furthermore, a slow speech rate and an imprecise vowel
articulation enhance a bored expression. For all these features the reverse
modifications weaken the assessment of the stimuli as bored.
Sadness: The expression of sadness is revealed by a narrow range and a
slow speech rate as well as a breathy articulation. Surprisingly, a raised
pitch contour and falsetto voice also enhance a sad impression.
Anger: For anger there are few, but obvious results. A faster speech rate
and tense phonation is judged by the majority as angry.
Another vocal model of emotion is outlined by (Jeremy Ang, et al. [2]), who
investigate the use of prosody for the detection of frustration and annoyance in
human computer dialog. The prosodic features that are used in the model include:
speech duration and rate, pause, pitch, energy and spectral tilt. Speech repetition and
correction along with the position in the dialog are also considered as emotional
features. Their approach uses a brute force algorithm for classifying utterances into
the categories of: neutral, annoyed, frustrated, tired, amused, other and non-
applicable (contained no speech data). Their classification algorithm is based on
decision trees.
18
The results of the study entitled, �Prosody-Based Automatic Detection of Annoyance
and Frustration In Human-Computer Dialog� (Jeremy Ang, et al. [2]) show that their
prosodic model can predict whether an utterance is neutral versus �annoyed or
frustrated� with the accuracy, they say, on a par with that of inter-human agreement.
3.2.3 Emotion Models based on Facial Expressions
By far the most common models of emotion have been created by measuring facial
expressions. This is perhaps the most intuitive one, due to the fact we use our facial
expressions constantly to express our internal feelings, and as a universal method of
communication. As infants, before we learn to speak languages we intuitively
recognise emotions in those around us and intuitively communicate back. In fact
there have been studies conducted on neonates (newborn infants) by (H Oster and P
Ekman 1978 [41]) and blind infants and children by (Charlesworth and Kreutzer
1973 [10]) showing that the basis for facial expression as a medium of emotion
expression is hardwired in us from the beginning as there is simply no time to learn
this behaviour. This strongly supports the view of the Darwinian (evolutionary)
theorists, who believe that basic emotions have evolved through a process of natural
selection. The basis for this point of view was (C Darwin 1872 -1998 [16]) where he
argued that all mammals show emotion reliably on their faces, and, therefore, it
could not be an entirely culturally- learned phenomenon.
��the young and the old of widely different races, both with man and
animals, express the same state of mind by the same movement�
(C Darwin 1872 -1998 [16])
Ekman has been at the forefront of the study of human facial expression related to
emotion. He was the first to show the existence of a universally, cross culturally
recognisable set of human facial expressions, when he travelled to the most remote
villages in the jungles of Papua New Guinea, and found that the tribesmen there
could instinctively and accurately recognise and categorise his set of facial
expressions by their appropriate emotions.
On the back of this Ekman postulated and set out to prove that there are a set of rules
that govern the way we interpret facial expressions. From this he, along with Friesen,
19
produced perhaps the most popular and useful standardised model for the
categorisation of the physical expression of emotions. This model, called the Facial
Action Coding System (FACS) classifies every anatomic human facial expression.
FACS identifies combinations of over 45 different action units (AU�s), which relate
to the contracting or relaxation of facial muscles. It also outlines the rules for the
reading and interpretation of these action units, which allow us to determine a
subject�s emotional state. An outline of these rules can be seen in the example below,
which shows (P. Ekman & W. V. Friesen 1978 [24])�s FACS models for fear,
hapiness and disgust.
Fear = Action Units: One, two and four, or, more fully, one, two, four,
five, and twenty, with or without action units twenty-five, twenty-
six, or twenty- seven; the inner brow raiser (frontalis, pars
medialis) plus the outer brow raiser (frontalis, pars lateralis) plus
the brow-lowering depressor supercilli plus the levator palpebrae
superioris (which raises the upper lid), plus the risorius (which
stretches the lips), the parting of the lips (depressor labii),
and the masseter (which drops the jaw).
Happiness = Action Units: Six and twelve - contracting the muscles that
raise the cheek (orbicularis oculi, pars orbitalis) in combination
with the zygomatic major, which pulls up the corners of the lips
Disgust = mostly A.U. Nine, the wrinkling of the nose (levator labii
superioris, alaeque nasi), but it can sometimes be ten, and in either
case may be combined with A.U. fifteen or sixteen or seventeen.
(See Figure 0-1 for illustrations of first 20 action units)
Human facial expressions have a powerful link with our emotions, so much so that
the mere expression of emotion, whether made voluntarily or involuntarily, has an
impact on our emotional state. (Silvan S. Tomkins 1962 [53]) proposed that the
physical changes and sensations resulting from the facial expression of emotion, are
in fact the source of the qualitatively different feelings of emotion. E.g., happy from
sad, fear from anger etc. (R. Tourangeau and P.C.Ellsworth 1979 [54]) showed that
because of what they called the �facial-feedback system�, an expression you do not
even know you have, can create an emotion you did not choose to feel. It has also
20
been shown that when we experience a basic emotion, a corresponding affective
signal is sent to the areas that control the muscles of the face. This is evident in
stroke patients that have suffered damage to what is known as the pyramidal neural
system. These individuals are seen to be able to laugh normally to a joke, but are
unable to smile on command.
These arguments suggest that there may be a two-way link between the emotion
centres of the brain and the areas that control facial expression and that they may
even be the controlled by the same areas or mechanism in the brain.
3.2.4 Multi-Dimentional/Continuous Model of Emotion
�As everyone knows, emotions seem to be interrelated in various but
systematic ways: Excitement and depression seem to be opposites;
excitement and surprise seem to be more similar to one another; and
excitement and joy seem to be highly similar, often indistinguishable�
(Russel and Bullock 1985 [48])
So far in the models of emotion outlined I have been discussing emotions as discrete
phenomena, i.e. it must be either joy or surprise that is being felt, and not a bit of
both. Emotions however are not discrete but continuous. Because of this continuous
nature of emotion we should search for a model that defines emotion within a space
where any given emotional state can be represented. One such model was suggested
by Wilhelm Wundt, a German psychologist who is generally regarded as being the
founder of experimental psychology. Wundt suggested representing emotional
experience in a 3-dimensional space, where each emotion would be represented by a
measure of Valence, Arousal and Potency. He described valence as being the master
dimension, and is the value used to represent a subjects general positive or negative
emotion. Arousal or activation is used to represent a degree of a subjects
psychological engagement, alertness, or excitement, with or about the object of the
emotion. And finally potency is used to refer to the subjects sense of power or
control over events. Potency is generally left out of most implementations of
Wundt�s model as emotion studies tend not to have the scope to necessitate its
21
inclusion. Instead a 2D model is used, this is called the 2D Valence/Activation
model. This model has stood the test of time having been used extensively over the
years by psychologists and researchers of cognitive human emotion.
Studies by (Russel and Bullock 1985 [48]) and their use of multi-dimensional
scaling1 has successfully shown the usefulness and adequacy of the 2D valence /
activation model in describing emotion. The advantages of this model include its
simplicity and its ability to show not only discrete emotional states, but to describe
the distance between these discrete emotions.
Figure 3-1: 2D Valence / Arousal Space and the distance between emotions
(Russel and Bullock 1985 [48])
(Russel and Bullock 1985 [48]) got adult subjects to sort a set of pictures of facial
expressions, into �groups of people who feel alike�. The results of these groupings
were modelled using a 2D valence/arousal space, and used to show the distances
between a set of basic emotions. Their findings can be summarised as follows (also
see Figure 3-1):
1 Multi-Dimensional Scaling is a statistical procedure for determining the number of dimensions
needed to adequately describe the distances among a set of objects � e.g. Emotions
22
Happiness was placed most distant from all other emotions measured and
firmly with a positive valence.
Anger was placed close to disgust, but far from sadness and less far from
fear.
Sadness was placed farthest from fear (among the negatively valenced
emotions), followed by anger and disgust.
Fear was conversely placed farthest from sadness, followed by disgust and
anger.
Disgust was placed close to anger and almost equidistant from sadness
and fear.
Research undertaken by (P.Shaver, et al. 1987 [43]) provides evidence for
the fundamental nature of the valence dimension regarding emotion. Their
study concentrated on the categorisation of the meanings of a list of 135
emotion words. Subjects were again asked to place similar emotion words in
the same category. Using this data, co-occurrence matrices were produced
for each of the subjects illustrating the conceived similarity of the emotion
words. These matrices were then aggregated so that each cell in the resulting
matrix contained the percentage of subjects who categorised the two
emotion words as being similar. Finally, this matrix was subjected to a
hierarchical cluster analysis mechanism which further categorised the
emotions into a hierarchical emotion space. (See Figure 3-2 below)
23
Figure 3-2: Simplified version of emotion hierarchy
(P.Shaver, et al. 1987 [43])
3.3 Conclusion
Emotions cannot be objectively observed, directly measured or wilfully elicited.
Neither it seems can a definition be found for what an emotion is. Only subjective
definitions of emotions are possible (how they feel to us). What we can measure and
describe are some of the behavioural & physiological components or affects of
emotion. A broad definition of these components of emotion was outlined by (P.
Kleinginna and A. Kleinginna 1981 [35]).
In an attempt to better understand emotions, without requiring a concrete definition
we turn to models of emotion and begin to agree upon a set of basic emotions.
Regardless of whether you think that emotions are learned from birth, taught to us by
society and culture, or have developed through the process of evolution getting
passed down to us by our ancestors, it can be generally agreed that there exists some
set of basic emotions. These basic emotions can be viewed as either the most
universally taught emotions by society or those passed down in order to afford us
with the best chances for survival. A generally agreed set of basic emotions,
regardless of what mechanism or characteristics you use to define what a basic
emotion should be, are outlined below.
24
Anger /Rage Fear/Anxiety Happiness/joy Sadness/Sorrow/Grief
Disgust Surprise/Astonishment
When given the task of recognising and correctly categorising the emotion felt by a
subject by only measuring affective signals or components of that emotion, whether
they be subjective accounts, facial expressions or vocal utterances, the choice of
emotion model used is an important one. The simplicity, effectiveness and proven
adequacy of Wundt�s 2D valence/activation space as a model of human emotion and
emotion in general make it my favourite in this regard. What�s more, this
representation seems intuitively closer to real feelings, and provides the ability of
extracting emotion labels from a continuous representation of any sensed affective
signal. It also allows for the searching of the valence/arousal space for clusters of
emotion, which once identified could be used as the affective cue�s to represent
desired actions to be carried out on behalf of a user for the control of an application.
This classification of affective spaces also lends itself nicely to the modeling of
continuously - sensed emotional components such as those coming from the central
nervous system. The remainder of this section concentrates on justifying my choice
of the 2D valence/activation space with some supporting research and conclusion.
First of all, on examination of Figure 3-2 by (P.Shaver, et al. 1987 [43]) in support of
the 2D valence/arousal model of emotion, the immediate logical separation of
emotion into general positive and negative categories provides legitimacy to Wundt�s
view of valence as the master dimension representing a measure of an emotion�s
positivity or negativity. The presence of Surprise on the border line between positive
and negative may suggest that it should not be categorised as an emotion but rather a
higher level of arousal as it can be linked with both positive emotions and negative
emotions such as joy and fear. This may also explain Ekmans difficulty with finding
a specific pattern of autonomic nervous system activity for his basic surprise
emotion.
In Figure 3-2 it can also be seen that the emotions at the intermediate level: love, joy,
surprise, anger, sadness and fear, correspond closely both to the basic emotions
described in (Paul Ekman 1999 [19])�s basic emotions and (Silvan S. Tomkins 1962
[53]) �Innate Affects�. This shows that the hierarchical groupings of emotions
25
modelled by (P.Shaver, et al. 1987 [43]) using the 2D valence/arousal space
correspond to those categorised using the discrete models of Ekman and Tomkins.
(R. Cowie, et al. 2001 [13]) also used a 2D valence/activation space to model and
assess emotions derived from human speech. This model has stood the test of time
having been used extensively over the years by psychologists and by researchers of
human emotion, proving the models versatility.
Finally further analysis by (P.Shaver, et al. 1987 [43]) of their co-occurrence matrix
using multi-dimensional scaling (See footnote1 p21) revealed the three dimensions
first originally suggested by the founder of cognitive psychology, Wilhelm Wundt,
valence, arousal and potency.
26
Chapter 4: Affective Signals
4.1 Sources of Affective Signals
The area of affective computing relates to human computer interactions where a
device or system has the ability to react appropriately to its user�s emotional state.
Such a device must have the ability to sense at least one of the many affective signals
exhibited by an emotional human user. Sources of these affective or emotional
signals include, facial expression, tone of voice or speech patterns, gestures, body
language, etc
What all these affective signals do have in common is that they are all conscious,
communicative signals which we use continuously to express ourselves to others
around us. They add meaning or affect to what we say and do, and as such they are
used as descriptive tools, not just to communicate our own moods and emotions, but
also to act out or mimic those of the people around us. In order to describe
someone�s reaction to a piece of news, for instance, we would mimic the facial
expression and movements they made at the time.
We use many of our affective signals as natural, channels of expression in our
everyday communications and interactions with others. Even though these
expressions are easily observed and their meaning is generally understood, it cannot
be clear whether or not they are being used to convey a subject�s own emotions or to
describe or mimic someone else�s. Due to their descriptive nature all these affective
signals can, at least to some extent, be faked and therefore do not always present an
accurate measure of an individuals affective state. For this reason they are not
considered to be suitable sources of affect for this work.
27
Other possible sources of such information can be derived from a subject�s use of
established computer interfaces, such as mouse movements, number of mouse clicks,
patterns and indeed the force with which keys are pressed. This type of information
would, for instance, exhibit whether a user was frustrated with, or very tentative and
unsure of their control of a computing system. These sources of affective
information, however, require a tangible interface to be present, such as the keyboard
and mouse. This will not be the case for most ubiquitous applications.
I feel that it is more advantageous to use some less obvious affective signals that are
not so generally used for communication, only reflect a subjects own emotions, and
can not be easily or consciously manipulated. Such affective signals can be derived
from the detection of changes in our central nervous system (EEG, MEG, MRI, PET
) and peripheral nervous systems (EMG, GSR, Heart Rate). I will next examine each
of these features in turn.
4.2 Affect from Peripheral Nervous System (physiological sensors)
4.2.1 Electromyogram (EMG)
An Electromyogram (EMG) is used to record the electrical activity of muscles. When
a muscle is flexed or relaxed it produces an electrical signal. The strength of this
electrical signal can be used as an indicator of the level of muscle activity, as the
force with the muscle was contracted proves proportional to the strength of the
electrical signal generated. Another advantage of EMG allows even minor muscular
activity to be detected, even though it may not be apparent visually. This is
especially true for facial expression recognition studies, where image recognition
alone would overlook many of the important so called �tell tale� expressions. These
�tell tale� expressions refer to the unconscious and uncontrollable facial expressions
which manifest themselves automatically and often for only a fraction of a second. It
has been shown that these muscle activity gives a very reliable measure of what a
subject is actually feeling, and are very difficult or imposable to control or to fake.
28
Figure 4-1: Signals from EMG on the masseter muscle in micro-volts, indicating periods
during which a subject was asked to express, no emotion, anger, hate, grief, love, romantic
love, joy and reverence (J. Healey and R. W. Picard 1997 [25]).
EMG have mainly been used in facial recognition studies. Where only a few EMG
sensors are used the sensors are often fixed to either the masseter (jaw muscle) to
detect jaw clenching, an indicator of anger and frustration, or fixed to the frontalis
(the muscle controlling the eyebrows) to detect particular movements of the
eyebrows indicating, stress, surprise, bewilderment etc. However, having sensors
affixed to the face proves very uncomfortable, and in affect hampers the very
expressive movements which we are attempting to detect. For this reason the use of
EMG, at least on facial areas, would not be ideal for a future wearable affective
ubiquitous device.
4.2.2 Galvanic Skin Response (GSR) or Skin Conductivity Response (SCR)
GSR sensors are used to record the skins conductance over a small area. It is known
that sweat glands (eccrine glands) present in the hands and feet show a galvanic
effect. This effect is triggered not by thermoregulation, but by signals present in the
sympathic nervous system. These emotional signals such as arousal, are shown to
cause near instant changes in the level of sweat in these glands, in turn affecting the
skins conductivity in the area the glands are present. GSR measurements are
generally identified as the most reliable indicators of a subjects arousal, therefore a
GSR reading indicates the subjects level of arousal, on a scale from calm through
agitated, to outright frustration and anger.
29
Figure 4-2: Signals from SCR in micro-Siemens, indicating periods during which a
subject was asked to express, no emotion, anger, hate, grief, love, romantic love,
joy and reverence (J. Healey and R. W. Picard 1997 [25]).
GSR is seen as the most popular affective signal used in determining emotion, and is
used extensively by researchers in the area of affective computing. The actual
sensors are generally fixed about two inches apart, either to the top and bottom of the
middle finger or on the base of two adjacent fingers. This arrangement is quite
comfortable and unobtrusive, however, it does not bode well when washing ones
hands. On the down side, conductance readings are known to vary due to external
factors such as ambient temperature and humidity, necessitating the addition of extra
temperature and humidity sensors on the subject, used to negate these changes.
4.2.3 Heart Rate and Blood Volume Pulse (BVP)
Heart rate sensors are generally used to record inter-beat intervals or heart rate
variability. These measures are generally used only as discriminatory factors in
determining changes in a user�s affective state and do not carry significant, affective
or emotional information when viewed in isolation. This is due to the fact that it
cannot be determined whether changes in inter-beat intervals are the result of
physical activity, emotional stress, or the subject having just had a cup of coffee. It is
only in relation to other affective signals that this type of information becomes
useful.
Figure 4-3: Signals from Heart Rate in bpm, indicating periods during which a
subject was asked to express, no emotion, anger, hate, grief, love, romantic love,
joy and reverence (J. Healey and R. W. Picard 1997 [25]).
30
4.3 Affect from Central nervous system (The Brain)
4.3.1 Electroencephalography EEG
EEG is traditionally known as a medical imaging technique, used to measure and
study electrical potentials caused directly by underlying brain activity. These tiny
signals can be measured unobtrusively from the surface of the scalp using small
metal electrodes, and are caused by the collective activation of neurons (brain cells)
in the underlying brain areas. These activations cause local current flow between
neurons, indicating that some type of mental processing is taking place in that area.
The signal produced, along with the area of the brain from which the potential was
measured, can be used to give an indication of the type of mental task being
performed. EEG has been found to be a very powerful tool in the field of neurology
and cognitive science, due to its capability to show patterns of electrical signals
which can then be used to determine brain activity. The brain signals most apparent
in EEG data emanate from the area of the brain directly beneath the scalp, the
cerebrum, which contains the centres for movement initiation, conscious awareness
of sensation, complex analysis, and most importantly the expression of emotion and
behaviour.
These activity specific areas can be studied and measured using a technique known
as event related potentials (ERPs). ERPs represent the averaged electric brain
response to some form of sensory stimulation. Using this mechanism the reaction of
different areas of the brain to chosen stimuli can be investigated in order to identify
the various centres involved in reacting to these stimuli. For example using
movement as a stimulus, the centres of the brain controlling this movement can be
estimated, or by inducing the emotion of anger in a subject, the location of the source
of this emotion can be approximated from the EEG signals. (this method is further
described in Chapter 5: )
31
Figure 4-4: Brain wave samples indicating the four
most dominant frequencies.
The patterns of output evident in an electroencephalogram are commonly referred to
as brainwaves. These brainwaves have been classified into four main frequency
categories, used here to indicate the most basic and obvious form of affective
information evident from EEG signals. These categories can be used as a measure of
a subject�s alertness, simply by determining which brainwave is most dominant
within the raw EEG data;
• Beta relating to a high level of alertness, involving cognitive processing, reasoning
and concentration.
• Alpha relating to calmness, relaxation and problem solving.
• Theta relating to near sleep, daydreaming.
• Delta relating to deep dreamless sleep.
(Affect and EEG are examined thoroughly in Chapter 5: )
4.3.2 MEG, fMRI, PET � Other Advanced Brain Imaging Techniques
Magnetoencephalography (MEG), Functional Magnetic Resonance Imaging (fMRI)
and Positron Emission Tomography (PET) - are all techniques of brain imaging,
similar to EEG. MEG detects the same electrical signals detected by EEG, however it
32
measures the magnetic effects caused by the flow of current between neurons. This
has an advantage over EEG signals in that it allows greater temporal resolution,
allowing the source of electrical signals from within the brain to be more accurately
located and modelled. However, it has one major drawback, in that it is highly
susceptible to magnetic interference, even from the earths magnetic field and
requires shielding from this in order to correctly operate, making it unsuitable for this
application. The fMRI measures blood flow in the brain. The thinking here is that
active areas in the brain require more oxygen and energy to operate than inactive
areas, therefore areas exhibiting increased blood flow are areas which are currently
active. PET measures the emission of positrons, tiny particles which are emitted from
a radioactively marked substance (generally oxygen, water or glucose) which must
be first administered to the subject. Measurements of circulation of these positrons
allow for the complete imaging of the bodies systems, including brain activity and is
generally used to identify a number of diseases. The use of fMRI and PET imaging
in human emotion studies has increased significantly in recent years (A. R Damasio,
et al. 2000 [15, M. L Phillips, et al. 1997 [44]), however, they are both unsuitable
technologies for this application, as they are distinctly immobile and require
prohibitively expensive equipment.
4.4 Conclusion
In this section I have outlined the main sources of emotion information most often
used in affective computing systems. The majority of systems developed in this area
use affective information derived from the peripheral nervous system, namely GSR,
EMG and Heart Rate. However, in my opinion, the most promising of the affective
signals discussed is EEG. Despite this, very few studies directly related to the area of
affective computing have used EEG as a source of a subject�s emotions, even though
cognitive research has shown strong correlations between their subjects� brain
activities and subjective emotions.
It is known that even the division of raw EEG data into the various brainwave
frequencies, outlined above, can provide information regarding the levels of alertness
of a subject. This, along with the measurement of event related potentials can be
shown to provide accurate and dependable measures of a user�s affective state, and
33
has been played a pivotal role in studies of brain asymmetry and emotion. Studies
have shown that differences in the electrical activity between the two brain
hemispheres can be used to predict emotional responses to various stimuli, including
emotional stimuli, when measured from the frontal regions of the brain. The use of
EEG in brain controlled interfaces also attests to the suitability of the technique in
extracting and detecting cognitive information. The placement of scalp electrodes
over different areas of the brain known to control certain cognitive processes may
also allow for a choice of affective signals to be targeted and monitored for
application specific purposes. It could also be argued the use of only scalp electrodes
may be preferable over other affective sensors such as EMG or GSR which are most
often affixed to the face and hands, because of the fact that we simply don�t use our
scalp for that much and the electrodes placed there wouldn�t really get in our way.
For these reasons and others outlined in Chapter 5: I have decided to use EEG as the
source of affective signal for this project.
34
Chapter 5: The Brain, EEG & Emotion
�A Window Into The Brain�, Hans Berger
Tiny electrical potentials emanating from the brain�s cerebral cortex when measured
from the surface of the scalp can be amplified to produce an Electroencephalogram,
or EEG. These electrical potentials, caused by the collective activation of neurons
(brain cells) in the underlying brain areas, can be measured unobtrusively from the
surface of the scalp using small metal electrodes. The signals are caused by local
current flows between neurons, and indicate when mental processing is taking place.
The part of the brain that produces the clearest EEG signals lies just below the
surface of the scalp and is called the cerebral cortex. This region consists of the right
and left hemispheres of the brain and is said to control conscious experience,
including perception, thought, planning and most notably the management of
emotions. EEG has been found to be a very powerful tool in the field of neurology
and cognitive science, due to its is capability to show patterns of electrical signals
which can then be used to determine and differentiate between different types of
brain activity.
Hans Berger measured the first EEG in Germany in 1929 thus earning the
recognition of �Father of Electroencephalography�. From seventy-three EEG
recordings taken from his son, Klaus, Berger noted that the electrical patterns emitted
from his son�s brain could be seen to shift with attention, showing different patterns
and frequencies when Klaus was at rest, to when he was at rest but focusing on
solving a maths problem. Berger went on to categorise four main frequency bands
which could be shown to correlate with different levels of brain activation. These
frequency bands are commonly referred to as brain waves. (see Figure 4-4: Brain
wave samples indicating the four most
dominant frequencies.) These various brain waves can be used as a measure of a
35
subject�s alertness, simply by determining which brainwaves are most dominant
within raw EEG data;
• Beta relating to a high level of alertness, involving cognitive processing, reasoning
and concentration.
• Alpha relating to calmness, relaxation and creativity.
• Theta relating to near sleep, daydreaming.
• Delta relating to deep dreamless sleep.
These rather crude measures of affective state can already allow us to detect whether
a subject is alert, asleep, daydreaming or relaxed. However in order to extract more
useful affective information it is important that I first explore the various parts of the
brain that are involved in the processing of our emotions. This will allow us to
deduce how, and from which parts of the brain, to best measure our affective state.
5.1 How the Brain Processes Emotions
Historically research in the area of the brain and emotion generation was split into
two opposing groups. The peripheralists� view postulated the source of our
subjective feelings of emotion as simply a reaction to changes in the peripheral
nervous system. They believed that we become sad because we started to cry, or that
we become scared because we started to shake. This view was countered by the
centralists, who argued that the brain is the true source of our emotions which in turn
are the cause of the physiological changes evident in our peripheral nervous systems.
Recent studies suggest, in a triumph for bipartisan thinking, that they were in fact
both correct, supporting the view that, for example, if you smile you generally feel
happier, and if you feel happy you generally smile.
Evidence in support of the centralist view was first outlined by (Papez J.W. 1937
[29]) in his work entitled, �A Proposed Mechanism of Emotion� which stated that
anatomical structures in the brain must �deal with the various phases of emotional
dynamics�. His work in attempting to define the structures of the brain that deal with
emotions, (which bizarrely involved the injection of rabies into cats, to observe its
36
progress through the brain), led to his discovery of what is now called the Papez
Circuit. Unfortunately for Papez this circuit proved to have little involvement in
emotion but did form the basis of the limbic system theory of emotion.
Figure 5-1: (left) Cerebral Cortex, showing the major lobes (frontal, parietal, temporal and occipital)
and the brain stem structures (pons, medulla oblongata, and cerebellum). (Right) Limbic System
inside the brain. Consists of the fornix, hippocampus, cingulate gyrus, amygdala, parahippocampal
gyrus and thalamus.
Dr Paul MacLean, MD, the former director of the Laboratory of the Brain and
Behaviour at the US Institute of Mental Health, building upon Papez�s work,
outlined the importance of the limbic system in emotion processing (P.D MacLean
1949 [37, P.D MacLean 1952 [38]). The Limbic System, also known as the old
mammalian brain consists of a number of structures located inside the cerebral
cortex, including the fornix, hippocampus, cingulate gyrus, amygdala,
parahippocampal gyrus and the thalamus. (See Figure 5-1 above). MacLean went on
to develop a model of the brain based on its evolutionary development, known as the
Triune Brain (P.D MacLean 1973 [39]). This model consists of three parts, hence the
name triune, the limbic system making up the second part. MacLean proposed that
the evolution of the human brain could be differentiated into three stages, such that at
each stage, new layers of brain developed upon the old ones. In a process similar to
37
the way that, in object oriented programming, the functionality of one class can be
inherited by another in order to adapt to more specialised tasks without loosing any
of the underlying functionality. In this way MacLean proposed three classes of brain
function, each inheriting from the others, to form more complex and intelligent
systems.
Figure 5-2: MacLean's Triune Brain
The three layers outlined by MacLean are:
The Reptilian Brain (R-Complex) consists of the brainstem and cerebellum and is
considered the oldest part of the brain. Similar structures, found to dominate the
brains of reptiles, explain the name, but also go to illustrate the types of behaviour
governed by this primitive brain, which seem aptly reminiscent of our notions of
reptilian behaviour. This part of the brain is described as being obsessive,
compulsive, ritualistic and paranoid. It is also evident that the function of this
primitive brain is not to learn but to keep repeating the same actions over and over
again, as it does to control our respiration, circulation and autonomic functions.
The Limbic System (old mammalian brain), formed in part by the Papez Circuit, is
the middle part of the trio and corresponds to the brain of early mammals. It is seen
as the source of our basic emotions and instinctive behaviours such as feeding,
fleeing, fighting and sexual reproduction. The Limbic System is perhaps the most
important component of the brain when it comes to emotion, as it seems be the
38
primary seat of our basic emotions and emotion charged memories. MacLean notes
that when this part of the brain is stimulated by mild electrical current, emotions such
as fear, rage, joy, pleasure and pain are produced. The amygdala, a component of the
lymbic system is thought to be principally involved in our ability to associate certain
events with exacting emotions. Another component the hippocampus, seems
concerned with the conversion of experienced events into long-term memories and in
the recollection of these memories. Therefore, the limbic system provides the
mechanisms which can be used to learn from past experiences, giving rise to
emotionally charged memories that illicit feelings such as fear, pity, anger and
outrage. Fight or flight reactions, for example, coming from the reptilian brain can
also trigger these emotions in the limbic system, this has allowed instinctual
behaviour to evolve throughout our development, affording us the best chances for
survival.
The limbic system also has vast interconnections with the third part of the brain, the
Cerebral Cortex. The Cerebral Cortex (neo-mammalian brain, neocortex) is known
as the superior and rational brain. This brain corresponds to those of primates and
consequently to the human brain. The cerebral cortex makes up five sixths of the
human brain and is what has given us our capacity to exhibit unique abilities such as
language, writing, complex logical thought, rationalisation, visualisation, creativity
etc. MacLean describes the cerebral cortex as �the mother of invention and the father
of abstract thought�.
The cortex is divided into the better known left and right hemispheres, with the left
hemisphere, concerned with logical, rational and verbal abilities, controlling the right
side of the body, and the right hemisphere, concerned with spatial, artistic, musical
and abstract abilities, controlling the left side of the body.
As far as our basic emotions are concerned, the cortex is where we learn to control
them, and try, on some occasions to suppress them. MacLean considers the pre-
frontal cortex as the essential area concerned with the control and conscious
management of our emotional reactions. This area also allows us to plan and even
anticipate our own emotions, giving our rational brain the chance to step in, with our
pre-conceived checklist of cultural rules, as it were, before we find ourselves reacting
directly to the powerful basic emotions emanating from our more primal brains. The
39
importance of the pre-frontal cortex in emotion processing is strongly supported by
several studies, using EEG, MEG, PET and fMRI brain imaging techniques,(R.J.
Blair, et al. 1999 [3, R.J. Davidson 1992 [17]), (M. L Phillips, et al. 1997 [44])
among many others. Many of these studies concentrate on collecting evidence in
support of several hypotheses, which generally relate to the lateralisation of
emotions2.
5.2 The Lateralisation of Emotion
In section 3.2 Models of emotion, I concluded that Wundt�s 2D valence/activation
space as a fitting model of human emotion, however there are other models of
emotion that relate directly to the processing of emotions in the brain which must
also be examined, in order to attempt the extraction of affective information from
EEG data. The majority of these models examine the function of the frontal lobes in
the emotion processing and deal with the concept of hemispheric specialisation, or
the lateralisation of emotion. Two such hypotheses are the Right Hemispheric
hypothesis and the Valence hypothesis, both of which I will now examine.
5.2.1 Right Hemisphere hypothesis vs The Valence Hypothesis
The right hemisphere hypothesis (RHH) seeks to prove the dominance of the right
hemisphere over the left for the processing of all forms of emotion expression and
perception. The valence hypothesis on the other hand, states that the right
hemisphere is only dominant for negatively valenced emotions, while the left
hemisphere is dominant for the expression and perception of positive emotions. Both
hypotheses are supported by empirical evidence however neither has achieved
prominence. In fact some new studies using brain imaging techniques suggest that
both hemispheres are used in the processing of positive and negative emotions, the
right hemisphere exhibiting dominance only to certain stimuli.
2 lateralization of emotions - the act of using one hemisphere more than another in
the processing of emotions.
40
Evidence in support of the RHH first came from brain lesion studies carried out by
Jules Bernard Luys in 1881. He recognised that patients suffering from damage to
their right hemisphere, as a result of stroke or some other causes, which I generally
describe as brain lesions, exhibited only passive and indifferent behaviour acting
entirely without affect, whereas patients suffering with lesions on the left hemisphere
were described as emotionally volatile. Luys believed that this was a result of
damage to the normal emotion inhibiting centres of the brain, which he postulated
must then reside in the right hemisphere of the cortex. Behavioural evidence also
lends its support to the RHH. The left side of the face, for instance, is more
comprehensive and communicative in facial expressions, the muscles of which are
controlled by the right hemisphere of the brain (H.A. Sackheim and R.C. Gur 1978
[49]) . It can also be shown that the right hemisphere is also better at perceiving
emotion then the left hemisphere. Evidence for this comes again from lesion studies,
where patients with right hemisphere injury exhibit greater difficulty recognising
positive and negative emotions in faces than those with left hemisphere injuries (D.
Bowers, et al. 1985 [6]).
Contrary to this, evidence in support of the valence hypothesis (VH) seems also to be
readily available. It must be said that more of this evidence comes from brain
imaging techniques such as EEG, PET and fMRI and less from brain lesion studies
than it does in support of the RHH. (R.J. Davidson, et al. 1979 [18]) show empirical
evidence, taken from EEG studies of adults and infants, that the left frontal region of
the brain is involved in the experience of positively valenced emotions, the right
frontal region being involved with negatively valenced emotions. This evidence is
also supported by (N.A. Jones and N.A Fox 1992 [30]), who show greater left frontal
EEG activity associated with the viewing of film clips showing pleasant scenes and
greater relative right frontal EEG activity associated with the viewing of unpleasant
film excerpts. (L.A. Schmidt and L.J. Trainor 2001 [50]) noted similar results using
musical excerpts to illicit positively and negatively valenced emotions across the left
and right frontal regions respectively.
More importantly for this study (L.A. Schmidt and L.J. Trainor 2001 [50]) studies
show that the pattern of frontal brain activity reflects both the valence and intensity
of emotions. This proves that by examining both asymmetry and overall power of
frontal brain activity it is possible to distinguish between emotions within the
41
valence/activation model using EEG data alone. This adds final justification for the
conclusion drawn in section 4.4 to base the affective choice interface system
exclusively on EEG measurement of affective signals.
5.3 The recording of EEG
Now that I have demonstrated that it is possible to derive a user�s affective state from
their EEG signals alone, it is important that I outline further the process of EEG
acquisition.
EEGs are differential amplifiers. They take measurements of electrical potentials
emanating from the brain, between two points on the scalp using metal electrodes
and conductive gel, amplifying the difference between these measurements. All EEG
amplifiers therefore require at least two electrodes, generally referred to as the active
and reference electrode. In any practical amplifier a ground electrode is also
required, this electrode does need not be connected to the scalp and is often
connected to the ear lobe, or mastoid. It is used to record noise which can then be
identified in the active and reference signals for removal. When taking measurements
from more then one channel (area of the brain) two electrodes per channel are
required, in addition to the ground electrode. The EEG Amplifier used in the
development of this affective interface, provides just two channels.
Sources of noise in EEG recording most often result from; body movements that
result in the connected wires being pulled, resulting in the disruption of the
electrodes, muscle movements including minor movements such as eye blinks and
eye movements, and changes in the quality of scalp connection due to using too
much conductive gel, gel drying out or the subject sweating.
5.3.1 Types of electrode
The most common types of electrodes used include; disposable gel electrodes,
reusable disk electrodes (gold, silver, stainless steel or tin), electrode caps and saline-
based electrodes. In the development of the affective interface, simple disk electrodes
in combination with Ten20 conductive gel were used. These electrodes simply stick
42
to the surface of the scalp, by the application of a small amount of conductive gel.
The actual metal part of the electrode is not in contact with the scalp; instead the
conductive gel between the two makes the connection.
Alternative sensors available include flextrodes. These flextrodes are saline-based
electrodes which allow for better connection through the hair and provide a more
attractive alternative to gels and pastes. For the purposes of affective measurement
using EEG, the use of flextrodes in combination with either headbands or electrode
caps would provide a more practical wearable interface than using disk electrodes
with conductive paste. If this type of interface proves popular one can imagine the
design of EEG sensors going along the lines of audio headphones, or some more
adventurous sunglass designs. These more fashionable designs would still
incorporate the some basic flextrode design.
5.3.2 Electrode placement
With the increased research and use of EEG devices, a standard description for
electrode placement was developed by the International Federation in
Electroencephalography and Clinical Neurophysiology in 1958. This is known as the
10-20 electrode placement system. The name comes from the suggested arrangement
of electrodes in relation to areas of the scalp at intervals of 10 or 20 percent of the
proportional distance from easily recognisable landmarks such as the ears and nose.
Figure 5-3: 10-20 Electrode Placement System
43
5.4 Conclusion
In this chapter I have examined the relevant areas of emotion processing in the brain.
I presented information regarding the recording of EEG signals from the brain
including the choice of various electrodes and the 10-20 electrode placement system.
I examined MacLean�s Triune model of the Brain, based on its evolutionary
development (P.D MacLean 1973 [39]) and the role of the various parts of the brain
in the generation, processing, memory and management of emotions. Most
importantly, however, I have demonstrated that it is possible to derive a user�s
affective state from the recording and inspection of their EEG signals alone. This
pivotal piece of evidence allows me to now tie together all the research presented
regarding the valence/activation model of emotion, the choice of EEG and the
measurement of EEG signals from the pre-frontal cortex of the brain and to finally
outline the best design for the affective choice interface system.
44
Chapter 6: The Affective Interface System
The Affective Choice system is an affective computing user interface. It uses
affective signals derived from the emotion centres of the user�s brain and applies
these signals to a support vector machine (type of neural net) which learns to classify
these signals into various emotion states. This system is designed to provide
application developers with an easy to implement affective user interface which will
afford their systems an affective sense of their user�s emotional state, which can be
exploited to provide unique services to the user based entirely on their mood. The
system can detect and describe a user�s affective patterns (emotional cues), and learn
to equate these emotional patterns with the user�s desired actions (within an
application) through a reinforcement-learning framework.
6.1 Overview of the Affective Interface System
Affective Choice System
Configure System
Train Affective System
Add / Update Training Data
(Re) Build Affective Model
Predict Users Affective State
Sense User Affective Signal
Start System
Application User
Application Developer
Application
Cadifra Evaluation
Figure 6-1: Use Case Diagram of Affective Choice System
45
The affective interface system consists of two main components, the Brainmaster
EEG sensor and data acquisition module, and the Support Vector Machine, used for
classification and regression of EEG data. Both these components are brought
together to make up the affective choice system, which is largely a prototype
application programming interface. Written in C and C++, this �Affective API�
provides a high level object oriented interface, controlling the retrieval and
processing of EEG data, along with training and learning mechanisms used in the
classification of this EEG data into various forms of affective signals. The most
important work in the design of this system has already been outlined in the
preceding chapters, and largely involves the careful research and examination that
was pivotal to the choice of the correct components for the system. With these in
place, the next challenge would include the design of a reusable, easy to understand
and implement interface, suitable for use in future ubiquitous applications and not
least to demonstrate that the final system could in fact fulfil its objectives, as a
affective user interface, and therefore justify its reuse.
The final design of the affective interface system will now be outlined, along with
descriptions of its various components and classes.
6.2 System Components
6.2.1 Brainmaster Module
The Brainmaster Module is a portable, battery powered EEG monitor. It is designed
and marketed for use in the area of clinical psychophysiology3 and neurofeedback, a
technique used, for instance, in the treatment of epilepsy, attention deficit disorder
and rehabilitation of traumatic brain injury among others. The Brainmaster unit
consists of two EEG amplifiers, an analogue to digital converter and a sensor fusion
system. The module automatically applies digital filters, coherence processing and
Fast Fourier Transform (FFT) processing to the raw EEG signal, and provides
3 Physiological Psychology or Psychophysiology: is the branch of psychology that is concerned with
the physiological bases of psychological processes.
46
constantly updated and frequency resolved EEG data on the fly. The module also
applies various noise and artefact detection and reduction techniques. The successive
approximation digitiser runs at 120 samples per second, this can be boosted to 244-
800 samples per second if time critical measurements are required, the default
120samples/sec was used for the duration of development and testing. Conversion
time of 16msec is exhibited in both channels, with eight-bit precision and an
accuracy rating of half the least significant bit. The EEG module automatically
resolves raw EEG in to the following frequency ranges:
One Nominal Range - User Adjusted Delta - 1-3 Hz Theta - 4-7 Hz Alpha - 8-12 Hz Lo-Beta - 12-15 Hz Beta - 15-20 Hz Hi-Beta - 20-28 Hz Gamma - 28-42 Hz
EEG frequencies are measured using a number of techniques by the Brainmaster
module. One technique uses Fast Fourier Transform (FFT) which estimates
the amount of energy emitted over a user selectable time period (usually about one
second). This technique provides greater accuracy, but lacks a fast response. The
other technique applies software digital filters, this provides a faster response, and
readings are separated into their specific bands. Samples taken using this technique
gave preferred higher resolution measurements which proved more useful for this
application.(Ph.D Thomas F. Collura December 7, 1997 [52])
Figure 6-2: Picture of the Brainmaster EEG Module used in the development
of the Affective Choice system, (Thomas F. Collura [12])
47
The Brainmaster EEG module was provided to us by Thomas F. Collura of
BrainMasterTechnologies,Inc. (Thomas F. Collura BrainMasterTechnologies,Inc.
[11])
6.2.2 Support Vector Machine for Classification and Regression
Support Vector Machine (SVM) provides a technique for data classification, often
used in areas such as bioinformatics, and astro particle grouping studies. Support
Vector Machines are learning machines designed to perform pattern recognition and
function approximation or regression estimation tasks. SVMs provide the ability to
map non-linear input from a multi-dimensional space into a multi-dimensional
feature space. From this feature space a linear classifier or classification model is
constructed. SVMs were chosen for this project because they are seen to be more
easily understood than Neural Networks and therefore they provide a platform for
data categorisation which can be more easily adopted and used by application
developers. This is in line with the goals of this project, to provide a reusable, easy to
implement and understand, affective user interface for developers of ubiquitous
computing applications.
Figure 6-3: Graphical Depiction of how SVM learns to classify or group reading into categories.
(LEFT) This depicts a 2D set of training data, containing 3 distinct categories. (RIGHT) Depicts the
SVM attempt to build a classification model for this distribution. This Model can then be used to
determine the category of any future readings.
As depicted in Figure 6-3, the SVM uses a set of labelled training data, this training
data is arranged in a training file (text file) in LIBSVM format as outlined below.
Each line in the training file (training vector) contains one �target value� (Label) and
48
several �attributes� (feature values or dimensions). The goal of SVM is to produce a
multi-dimensional model, which predicts the target value of real-time data instance
attributes accurately.
<Label> 1:<1st Attribute Val> 2:<2nd Att Value> ���.. N:<Nth Att Val> <Label> 1:<1st Attribute Val> 2:<2nd Att Value> ���.. N:<Nth Att Val> <Label> 1:<1st Attribute Val> 2:<2nd Att Value> ���.. N:<Nth Att Val>
Figure 6-4: LIBSVM vector data format
As outlined in (Chih-Wei Hsu, et al. [26]) & (B. Boser, et al. 1992 [5]) support
vector machines (SVM) require the solution of the following optimisation problem:
Training vectors (lines of training file), denoted above, are mapped by the
function into higher and higher (maybe infinite) dimensional spaces. The SVM
finds separating boundaries within these multi-dimensional spaces. Parameter is
the penalty parameter of the error term. A large penalty parameter gives a more
precise model based on the training data, however this is not always desirable, as the
models become less predictive and very specific. A penalty parameter of 1 was used
for general purposes.
There are various kernels to choose from, some linear and some nonlinear. These
kernels are used to map samples into the higher dimensional spaces. The choice of
kernel has a large impact on the performance of the SVM. The best kernel for
general application is the RBF kernel, as it is a non-linear kernel (it can handle the
case when the relationship between labels and attributes is nonlinear), and it exhibits
the best results for general classification purposes(Chih-Wei Hsu, et al. [26]).
There are four basic kernels to choose from:
Linear:
Polynomial:
49
Radial Basis Function (RBF):
Sigmoid:
The LIBSVM C++ Library for Support Vector Machines provided the base class for
the implementation of SVMs in this thesis.
6.3 System Architecture
As shown in Figure 6-5, the Affective Choice Interface consists of 6 classes.
AffectiveInterface, BrainmasterInterface, SVMInterface, SVMPredictor, SVMScaler
and SVMTrainer. The BrainmasterInterface class is a singleton interface for the
setup and retrieval of data from the Brainmaster EEG module. Data retrieved from
the Brainmaster EEG module is sent to the SVMInterface which consists of three
main classes (see below). The SVMPredictor, is used along with the SVMScaler
class to scale and classify the user�s affective data coming from the
BrainmasterInterface, according to an n-dimensional, multi attribute model of the
user�s EEG signals.
Figure 6-5: Shows Class Diagram of Affective Interface System, including ( system components ),
Brainmaster EEG Module, and LIBSVM base class.
50
The result of this classification indicates the predicted affective state (mood,
emotion) of the user based on their current EEG signals, and is derived from the
previously created SVM classification model of the user�s possible affective states.
This allows an application using the interface to simply call the
getCategoryPrediction() method of the AffectiveInterface class, and the system will
quickly return the predicted affective state of the user, 1:angry, 2:calm, 3:focused etc.
The SVMTrainer is used along with the SVMScaler class in the recording of training
data from the user and in the creation and update of the SVM model of that user�s
affective signals. For instance, a happy/sad model can be created by calling the
writeNewTrainingFileData() method specifying the number of samples to take and
the type of EEG signal to build the model from(e.g. alpha, beta, lobeta), when the
user is feeling happy. Later when the user is feeling sad, a number of samples using
the same criteria are appended to the training file by calling the
appendToTrainingFileData() method of the AffectiveInterface. A new model for
happy/sad can now be created using a simple call to the createNewModel() method,
this scales the training data and applies the SVM classifier to the data. The SVM
classifier searches out any patterns or groups within the n-dimensional training set
and produces a model of the attributes of the user�s EEG signals when they are
happy, as opposed to when they are sad.
This model can then be applied to real-time data, using the prediction mechanism, to
let an application know whether the user is happy or sad, based on their affective
signals.
6.4 Overview of the Brainmaster Interface
The BrainmasterInterface singleton class provides a simple interface for the setup
and retrieval of data from the Brainmaster EEG module. Control and retrieval from
the BrainMaster is achieved by reading and writing to memory locations within the
BrainMaster module�s dll memory space. The dll extension library header file (see
Appendix: Source Code), provided with the module, defines all the data offsets into
the BrainMaster dll memory space.
51
Figure 6-6: BrainmasterInterface Class Diagram
This class includes code to access data in the BrainMaster 32bit Dll. Once connected
and logged into the BrainMaster module the class exposes 5 main methods:
printValue(memorySpaceBlock, offset) � prints EEG value at memory location memorySpaceBlock+offset in the BrainMaster dll memory space to std out. getValue(memorySpaceBlock, offset) � returns integer EEG value at memory location memorySpaceBlock+offset in the BrainMaster dll memory space. getNewValue(memorySpaceBlock+offset) � returns the next EEG reading, blocks until new reading is written to the dll memory space. memorySpaceBlock+offset is passed in to identify parameter to be read. getAverageValue(memorySpaceBlock+offset, numSamplesToAverage) � returns the average of a specified number of new EEG samples. Useful to ensure more accurate readings and reduce signal artifacts. setValue(memorySpaceBlock, offset) � is used to write configuration information to the BrainMaster dll memory space control block, in order to configure the EEG module.
All block and offsets into BrainMaster dll memory space are #defined in Dlldefs.h,
see Appendix: Source Code for details.
52
6.5 Overview of SVM Machine Learning Interface
The SVMInterface class provides a high level interface for the easy control and use
of the LIBSVM library (Chih-Chung Chang and Chih-Jen Lin 2001 [9]). The
SVMInterface class is designed to be used by application developers who need not
have any experience using an SVM. It includes a default setup configuration that is
tailored for general application, and will provide good classification for most
implementation purposes. It still, however, allows more experienced developers to
configure the SVM exactly as they please.
Figure 6-7: SVMInterface Class Diagram
The SVMInterface has two constructors. The first provides default values for most of
the parameters required for SVM classification. The only parameters needed are:
• bool produceNewScale: If true, a new SVMScaler is created with its default configuration. This configuration is then saved for future use to scaleConfigFile. If false, an instance of SVMScaler is recreated using the data saved in scaleConfigFile.
• bool produceNewModel: If true, a new model is created by using the training
data and new SVMScaler configuration. This model is then saved for future use to modelFileName. If false, a previously existing model is recreated using the data saved in modelFileName.
• char* scaleConfigFile: Path and filename of scalers save/restore config file • char* trainingFileName: Path and filename of training file (LIBSVM format) • char* modelFileName: Path and filename of model file
53
The second constructor requires that the user explicitly creates an instance of each of
SVMScaler, SVMTrainer and SVMPredictor, and these are passed as parameters to
the constructor.
SVMInterface allows complete control of the underlying SVM classes, SVMScaler,
SVMTrainer and SVMPredictor through the following five methods:
makeModel() � produces a new model from the supplied/updated trainingDataFile. addTrainingData(value, inputLine) � appends training data to training data set, this should be done when the predictor gets a value wrong. This allows the system to learn from its mistakes. predictLine(inputLine) � predicts category classification of data provided in inputLine. testModel(testFileName) � tests the model by predicting values from testFileName and comparing them to the desired 'target' values. scaleFile(trainingFileName) � used to rescale trainingFileName when needed. scaleLine(line) � scales the line according to SVMInterface scaling
configuration.
6.5.1 SVMInterface Component Classes
The three component classes of the SVMInterface themselves provide abstraction to
the LIBSVM system under the categories of scaling, training and prediction. Each of
these three component classes will now be outlined.
6.5.2 SVMScaler
SVMScaler is used to scale all training and input data before it�s used either to create
a model or used with SVMPredictor. It is advised to use scaling in all SVM systems
in order to avoid attributes in greater numerical ranges dominating those in smaller
numerical ranges. Another advantage is to avoid numerical difficulties during the
54
calculation. The use of smaller number ranges also improves efficiency in
classification.
Figure 6-8: SVMScaler Class Diagram
SVMScaler has three constructors. The first constructor is a default constructor. The
second constructor requires the path to the scaler configuration file, which is used to
recreate a previously existing instance of an SVMScaler. The third constructor
allows experienced developers to specify unique configuration of the scaler
parameters.
These parameters include:
• char* saveSettingsFilename: Path & filename of scaler config save file • char* restoreSettingsFilename: Path & filename of scaler config restore file
(usually the same) • double lowerBound: Lower x bound (usually -1) • double upperBound: Upper x bound (usually 1) • bool yScaling: Whether y scaling is to be used • double y_lowerBound: Lower y bound • double y_upperBound: Upper y bound
55
�x� and �y� in the above specification refer to the sampled attribute values and
classification labels of each line or training vector. (See Figure 6-4)
6.5.3 SVMTrainer
SVMTrainer is used expressly for the creation of classification models. It is the most
important of the three component classes and due to this it is also the most complex
to initialise. In order to initialise SVMTrainer it is necessary to first decide how you
wish to apply the Support Vector Machine. Once the SVMTrainer has been
initialised, and the model produced, all configuration data is recorded as part of the
model file which allows for easy re-creation of previous implementations of the
SVMTrainer.
Figure 6-9: SVMTrainer Class Diagram
SVMTrainer has two constructors. The first allows for easy re-implementation of a
previously created model and requires only pointers to the training data file name and
the previous model file name and uses general default values optimised for use with
generalised classification problems. The second constructor is used in the
construction of new implementations of the Support Vector Machine. In order to call
this constructor it is necessary that we pass in all the variables required and used by
structures within the LIBSVM base class (For implementation details see (Chih-Wei
Hsu, et al. [26]) ). These parameters are outlined below:
56
• svm_type : Set type of SVM (default 0) o 0 -- C-SVC o 1 -- nu-SVC o 2 -- one-class SVM o 3 -- epsilon-SVR o 4 -- nu-SVR
• kernel_type : Set type of kernel function (default 2) o 0 -- linear: u'*v o 1 -- polynomial: (gamma*u'*v + coef0)^degree o 2 -- radial basis function: exp(-gamma*|u-v|^2) o 3 -- sigmoid: tanh(gamma*u'*v + coef0) o 4 -- precomputed kernel (kernel values in training_set_file)
• kernalFunctDegree : Set degree in kernel function (default 3) • kernalFunctGamma : Set gamma in kernel function (default 1/k) • kernalFunctCoef0 : Set coef0 in kernel function (default 0) • KernalFuctNU : Set the parameter nu of nu-SVC, one-class SVM, and nu-
SVR (default 0.5) • cacheSizeMB : Set cache memory size in MB (default 100) • svmCostVal : Set the parameter C of C-SVC, epsilon-SVR, and nu-SVR
(default 1) • epsilonTerminationCriterion : Set tolerance of termination criterion (default
0.001) • epsilonSVR_EPS_Val : Set the epsilon in loss function of epsilon-SVR
(default 0.1) • shrinking: Whether to use the shrinking heuristics, 0 or 1 (default 1) • probabilityEst: Whether to train a SVC or SVR model for probability
estimates, 0 or 1 (default 0) • wi weight: Set the parameter C of class i to weight*C, for C-SVC (default 1) • crossValidation: Whether to use n-fold cross validation
o numFold: n-fold cross validation mode • C_SVCClass: Set the parameter C of class <C_SVCClass> to <weight>*C,
for C-SVC o weight: (default 1)
• input_file_name: Path & filename of file containing training data • model_file_name: Path & filename of model file
6.5.4 SVMPredictor
SVMPredictor is used to make classification predictions based on attribute line
inputs (vectors). It consists of two prediction methods. The most commonly used of
these is the predictLine() method which performs model categorisations based on
vector line inputs. The second method, testPredictFromFile(), is used expressly for
testing purposes. This method takes an input file name and output file name as
parameters. The input file contains test data in LIBSVM format. Predictions are
57
made based on the vector attributes of each line within the file, and compared to the
test categories (labels) assigned to each line. Predicted categories are written to the
output file and statistics outlining the how well the model performed are printed to
stdout for evaluation.
Figure 6-10: SVMPredictor Class Diagram
SVMPredictor contains two constructors. The first takes as parameter the model file
name, and the second takes this along with the option of using probability estimates4
with the model.
6.6 Overview of AffectiveInterface
As seen in Figure 6-5 the AffectiveInterface class acts as a façade for the entire
affective user interface system and allows high level control of the underlying
components. These high level methods provide a high level of abstraction allowing
developers with only a rudimentary knowledge of the Affective Interface to
successfully implement and use the system. Methods such as
writeNewTrainingFileData(), appendToTrainingFileData(), createModel() and
makePrediction() are all that are needed to operate and benefit from the Affective
Choice System. I will now outline the class attributes and methods of the
AffectiveInterface.
58
Figure 6-11: AffectiveInterface Class Diagram
The AffectiveInterface class provides two constructors. The first constructor requires
the following parameters:
• bool produceNewTrainingSet: Set to true if you wish to produce a new training set. Set to false if you wish to reuse and existing training set
(training files already exist)
• char * dataDirectory: Path to data directory where training files are stored (all training files must be present and in the correct format in this directory, these files include the <trainingDataFile>, the <trainingDataFile>.scaleConfig and the <trainingDataFile>.model)
• char * trainingDataFileName: Filename of training file in (LIBSVM format)
in directory (filenames for scaleConfig file and model file are derieved from this trainingDataFileName by appending �.scaleConfig� or �.model� to it)
The second also requires a SVMInterface pointer, as well as those outlined above.
4 For a classification model with probability information, this function gives probability estimates. The class with the highest probability is returned..
59
The AffectiveInterface class also provides the following methods:
createNewModel() � creates a new classification model from trainingDataFile (LIBSVM format)
makeLine(int, int [] ) - makes a line (vector) from live EEG Data in
(LIBSVM format) to be used in the creation of a training file or catrgory prediction. Parameters include: number of attributes you wish to include on a line and an integer array containing the blockPlusOffset values for each of the EEG attribute readings(see Appendix: Source Code)
makeLine(int, int [], int) � Same as previous method, but here the number of
EEG samples you wish to be averaged to make up each line vector value is also specified.
getEEGValue(int) � reads EEG value from block+offset in Brainmaster Dll
memory getEEGValue(int, int) - reads EEG value from block, offset in Brainmaster
Dll memory getEEGAverageValue(int, int) - reads average of numSamples of EEG values
from block+offset in Brainmaster Dll memory getEEGAverageValue(int, int, int) - reads average of numSamples of EEG
values from block, offset in Brainmaster Dll memory setValue(int, int) � used to set control parameter within the Brainmaster Dll
memory space as the location specified in the block & offset values. printValue(int, int) - prints EEG value from block, offset in Brainmaster Dll
memory to stdout writeNewTrainingFileData(int, double, int, int []) � writes training data to a
new training file. Parameters include: the number of lines (vectors) to write, the target value (Label) for the readings, number of attributes you wish to include on a line and an integer array containing the blockPlusOffset values for each of the EEG attribute readings(see Appendix: Source Code)
appendToTrainingFileData(int, double, int, int [])) � appends training data to
a new training file. Parameters include: the number of lines (vectors) to write, the target value (Label) for the readings, number of attributes you wish to include on a line and an integer array containing the blockPlusOffset values for each of the EEG attribute readings(see Appendix: Source Code)
60
writeNewTrainingFileData(int, double, int, int [], int) � as above but also specifies the number of EEG samples to average for each vector value reading.
appendToTrainingFileData(int, double, int, int [], int) � as above but also
specifies the number of EEG samples to average for each vector value reading.
makeConsistentPrediction(int, int, int []) � makes a clasification for a
consistent prediction for a line using SVM. Method blocks until a certain number of consistent target values are predicted, user is definitely and consistently in that state. Parameters include: number of consistent predictions in a row, number of attributes you wish to include on a line and an integer array containing the blockPlusOffset values for each of the EEG attribute readings(see Appendix: Source Code)
getCategoryPrediction(char*) � makes a clasification prediction for a line
using SVM. Attributes input data line (vector) containing attributes in LIBSVM format
getCharCategoryPrediction(char*) � makes a clasification prediction for a
line using SVM. Attributes input data line (vector) containing attributes in LIBSVM format, returns string representation of double target
giveFeedback(double, char*) � appends training data to training data set,
should be done predictor gets a value wrong. allows system to learn from its mistakes. Attributes include target, and vector line.
testModel(char*) - tests model by predicting values from 'testFileName' and
comparing them to the desired 'target' values. Parameter, name of test file(must be in <directory>)
6.7 Using the Affective Interface System
In order to use the Affective Interface System a number of decisions first need to be
made. What affective information do you wish from the user, emotion, concentration,
valence or arousal? What EEG signal provides the best measure of this affective
information, alpha, beta, left to right hemisphere coherence, FFT Lobeta, etc? (See
Chapter 5: The Brain, EEG & Emotion p34) Once you know this you must then
examine the optimum EEG electrode placement for the reading of these signals,
bearing in mind the underlying areas of the brain and the activities they are involved
in. These placements are specified using the 10-20 electrode placement system
discussed in section 5.3.2 p42.
61
All of these considerations are very important, and can mean the difference between
the interface performing correctly or simply not performing at all. From the
discussion in Chapter 5: The Brain, EEG & Emotion I presented the work of (L.A.
Schmidt and L.J. Trainor 2001 [50]) as proof that the pattern of frontal brain
asymmetry and overall power can be used to distinguish between emotions within
the valence/activation model using EEG data alone. From this I was able to define an
effective set of EEG electrode placements (Fp1, P3)(Fp2, P4)(Cz)(shown in Figure
6-12). Signals from this arrangement of electrodes when recorded and modelled
using the affective interface system, could be shown, in third party demonstrations,
to provide both accurate affective information in the activation domain,
(relaxed/active) but also in the emotional valence domain (happy/sad). This was the
breakthrough that confirmed the usefulness of EEG as a source of affective
information, and provided justification for the choice of EEG modelling as the
mechanism for the derivation of a subjects emotional state, made in this thesis.
Figure 6-12 : Effective EEG Sensor Placements
Once the type and source of EEG signals are decided upon it is possible to start
building a set of training data. This training data will then be used to construct a
model for the classification of future affective signals.
Using the writeNewTrainingFileData() method of the AffectiveInterface, a specified
number of lines of raw training data can be measured from a subjects EEG signals.
62
The first set of samples would be taken when the subject is experiencing �type one�
emotion (labelled 1 for example). The second, third, � set of samples would be
appended to the training file using the appendToTrainingFileData() method, and
taken when subject is experiencing �type two/three� emotion and given another label
(e.g. 0 or 2). Any number of distinct emotions can be recorded in this way. When this
process recording training data has been completed, this data is scaled, generally
from �1 to1 as shown below.
Figure 6-13: 2D training file data showing labels and sample values, this data is then shown scaled
from �1 to 1.
This leaves a scaled set of affective EEG data samples each labelled with the subjects
corresponding subjective emotion. When the SVM is then applied to the training data
a model of the user�s affective EEG signals is produced. This is achieved by calling
the createNewModel () method. This model can then be used to categorise affective
signals coming from the user into the various labels (0,1,�. ) therefore identifying
what emotion the user is exhibiting. This is achieved by simply calling one of the
getPrediction() methods of the AffectiveInterface class.
The sensitivity of the affective predictions can be selected by the developer simply
by demanding that a certain number of consistent predictions in a row occur before
their system acts upon the affective information. By increasing this buffer the
application developer can filter out one off affect changes, and artefacts caused by
muscle movement and other noise, ensuring that any prediction received has been
consistent over a certain period of time. This is achieved by calling the
getConsistentPrediction(method) which allows the number of hits in a row
(consistent predictions) required to be specified. This proved to provide a simple and
effective method of sensitivity control and noise filtration.
63
Chapter 7: Evaluation & Conclusion
7.1 Evaluation and Effectiveness of Demo Application (Affective MP3 player)
In order to test and demonstrate the effectiveness of the developed interface, a
sample demo application was developed. The demo application consisted of a simple
MP3 music player, which used 2 interfaces, a standard keyboard and of course the
Affective Choice Interface using the electrode placement outlined in Figure 6-12.
The demo application had all the usual functionality of a standard MP3 player,
volume, play, pause/resume, and skip song, however this Affective MP3 player
implementation also had the capability of building training data, creating affective
models and viewing affective signal classification predictions.
The affective interface also allowed for some extra features - for instance, a brain
controlled mechanism was included that allowed the user to control the pause/resume
function of the player using only conscious thought. This SVM model which
controlled this function was trained to recognise changes in the arousal level of the
user. Alpha brainwave frequencies evident when the user closed their eyes and
relaxed, clearing their mind were recorded and compared to normal eyes open,
focused, and alert states. Only when all 3 affective relaxation conditions occurred
would the player switch to paused mode. For instance, if the user closed his/her eyes
but didn�t clear his/her mind (was attempting arithmetic or was focused on a
problem) the affective interface would not give a consistent reading, and playback
would continue.
Another SVM model was trained to classify differences in valence level of the user.
Affects evident in the user�s beta brainwave patterns when they were happy (smiling)
as opposed to when they were sad (frowning) were measured and modelled with the
SVM. This model was applied to control the choice of song, so that whenever the
user chose to skip a song or when the end of a song was naturally reached, the
64
affective state of the user would be read, and an affective choice would be made on
their behalf whether to play a happy song or a sad song. This informed choice could
be made without explicit interaction by the user, and was based on the most current
affective state of the user. This affective knowledge allowed the system to make a
desirable choice on behalf of the user, by simply accessing the mood of the user.
7.1.1 Evaluation
In order to evaluate the effectiveness of the Affective Choice Interface System, seven
demonstrations of the affective MP3 application were made to third party
participants. These participants watched and prompted a trained user to control the
system, pausing/resuming playback on cue. During these demonstrations, which on
average lasted 9.4 minutes, the participant was presented with and asked to fill out a
questionnaire (See Appendix Table 0-3: Demonstration Questionnaire). While using
the feedback window of the demo application, the participant was asked to prompt
the user to smile and induce a happy emotion, or to frown and induce a sad emotion,
while the feedback window outputted either ones or zeros (1 trained to happy, and 0
trained to sad). The participants where then asked as to what percentage range they
felt the system predicted the user�s mood correctly? They responded on average in
the range of 75% to 94% accuracy for this valence test.
The MP3 player was then started and the participants were asked, on 10 occasions, to
prompt the user to pause and resume playback, estimating how many seconds the
user took on each attempt. Results from this test showed that an average of 4.7
seconds was needed to cause pause/resume to be triggered by a trained user. The best
values achieved in one demonstration averaged just 2.3 seconds for this test. In order
to ensure that the pause/reply mechanism was not simply triggering itself, many
users did not prompt the user to pause for very long periods, during which the
mechanism was not triggered(apart from one occasion). Following this second test
the participants were asked to state the percentage of time they felt the system was
being controlled by the user, to which all participants replied in the 81-100%
category, showing that the user was, indeed controlling the system using this arousal
model.
65
To the remaining questions all participants categorised their impression of the
responsiveness of the system as either good or very good and the usefulness of the
system generally was regarded as either useful or very useful. One participant
commented that they could see how such a system could, in the future, become
pivotal in modern technology applications, especially in home and office
applications.
7.1.2 Demonstration conclusion
The evaluation presented above was somewhat limited by both the number of
participants and the fact that the same user was controlling the system for all of the
demonstrations. Notwithstanding that it does, however, however demonstrate a
number of important facts. First of all it shows that the affective choice interface,
whose design and implementation is outlined in this thesis, is indeed capable of
measuring, modelling and predicting the affective state of a user. What�s more it can
be demonstrated to predict a user�s affective state very accurately - up to 94%
accuracy as reported by the participants. This simple evaluation also outlines the
systems ability to model both valence and arousal signals from Pre-frontal EEG
measurements. This ability demonstrates that the system may be able to model a
whole range of affective emotions, according to Wundt�s valence/activation model of
emotion, allowing it to be used as a valuable interface to a potentially wide range of
applications.
7.2 User tests
Two user tests were carried out on the Affective Interface System. The first user
right-handed male 24, and the second user a right-handed, female 23. Both users
were hooked up to the system using electrode placements outlined in Figure 6-12 :
Effective EEG Sensor Placements. Both users affective EEG signals were modelled for use
with the Affective MP3 demo application as outlined in 7.1. Both users were able to
control the pause/resume function in a similar fashion, with models created from
minimum training data (200 line vectors). The Happy/Sad valence model also could
be utilised by both users receiving accurate affective predictions from the affective
interface system.
66
This quite minimal user testing, is used to illustrate that the Affective Choice system
can be easily trained to different users affective EEG signals and to show that even
with only minimal training data, good affective classification can be achieved.
7.3 Conclusion
This thesis presents the successful outcomes relating to the careful research, design,
implementation and proven usability of this affective choice, user interface system.
The wholly achieved aim of this project was to design and build a reusable affective
user interface, which could can be easily reused as a component of future ubiquitous
applications, and could be shown to provide an additional �emotional channel� for
human computer interaction.
The research involved in the choice of EEG as the source of affective signals played
a pivotal role in the success of this project. The EEG data proved to provide a very
useful measure of a subject�s affective signals, so much so that it seems strange that
EEG has not been exploited more often in the development of affective computing
systems. The demonstrated ability of the system to recognise a user�s modelled
emotional state from EEG signals recorded from the pre-frontal hemispheres of the
brain also provides evidence in support of the importance of this area in the
management of emotion.
This thesis also puts forward an argument for the use of affective user interfaces as a
solution to the difficulties of sentient system control. As shown in Section 7.1, by
providing invisible affective user interfaces, such as the affective choice system to
these ubiquitous (sentient) applications, it will allow them to make desirable choices
on our behalf by simply basing its decisions and actions on our affective state, or
mood without the need for physical actions or explicit interaction.
7.4 Future Work
The prototype EEG based affective user interface outlined, has definite potential as a
main stream user interface. There are an infinite number of applications that could
benefit from the use of this system. The evaluation of the system illustrated its
67
ability to provide both conscious control to the user, analogous to a Brain Controlled
Interface, and unconscious control, where a device or application could make an
informed choice on behalf of the user based on their affective state. This �Affective
Choice� ability that can be gained by any application using the affective interface
system, is still, in my view, the most exciting application of the system. I feel that
future work should concentrate on working towards a more invisible affective
interface, for use with sentient and smart space type applications, allowing them to
serve up what we need and desire without needing to ask.
68
References
[1] T Turner A Ortony What�s basic about basic emotions?, Psychological Review, 97 (1990).
[2] Jeremy Ang, Rajdip Dhillon, Ashley Krupski, Elizabeth, Shriberg and Andreas Stolcke, PROSODY-BASED AUTOMATIC DETECTION OF ANNOYANCE AND FRUSTRATION IN HUMAN-COMPUTER DIALOG, California, Berkeley.
[3] R.J. Blair, J.S. Morris, C.D. Frith, D. I. Perrett and R.J Dolan, Dissociable neural responses to facial expressions of sadness and anger, Brain (1999).
[4] B Bloom, The Taxonomy of Educational Objectives, (1972).
[5] B. Boser, I. Guyon and V. Vapnik., A training algorithm for optimal margin classifiers., In Proceedings of the Fifth Annual Workshop on Computational Learning Theory. (1992).
[6] D. Bowers, R.M. Bauer, H.B. Coslett and K.M. Heilman, Processing of Face by Patients with Unilateral Hemisphere Lesions: Dissociations Between Judgements of Facial Effect and Facial Identity, Brain Cog (1985).
[7] W. Burleson, R. W. Picard, K. Perlin and J. Lippincott, A Platform for Affective Agent Research, Workshop on Empathetic Agents, International Conference on Autonomous Agents and Multiagent Systems, Columbia University, New York, NY, 2004.
[8] Winslow Burleson and Rosalind Picard, Affective Learning Companion, http://affect.media.mit.edu/projectpages/lc/ALC/ALC.htm, 2006.
[9] Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines, Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm (2001).
[10] Charlesworth and Kreutzer, Facial expression of infants and children, New York: Academic Press. (1973).
[11] Thomas F. Collura, 24490 Broadway AvenueOakwood Village, Ohio 44146 Corporate Offices (440) 232-6000 Fax (440) 232-7171, BrainMasterTechnologies,Inc.
[12] Thomas F. Collura, BrainMasterTechnologies,Inc. 24490 Broadway Avenue, Oakwood Village,
Ohio 44146, Corporate offices (440) 232-6000 Fax (440) 232-7171, http://www.brainmaster.com.
[13] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz and J.G. Taylor, Emotion recognition in human computer interaction, IEEE Signal Processing Magazine, 2001.
69
[14] A Damasio, Descartes� Error: Emotion, Reason, and the Human Brain, New York: Gosset/Putnam. (1994).
[15] A. R Damasio, T. J Grabowski, A. Bechara, H. Damasio, L. L. Ponto, J. Parvizi and R. D. Hichwa, Subcortical and cortical brain activity
during the feeling of self-generated emotions, (2000).
[16] C Darwin, The Expression of the Emotions in Man and Animals, New York: Philosophical Library. 3rd edn (1998) with Introduction. Afterword and Commentary by Paul Ekman: London: Harper Collins New York: Oxford University Press. (1872 -1998).
[17] R.J. Davidson, Anterior Cerebral Asymmetry And The Nature of Emotion, Brain and Cognition (1992).
[18] R.J. Davidson, G.E. Schwartz, C. Saron, J Bennett and D.J. Goleman, Fontal Versus Parietal EEG Asymmetry During Possitive and Negative Affect, Psychophysiology (1979).
[19] Paul Ekman, Basic Emotions - The Handbook of Cognition and Emotion, John Wiley & Sons, Ltd., Sussex, U.K, 1999.
[20] Frank Enos, A Survey of Models of Emotion Useful in the Context of Computation, Columbia University (2003).
[21] Beverly Fehr and James Russell, Concept of emotion viewed from a prototype perspective, Journal of Experimental Psychology (1984).
[22] Walter Sendlmeier Felix Burkhardt, Verification of Acoustical Correlates of Emotional Speech using Formant-Synthesis, Technical University of Berlin, Germany.
[23] Ekman and Friesen, FACS - Facial Action Coding System, 1978 - http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm.
[24] P. Ekman & W. V. Friesen, The facial action coding system, Consulting Psychologists Press (1978).
[25] J. Healey and R. W. Picard, Digital Processing of Affective Signals, Proceedings of ICASSP, Seattle, WA, 1997.
[26] Chih-Wei Hsu, Chih-Chung Chang and Chih-Jen Lin, A Practical Guide to Support Vector Classification, Department of Computer Science and Information Engineering National Taiwan University.
[27] C Izard, The Psychology of Emotions, New York: Plenum Press (1991).
[28] C Izard and S Buechler, Aspects of consciousness and personality in terms of differential emotions theory In R. Plutchik and H. Kellerman, New York: Academic Press. (1980).
[29] Papez J.W., A Proposed Mechanism of Emotion, Journal of Neuropsychiatry and clinical neurosciences. (1937).
[30] N.A. Jones and N.A Fox, Electroencephalogram Asymmetry During Emotionally Evocative Films and its Relation to Positive and Negative Affectivity, Brain and Cognition (1992).
70
[31] R. el Kaliouby, Mind-reading Machines: Automated Inference of Complex Mental States. , PhD Dissertation, University of Cambridge Computer Laboratory Technical Report 636 (July 2005).
[32] R. el Kaliouby, A. Teeters and R.W. Picard, An Exploratory Social-Emotional Prosthetic for Autism Spectrum Disorders, International Workshop on Wearable and Implantable Body Sensor Networks, MIT Media Lab, 2006.
[33] A. Kapoor, M. Selene and R.W. Picard, Towards a Learning Companion that Recognizes Affect, Proceedings from Emotional and Intelligent II: The Tangled Knot of Social Cognition, AAAI Fall Symposium (2001).
[34] T. D. Kemper, A Social Interactional Theory of Emotions, New York: Wiley. (1978).
[35] P. Kleinginna and A. Kleinginna, A categorized list of emotion definitions with suggestions for a consensual definition, Motivation and Emotion (1981), pp. 355.
[36] J. LeDoux, The Emotional Brain, New York: (1996).
[37] P.D MacLean, Psychosomatic Disease and the Visceral Brain. Recent Developments Bearing on the Papez Theory Of Emotion., Psychosomatic Medicine (1949).
[38] P.D MacLean, Some Psychiatric Implications of Physiological Studies on Frontotemporal Portion of Limbic System (Visceral Brain), Electroencephalography and clinical neurophysiology. (1952).
[39] P.D MacLean, A Triune Concept of the Brain and Behaviour, University of Toronto Press (1973).
[40] Michaela Esslen Pascual Marqui, HUMAN BRAIN IMAGING OF EMOTION AND LANGUAGE USING LOW RESOLUTION BRAIN ELECTROMAGNETIC TOMOGRAPHY, University of Zürich (2002).
[41] H Oster and P Ekman, Facial behavior in child development. , Minnesota Symposium on Child Psychology (1978), pp. 231-276.
[42] W.V. Friesen P Ekman, Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues, Prentice-Hall (1975).
[43] P.Shaver, Schwartz, Kirson and O'Connor, Emotion Knowledge:Further Exploration Of A
Prototype Approach, Journal Of Personality And Social Psychology, 52 (1987).
[44] M. L Phillips, A. W. Young, C. Senior, M. Brammer, C. Andrew, A. J. Calder, E. T. Bullmore, D. I. Perrett, D. Rowland, S. C. R Williams, J. A Gray and A. S. David, A specific neural substrate for perceiving facial expressions of disgust., Nature (1997).
[45] Rosalind W. Picard, Affective Computing, MIT Press, 1997.
[46] R Plutchik, The Psychology and Biology of Emotion, New York: HarperCollins. (1994).
[47] F. D. Ross, The aprosodias: functional-anatomical organization of the affective components of language in the right hemisphere, Archives of Neurology 38 (1981), pp. 561-569.
71
[48] Russel and Bullock, Multidimensional scaling of emotional facial expressions, Journal of Personality and Social Psychology (1985), pp. 1290-1298.
[49] H.A. Sackheim and R.C. Gur, Lateral Asymmetry in the Intensity of Emotional Expression, Neuropsychologia (1978).
[50] L.A. Schmidt and L.J. Trainor, Frontal Brain Electrical Activity (EEG) distinguishes valence and intensity of musical emotions., Cognition and Emotion (2001).
[51] M. Strauss, C. Reynolds, S. Hughes, K. Park, G. McDarby and R. W. Picard, The HandWave Bluetooth Skin Conductance Sensor, The 1st International Conference on Affective Computing and Intelligent Interaction, Beijing, China. , Oct 2005.
[52] Ph.D Thomas F. Collura, The Measurement, Interpretation, and Use of EEG Frequency Bands (December 7, 1997).
[53] Silvan S. Tomkins, Affect, Imagery, Consciousness, Springer, New York, 1962.
[54] R. Tourangeau and P.C.Ellsworth, The role of facial response in the experience of emotion, Journal of Personality and Social Psychology (1979).
(B Bloom 1972 [4])
72
Appendix: Tables
Table 0-1: Basic emotions according to different authors (Michaela Esslen Pascual Marqui 2002 [40])
73
Table 0-2: Average differences evident in speech components my emotion
(Walter Sendlmeier Felix Burkhardt [22]) (See section 3.2.2)
74
Table 0-3: Demonstration Questionnaire
75
Appendix: Figures
Action Description Facial muscle Example
1 Inner Brow Raiser Frontalis, pars medialis
2 Outer Brow Raiser Frontalis, pars lateralis
4 Brow Lowerer Corrugator supercilii, Depressor supercilii
5 Upper Lid Raiser Levator palpebrae superioris
6 Cheek Raiser Orbicularis oculi, pars orbitalis
7 Lid Tightener Orbicularis oculi, pars palpebralis
9 Nose Wrinkler Levator labii superioris alaquae nasi
10 Upper Lip Raiser Levator labii superioris
11 Nasolabial Zygomaticus minor
12 Lip Corner Puller Zygomaticus major
13 Cheek Puffer Levator anguli oris (a.k.a. Caninus)
14 Dimpler Buccinator
15 Lip Corner Depressor anguli oris (a.k.a. Triangularis)
16 Lower Lip Depressor labii inferioris
17 Chin Raiser Mentalis
18 Lip Puckerer Incisivii labii superioris and Incisivii
20 Lip stretcher Risorius w/ platysma
Figure 0-1: Ekman & Friesen's Action Units (Ekman and Friesen 1978 -
http://www.cs.cmu.edu/afs/cs/project/face/www/facs.htm [23])
76
Appendix: Source Code
/************************************************************************* Filename: dlldefs.h BrainMaster DLL extension library header file defining data offsets into dll memory space begun: 11/21/96 1/12/97: enlarged value blocks, added control blocks 1/20/97: revised Control Flag area to 0x600 from 0x500 7/11/97: changed frequencies to new 8-band system 7/14/97: added CONTROL_PARMS 7/14/97: added component LOW & HIGH limits 08/22/01 @gtw Add DLL_DATA_SIZE and remove extra defines. 08/28/01 tfc added defines for waveform & filtered 09/09/01 tfc added value blocks for stdev and nextthresh 11/14/01 gtw Updated IAW design review of 13 Nov 01. 1. Added xx_DAMP_FFT_START and remove extra channel area. 2. Added xx_FFT_SUMM_START for 32 FFT bins for 60 seconds of summary data. 11/27/01 gtw Updated IAW design review of 27 Nov 01 1. Added SIZEOF_SUMM_FFT, RT_FFT_SUMM_CTRL_START HEG_VALUE 2. Added AUTO_THRESH, HEG flags 3. Added <component>_PERCENT_TARGET values. 4. Updated commentary.
Copyright (C) 2001 BrainMaster Technologies, Inc. copyright (c) 1996,1997,1998,1999,2000 Thomas F. Collura, Ph.D., P.E. - all rights reserved
To read a value, use e.g. value = DllCFunc(LT_VAL_START+THETA_VAL, 0); NOTE: TOTAL IS ONLY 16K. DO NOT GO ABOVE 3FFF THIS IS DUE TO THE 16-BIT LIMITATION. FIRST 64K OF DLL SPACE IS DIVIDED UP AS FOLLOWS (addresses in hexadecimal): NOTE THAT ALL ADDRESSES ARE OF 16-bit WORDS 0000-03FF:***********VALUE BLOCKS FOR 8 CHANNELS 1/2 K TOTAL**************** VALUE BLOCKS CONTAIN VALUES, THRESHOLD, MAX, MIN, ETC. FOR EACH COMPONENT 0X0000 - BEGIN 1K BLOCK ALLOCATED TO CHANNELS 1-8 VALUE BLOCKS 0X0000 - 0x007F: 128 words: Channel 1 value block ( "Left" ) 0X0080 - 0x00FF: 128 words: Channel 2 value block ( "Right") 0X0100 - 0x017F: 128 words: Channel 3 value block 0X0180 - 0x01FF: 128 words: Channel 4 value block 0X0200 - 0X03FF: (512 words for Channels 5,6,7,8) 0400-05FF:********************LIVE FFT BLOCKS******************************* 64 bytes per spectrum x 8 channels = 512 bytes 0400-043F: CHANNEL 1 LIVE FFT BLOCK POSTED 4 PER SECOND 0440-047F: CHANNEL 2 LIVE FFT BLOCK 0480-04FF: Spare. 0500-053F CHANNEL 1 DAMPED FFT BLOCK POSTED 4 PER SECOND 0540-057F CBANNEL 2 DAMPED FFT BLOCK POSTED 4 PER SECOND 0600-09FF:****CONTROL FLAGS, PARAMETERS, AND SESSION AND CALIBRATION INFO***** 0600-06FF: 128 WORDS CONTROL FLAGS FOR REMOTE SETUP 0700-0800: 128 WORDS CONTROL PARAMETERS 0800-09FF: 256 WORDS SESSION & CALIBRATION INFORMATION DO NOT USE REGIONS DEFINED BELOW 3000-3FFF:****SUMMARY FFT AREA FOR COMPRESSED SPECTRAL ARRAY 4K TOTAL******** BEGIN SUMMARY FFT BLOCKS FOR 2 CHANNELS, POSTED 1 PER SECOND (64 bytes/second = 16 seconds per 1K, or 3.75K per minute) 7.5K: Channel 1 FFT block 1 minute 0.5K: TBD; reserved for control information 7.5K: Channel 2 FFT block 1 minute 0.5K: TBD; reserved for control information 1000-2FFF:*************LIVE EEG AREA 8K TOTAL*************************** TOTAL OF 30K FOR LIVE EEG CHANNELS 2 CHANNELS RAW & FILTERED
77
BEGIN LIVE EEG CHANNELS 2 seconds = 240 bytes per record (USE 256 bytes for round numbers) CHAN 1 RAW 256 samples CHAN 2 FILTERED 8 x 256 samples CHAN 1 RAW 256 samples CHAN 2 FILTERED 8 * 256 samples CONTROL INFO - SIZES, NUMTRACES, ETC - 128 bytes *************************************************************************/ //#define WRITE_COUNT 0x0000 //INCREMENTED EVERY TIME A PROCESS WRITES TO AREA #define DLL_DATA_SIZE 0x3fff #define DLL_EEG_BLOCK_SIZE 0x100 // VALUE BLOCK ASSIGNMENTS FOR CHANNELS 1 and 2 (LEFT AND RIGHT) #define LT_VAL_START 0x0000 #define RT_VAL_START 0x0080 #define CHAN_1_VAL_START 0x0000 #define CHAN_2_VAL_START 0x0080 #define LT_LIVE_FFT_START 0x0400 #define RT_LIVE_FFT_START 0x0440 #define CHAN_1_LIVE_FFT_START 0x0400 #define CHAN_2_LIVE_FFT_START 0x0440 #define LT_DAMP_FFT_START 0x0500 #define RT_DAMP_FFT_START 0x0540 #define CHAN_1_DAMP_FFT_START 0x0500 #define CHAN_2_DAMP_FFT_START 0x0540 #define CONTROL_FLAGS_START 0x0600 #define CONTROL_PARMS_START 0x0700 #define SESSION_INFO_START 0x0800 #define RT_SUMM_FFT_START 0x4000 #define LT_SUMM_FFT_START 0x6000 #define CHAN_1_SUMM_FFT_START 0x4000 #define CHAN_2_SUMM_FFT_START 0x6000 #define LT_LIVE_EEG_START 0x1000 #define LT_LIVE_DELTA_START 0x1100 #define LT_LIVE_THETA_START 0x1200 #define LT_LIVE_ALPHA_START 0x1300 #define LT_LIVE_LOBETA_START 0x1400 #define LT_LIVE_BETA_START 0x1500 #define LT_LIVE_HIBETA_START 0x1600 #define LT_LIVE_GAMMA_START 0x1700 #define LT_LIVE_USER_START 0x1800 #define RT_LIVE_EEG_START 0x1900 #define RT_LIVE_DELTA_START 0x2000 #define RT_LIVE_THETA_START 0x2100 #define RT_LIVE_ALPHA_START 0x2200 #define RT_LIVE_LOBETA_START 0x2300 #define RT_LIVE_BETA_START 0x2400 #define RT_LIVE_HIBETA_START 0x2500 #define RT_LIVE_GAMMA_START 0x2600 #define RT_LIVE_USER_START 0x2700 #define LIVE_EEG_CONTROL 0x2800 #define LIVE_EEG_POSITION 0x2801 #define LIVE_EEG_SIZE 0x2802 #define LIVE_EEG_SAMPLESREAD 0x2803 #define LIVE_EEG_SAMPLERATE 0x2804 #define LIVE_EEG_NUMTERMS 0x2805 // The follow area is for the summary FFT // information used by the CSA display. // #define SIZEOF_SUMM_FFT 0x0020 #define LT_FFT_SUMM_START 0x3000 #define CHAN_1_FFT_SUMM_START 0x3000 #define LT_FFT_SUMM_CTRL_START 0x3780 #define FFT_SUMM_PTR 0x3780 #define RT_FFT_SUMM_START 0x3800 #define CHAN_2_FFT_SUMM_START 0x3800
78
#define RT_FFT_SUMM_CTRL_START 0x3F80 // THE FOLLOWING OFFSETS MAP INTO VALUE SPACE // THESE ARE VALUES THAT MASTER WRITES #define DELTA_VAL 0x0001 #define THETA_VAL 0x0002 #define ALPHA_VAL 0x0003 #define LOBETA_VAL 0x0004 #define BETA_VAL 0x0005 #define HIBETA_VAL 0x0006 #define GAMMA_VAL 0x0007 #define USER_VAL 0x0008 #define RESRV_VAL 0x0009 #define BASE_MODALFREQ 0x000A #define DELTA_MODALFREQ 0x000B #define THETA_MODALFREQ 0x000C #define ALPHA_MODALFREQ 0x000D #define LOBETA_MODALFREQ 0x000E #define DELTA_DAMPED_VAL 0x000F #define BASE_THRESH 0x0010 #define DELTA_THRESH 0x0011 #define THETA_THRESH 0x0012 #define ALPHA_THRESH 0x0013 #define LOBETA_THRESH 0x0014 #define BETA_THRESH 0x0015 #define HIBETA_THRESH 0x0016 #define GAMMA_THRESH 0x0017 #define USER_THRESH 0x0018 #define RESRV_THRESH 0x0019 #define DELTA_COHER 0x001A #define THETA_COHER 0x001B #define ALPHA_COHER 0x001C #define LOBETA_COHER 0x001D #define THETA_DAMPED_VAL 0x001E #define ALPHA_DAMPED_VAL 0x001F #define BASE_NEXTTHRESH 0x0020 #define DELTA_NEXTTHRESH 0x0021 #define THETA_NEXTTHRESH 0x0022 #define ALPHA_NEXTTHRESH 0x0023 #define LOBETA_NEXTTHRESH 0x0024 #define BETA_NEXTTHRESH 0x0025 #define HIBETA_NEXTTHRESH 0x0026 #define GAMMA_NEXTTHRESH 0x0027 #define USER_NEXTTHRESH 0x0028 #define RESRV_NEXTTHRESH 0x0029 #define BETA_COHER 0x002A #define HIBETA_COHER 0x002B #define GAMMA_COHER 0x002C #define USER_COHER 0x002D #define LOBETA_DAMPED_VAL 0x002E #define BETA_DAMPED_VAL 0x002F #define BASE_STDEV 0x0030 #define DELTA_STDEV 0x0031 #define THETA_STDEV 0x0032 #define ALPHA_STDEV 0x0033 #define LOBETA_STDEV 0x0034 #define BETA_STDEV 0x0035 #define HIBETA_STDEV 0x0036 #define GAMMA_STDEV 0x0037 #define USER_STDEV 0x0038 #define RESRV_STDEV 0x0039 #define DELTA_PHASE 0x003A #define THETA_PHASE 0x003B #define ALPHA_PHASE 0x003C
79
#define LOBETA_PHASE 0x003D #define HIBETA_DAMPED_VAL 0x003E #define GAMMA_DAMPED_VAL 0x003F #define BASE_MODE 0x0040 #define DELTA_MODE 0x0041 #define THETA_MODE 0x0042 #define ALPHA_MODE 0x0043 #define LOBETA_MODE 0x0044 #define BETA_MODE 0x0045 #define HIBETA_MODE 0x0046 #define GAMMA_MODE 0x0047 #define USER_MODE 0x0048 #define RESRV_MODE 0x0049 #define BETA_PHASE 0x004A #define HIBETA_PHASE 0x004B #define GAMMA_PHASE 0x004C #define USER_PHASE 0x004D #define USER_DAMPED_VAL 0x004E #define DLL_OFFSET_4F 0x004F #define BASE_HITS 0x0050 #define DELTA_HITS 0x0051 #define THETA_HITS 0x0052 #define ALPHA_HITS 0x0053 #define LOBETA_HITS 0x0054 #define BETA_HITS 0x0055 #define HIBETA_HITS 0x0056 #define GAMMA_HITS 0x0057 #define USER_HITS 0x0058 #define RESRV_HITS 0x0059 #define BETA_MODALFREQ 0x005A #define HIBETA_MODALFREQ 0x005B #define GAMMA_MODALFREQ 0x005C #define USER_MODALFREQ 0x005D #define RESRV_MODALFREQ 0x005E #define DLL_OFFSET_5F 0x005F #define BASE_MEAN 0x0060 #define DELTA_MEAN 0x0061 #define THETA_MEAN 0x0062 #define ALPHA_MEAN 0x0063 #define LOBETA_MEAN 0x0064 #define BETA_MEAN 0x0065 #define HIBETA_MEAN 0x0066 #define GAMMA_MEAN 0x0067 #define USER_MEAN 0x0068 #define RESRV_MEAN 0x0069 #define BASE_PEAKFREQ 0x006A #define DELTA_PEAKFREQ 0x006B #define THETA_PEAKFREQ 0x006C #define ALPHA_PEAKFREQ 0x006D #define LOBETA_PEAKFREQ 0x006E #define DLL_OFFSET_6F 0x006F #define BASE_PERCENTTIMEOVERTHRESH 0x0070 #define DELTA_PERCENTTIMEOVERTHRESH 0x0071 #define THETA_PERCENTTIMEOVERTHRESH 0x0072 #define ALPHA_PERCENTTIMEOVERTHRESH 0x0073 #define LOBETA_PERCENTTIMEOVERTHRESH 0x0074 #define BETA_PERCENTTIMEOVERTHRESH 0x0075 #define HIBETA_PERCENTTIMEOVERTHRESH 0x0076 #define GAMMA_PERCENTTIMEOVERTHRESH 0x0077 #define USER_PERCENTTIMEOVERTHRESH 0x0078 #define RESRV_PERCENTTIMEOVERTHRESH 0x0079 #define BETA_PEAKFREQ 0x007A #define HIBETA_PEAKFREQ 0x007B #define GAMMA_PEAKFREQ 0x007C
80
#define USER_PEAKFREQ 0x007D #define RESRV_PEAKFREQ 0x007E #define HEG_VALUE 0x007F // THE FOLLOWING OFFSETS MAP INTO FLAG SPACE // THESE ARE SWITCHES THAT MASTER READS // WE CAN HAVE UP TO 128 SUCH FLAGS // The gaps between values are not important, // but care should be used in changing the values. #define DLL_BUSY 0x0000 #define WRITE_COUNT 0x0001 #define READ_COUNT 0x0002 #define DELTA 0x0011 #define THETA 0x0012 #define ALPHA 0x0013 #define LOBETA 0x0014 #define BETA 0x0015 #define HIBETA 0x0016 #define GAMMA 0x0017 #define USER 0x0018 #define RESRV 0x0019 #define SOUND 0x0020 #define SAVETODISK 0x0021 #define WAVEFORM 0x0031 #define PHASE 0x0032 #define FFT 0x0033 #define MIRROR 0x0034 #define THERM 0x0035 #define ONEDTREND 0x0036 #define TWODTREND 0x0037 #define THREEDTREND 0x0038 #define CSA 0x0039 #define STERMAN 0x0040 #define OTHMER 0x0041 #define PACMAN 0x0042 #define SMILEY 0x0043 #define SIMILARITY 0x0044 #define PHASE_SIMILRTY 0x0045 #define MIDI_VOICE 0x0046 #define MIDI_MODE 0x0047 #define MIDI_MODULATION 0x0048 #define COHERENCE_THRESHOLD 0x0049 #define AUTO_THRESH 0x004A #define HEG 0x004B #define EQUALIZER 0x004C #define MASTER_WRITE_COUNT 0x004D #define MASTER_RUNNING 0x004E #define MASTER_PAUSE_FLAG 0x004F #define MASTER_ARTIFACT_FLAG 0x0050 #define MASTER_INHIBIT_FLAG 0x0051 #define MASTER_INHIBIT1_FLAG 0x0052 #define MASTER_ENHANCE1_FLAG 0x0053 #define MASTER_NUM1_ENHANCES 0x0054 #define MASTER_INHIBIT2_FLAG 0x0055 #define MASTER_ENHANCE2_FLAG 0x0056 #define MASTER_NUM2_ENHANCES 0x0057 // THE FOLLOWING OFFSETS MAP INTO CONTROL PARAMETER SPACE // THESE ARE VALUES THAT MASTER USES // WE CAN HAVE 128 SUCH PARAMETERS #define DELTA_LOW 0x0000 #define DELTA_HIGH 0x0001 #define THETA_LOW 0x0002 #define THETA_HIGH 0x0003 #define ALPHA_LOW 0x0004 #define ALPHA_HIGH 0x0005 #define LOBETA_LOW 0x0006 #define LOBETA_HIGH 0x0007 #define BETA_LOW 0x0008 #define BETA_HIGH 0x0009 #define HIBETA_LOW 0x0010
81
#define HIBETA_HIGH 0x0011 #define GAMMA_LOW 0x0012 #define GAMMA_HIGH 0x0013 #define USER_LOW 0x0014 #define USER_HIGH 0x0015 #define RESRV_LOW 0x0016 #define RESRV_HIGH 0x0017 #define FILTER_ORDER 0x0018 #define NCHANS 0x0019 #define CHAN1_DELTA_PERCENT_TARGET 0x0020 #define CHAN1_THETA_PERCENT_TARGET 0x0021 #define CHAN1_ALPHA_PERCENT_TARGET 0x0022 #define CHAN1_LOBETA_PERCENT_TARGET 0x0023 #define CHAN1_BETA_PERCENT_TARGET 0x0024 #define CHAN1_HIBETA_PERCENT_TARGET 0x0025 #define CHAN1_GAMMA_PERCENT_TARGET 0x0026 #define CHAN1_USER_PERCENT_TARGET 0x0027 #define CHAN2_DELTA_PERCENT_TARGET 0x0028 #define CHAN2_THETA_PERCENT_TARGET 0x0029 #define CHAN2_ALPHA_PERCENT_TARGET 0x002A #define CHAN2_LOBETA_PERCENT_TARGET 0x002B #define CHAN2_BETA_PERCENT_TARGET 0x002C #define CHAN2_HIBETA_PERCENT_TARGET 0x002D #define CHAN2_GAMMA_PERCENT_TARGET 0x002E #define CHAN2_USER_PERCENT_TARGET 0x002F
