FusionofMotif-andSpectrum-RelatedFeaturesforImproved EEG...

Research ArticleFusion of Motif- and Spectrum-Related Features for ImprovedEEG-Based Emotion Recognition

Abhishek Tiwari and Tiago H Falk

Institut National de la Research Scientifique Universite du Quebec Montreal Quebec Canada

Correspondence should be addressed to Abhishek Tiwari abhishektiwariemtinrsca

Received 30 October 2018 Revised 11 December 2018 Accepted 12 December 2018 Published 17 January 2019

Academic Editor Pietro Arico

Copyright copy 2019 Abhishek Tiwari and Tiago H Falk is is an open access article distributed under the Creative CommonsAttribution License which permits unrestricted use distribution and reproduction in anymedium provided the original work isproperly cited

Emotion recognition is a burgeoning field allowing for more natural human-machine interactions and interfaces Electroen-cephalography (EEG) has shown to be a useful modality with which user emotional states can be measured and monitoredparticularly primitives such as valence and arousal In this paper we propose the use of ordinal pattern analysis also called motifsfor improved EEG-based emotion recognition Motifs capture recurring structures in time series and are inherently robust tonoise thus are well suited for the task at hand Several connectivity asymmetry and graph-theoretic features are proposed andextracted from the motifs to be used for affective state recognition Experiments with a widely used public database are conductedand results show the proposed features outperforming benchmark spectrum-based features as well as other more recentnonmotif-based graph-theoretic features and amplitude modulation-based connectivityasymmetry measures Feature and score-level fusion suggest complementarity between the proposed and benchmark spectrum-based measures When combined thefused models can provide up to 9 improvement relative to benchmark features alone and up to 16 to nonmotif-based graph-theoretic features

1 Introduction

Human-machine interaction can become more natural oncemachines become aware of their surroundings and theirusers [1 2] ese so-called context-aware or affective in-terfaces can open up new dimensions of device functionalitythus more accurately addressing human needs while keepingthe interfaces as natural as possible [3] For example af-fective computing can enable applications in which themachine can learn user preferences based on their reactionsto different settings or even become a more effective tutor byassessing the studentrsquos emotionalstress states [3] Auto-mated recommender and tagging systems in turn can makeuse of affect information to better understand user prefer-ences thus improving system usability [4] Measuring af-fective state and engagement levels can also be used by amachine to infer the userrsquos perceived quality of experience[5ndash9] thus providing the machine with an objective crite-rion for online optimization

Human emotions are usually conceived as physiologicaland physical responses and are part of natural human-human communications Emotions are able to influenceour intelligence shape our thoughts and govern our in-terpersonal relationships [10ndash13] Emotion is usuallyexpressed in a multimodal way either verbally throughemotional vocabulary or by expressing nonverbal cues suchas intonation of voice facial expressions and gestures Assuch audio-visual cues have been widely used for affectivestate monitoring [14] Alternately emotions have also beenknown to effect neurophysiological signals thus biosignalmonitoring has been extensively explored Representativephysiological signal modalities have included galvanic skinresponse (GSR) skin temperature and breathing and car-diac activity (via electrocardiography (ECG) and photo-plethysmography (PPG)) [15ndash18]

More recently brain-computer interfaces (BCIs) haveemerged as another tool to accurately monitor implicit userinformation such as mood stress level andor emotional

HindawiComputational Intelligence and NeuroscienceVolume 2019 Article ID 3076324 14 pageshttpsdoiorg10115520193076324

states [9 19 20] Within BCI-based affective computingmethods electroencephalography (EEG) has remained themost popular modality due to its noninvasiveness hightemporal resolution and portability [21] though alternativemodalities such as near-infrared spectroscopy (NIRS) areslowly emerging [5 22 23] Typically with EEG-basedsystems spectral power features have been widely used(eg [18 24ndash26]) including frontal interhemisphericasymmetry features [27ndash32] EEG signals however are verysensitive to artefacts such as eye blinks and musclemovement [33] To overcome such issues artefact removalalgorithms can be used Alternately new noise-robust fea-tures can be developed andor multimodal fusion strategiescan be explored [34]

In this paper focus is placed on the latter and motif-based features are proposed and tested alone or alongsidealternate complementary features Motif-based analysis hasshown to be useful in the past to recognize sleep states [35]as well as the effects of anesthesia [36] to detect seizures[37 38] and to measure alertness [39] Motif-based methodsare inherently robust to noise as they deal with the shape ofthe time series and are unaffected by the magnitude[38 40 41] To the best of our knowledge they have yet to beexplored for affective state monitoring thus this paper fillsthis gap In particular we compare the proposed featureswith spectral power and spectral asymmetry benchmarkfeatures Notwithstanding one main limitation of motiffeatures concerns the loss of both amplitude and rate-of-change information when time series are converted intomotif series [40 42] As such we also explore three differentfusion strategies to combine information from the proposedmotif features and classical benchmark features Experi-mental tests on a publicly available database [18] are per-formed which show the advantages of the proposed featuresover benchmark ones as well as the benefits of fusion foraffective state monitoring

e remainder of this paper is organized as followsSection 2 describes the materials and methods used in-cluding the database considered proposed and benchmarkfeatures fusion methods used and performance metricsused Section 3 then presents and discusses the results ob-tained and conclusions are drawn in Section 4

2 Materials and Methods

Here we describe the database used benchmark featuresproposed motif features as well as the feature selectionschemes employed classifiers and fusion schemes explored

21 DEAP Database is study relies on the publiclyavailable widely used DEAP (Dataset for Emotion Analysisusing EEG and physiological signals) database As detailed in[18] thirty-two healthy participants (50 females averageage 269 years) were recruited and consented to participatein the study irty-two channel EEG data were recordedusing a Biosemi ActiveTwo system (Amsterdam Nether-lands) at a sampling rate of 512Hz Electrodes were placedon the scalp according to the International 10ndash20 system

Participants were presented with 40 one-minute longmusic videos with varying emotional content ese videoclips were selected based on a previous analysis of severalhundred videos as they were shown to elicit the strongestreactions across the four quadrants in the valence-arousalspace (ie low valence low arousal low valence higharousal high valence low arousal and high valence higharousal) e valence-arousal space is a two dimensionalscale used to characterize emotions [43] Valence refers tothe (un)pleasantness of an event whereas arousal refers tothe intensity of the event ranging from very calming tohighly exciting Using this space various emotions can bemapped as shown in Figure 1 Prior to each video there wasa baseline period of five seconds where the participants wereasked to fixate at a cross in the middle of the screen Fol-lowing the presentation of each video participants wereasked to rate the music videos on discrete 9-point scales forvalence and arousal using the self-assessment manikins(SAM) [44] While other dimensional ratings such asdominance and liking were also collected these have notbeen explored herein

e EEG data are available for public download in rawformat or in preprocessed format which includes commonreferencing down-sampling to 128Hz bandpass filteringbetween 4 and 45Hz and eye blink artefact removal viaindependent component analysis Moreover only the lastthree seconds of the five-second baseline are available Sincethis is a standard pipeline for EEG processing the analysisreported herein is done on the preprocessed data Data persubject were epoched into forty 60 s long trials with a 3 s longprestimulus baseline e prestimulus baseline was thensubtracted from the preprocessed datae interested readercan refer to [18] for more details on the DEAP database andits data collection process

22 Benchmark Features As mentioned previously spectralpower features in different EEG bands have been widely usedfor affective state monitoring including for the DEAP data-base [18 45] Moreover an interhemispheric asymmetry inspectral power has also been been reported in the affectivestate literature [27 28 30ndash32] particularly in frontal brainregions [29 31] Typically EEG signals are band decomposedinto theta (4lt θlt 8 Hz) alpha (8lt αlt 13 Hz) beta(13lt βlt 30 Hz) and gamma (30lt clt 45 Hz) bands Here48 asymmetry index (AI) features (12 interhemisphereicelectrode pairs times 4 bands) were computed for the followingelectrodes pairs Fp2-Fp1 F3-F4 F7-F8 FC1-FC2 FC5-FC6C3-C4 T7-T8 Cp1-Cp2 Cp5-Cp6 P3-P4 P7-P8 andO1-O2

Moreover EEG band ratios have also been explored inthe past for tasks such as human mental state monitoringfatigue attention control and negative emotional responsemonitoring [46ndash48] thus are also included here as bench-mark features e ratios computed include cβ βθ

αθ (α + β)c and (c + β)θ e ratios are computed in-dividually over each electrode Lastly the Shannon entropy[49] has been used as a feature to measure the complexity ofthe EEG time series Shannon entropy can be calculated asfollows

2 Computational Intelligence and Neuroscience

SE minussumj

Pj middot log Pj( ) (1)

where Pj is the power in sub-band j

23 Motif-Based Features A motif is a pattern or structurecharacterized by the number of nodes or degree (representedby n) and the connection between them and the number ofpoints used between these nodes (called lag represented byλ) Each motif can be represented as an alphabet or anumber e robustness of motif features comes from thefact that they only consider the underlying shape of the timeseries and not the amplitude Using this denition any timeseries X(i) can be converted into a motif series Xm(i) usingthese given rules (eg for degree n 3)

Xm(i)

1 if X(i)ltX(i + λ)X(i + λ)ltX(i + 2λ)X(i)ltX(i + 2λ)2 if X(i)ltX(i + λ)X(i + λ)gtX(i + 2λ)X(i)ltX(i + 2λ)3 if X(i)gtX(i + λ)X(i + λ)ltX(i + 2λ)X(i)ltX(i + 2λ)4 if X(i)ltX(i + λ)X(i + λ)gtX(i + 2λ)X(i)gtX(i + 2λ)5 if X(i)gtX(i + λ)X(i + λ)gtX(i + 2λ)X(i)gtX(i + 2λ)6 if X(i)gtX(i + λ)X(i + λ)ltX(i + 2λ)X(i)gtX(i + 2λ)

(2)

Figure 2 shows the dierent motifs possible for degreen 3 appearing in a particular time series Once the motifseries has been derived dierent features can be extractedbased on the statistics of recurring patterns within the motifseries e features proposed herein are detailed in thesubsections below and only consider motifs of degree n 3

and lag value λ 1ese parameters have been suggested inthe past for related tasks [39 50]

231 Permutation Entropy Permutation entropy (PE) [41]is a commonly derived motif-based metric and is calculatedas

PE minussumn

j

p(j) middot log(p(j)) (3)

where p(j) is the relative frequency of the motif patternrepresented by j

232 Ordinal Distance Dissimilarity Ordinal distance-based dissimilarity [38] is a metric with close parallel tothe benchmark asymmetry index and measures the dis-similarity between two motif series for dierent electrodepairs using

Dm(X Y)

n

nminus 1

radic

sumn

i

px(i)minuspy(i)( )2

radicradic

(4)

where px(i) and py(i) are the relative frequencies of themotif pattern represented by i in electrodes X and Y re-spectively and n is the degree of the motif In order tocompare against the benchmark asymmetry index ordinaldissimilarity is calculated for the same electrode pairs re-ported in Section 22

Valence

Aro

usal

High (pleasant)Low (unpleasant)

Hig

h (e

xcite

d)Lo

w (c

alm

)

Neutral

High arousal-high valenceHigh arousal-low valence

Low arousal-low valence Low arousal-high valence

Happy

Contented

Serene

Relaxed

Excited

AlertTense

Nervous

Upset

Bored

Depressed

Sad

Figure 1 Valence-arousal plot with representative emotions

Computational Intelligence and Neuroscience 3

233 Motif Synchronization Functional connectivity givesinsight into the dynamic neural interaction of the dierentregions of the brain Recently motif synchronization hasbeen proposed as a functional connectivity analysis tool [50]and measures the simultaneous appearance of motifs in twotime series For two motif series Xm and Ym c(XmYm) isdened as the highest number of times in which the samemotif can appear in Ym shortly after it appeared in Xm fordierent delay times ie

c XmYm( ) cXY max sumlm

i1Jτ0i sum

lm

i1Jτ1i sum

lm

i1Jτni

(5)

with

Jτi (i) 1 if XMi

YMi+τ

0 else (6)

e time delay τ ranges from τ0 0 to τn where τn is themaximum value to be considered and lm is the size of thetime varying window within the time series Similarly theopposite measure cYX can be obtained by changing only theorder of the time series to YMi

XMi+τ Finally the degree of

synchronization QXY and the synchronization direction qXYare given by

Qxy max cXY cYX( )

lm

qXY 0 if cXY cYX

sign cXY minus cYX( ) else

(7)

e degree of synchronization QXY is scaled between 0and 1 with 0 representing no interaction and 1 suggestingvery high interactions Feature qXY in turn gives the di-rection of information ow with 0 indicating no preferreddirection 1 indicating direction from X to Y and minus1 in-dicating direction from Y to X For our calculation τn hasbeen chosen as 5 and the window size lm is chosen as 256

234 Graph Features e dierent functional connectionsobtained by motif synchronization analysis can be furtherextended by means of graph-theoric analysis where each

electrode on the scalp represents a node on the brain net-work Weighted graphs have weights that represent the levelof interaction between the two nodes Edges with smallerweights are believed to represent noisyspurious connections[51] thus a thresholding is done to obtain an unweightedgraph Previously graph-theoretic features have been ex-plored for aect recognition based on EEG spectral co-herence measures [52] Graph-theoretic analysis based onmotifs however has yet to be explored thus both weightedand unweighted graphs (thresholded to the average value ofthe graph weighted) are tested herein An advantage of motifsynchronization over more popular connectivity approachesis the ability it provides to measure direction of informationow for the dierent nodes in the brain network From theweighted and unweighted graphs several features areextracted namely

(i) Degree of connectivity (k) the degree of connec-tivity is dened as ki where i is a given node For theunweighted network it is calculated as

ki sumjϵNe

aij (8)

where Ne represents the nodes in the network andaij ine j represents the value of the unweightedadjacency matrix For the weighted network theformula is

kwi sumjϵNe

wij (9)

where instead of aij the weights wij assigned toeach edge are used e average degree of con-nectivity for the whole network is used as a featurein our analysis

(ii) Clustering coecient (C) the mean clusteringcoecient for an unweighted network is given by

C 1NesumNe

i

eiki ki minus 1( )

(10)

where ei is the number of existing edges betweenthe neighbors of i and ki is the degree of con-nectivity for the unweighted network For aweighted network the clustering coecient valueis given by

X (i) X (i + λ)

X (i + 2λ)

Time seriesX (t)

Figure 2 All motifs of degree n 3 appearing in a time series


Cw

1

Ne

1113944

Ne

i

twi

kwi kw

i minus 1( 1113857 (11)

where ti is calculated as

ti 12

1113944jhϵNe

wijwjhwhi1113872 111387313

(12)

and represents the geometric mean of the trianglesconstructed from the edges around a particularnode i and kw

i represents the weighted degree ofconnectivity

(iii) Transitivity (Tr) transitivity is defined as the ratioof ldquotriangles to tripletsrdquo in a network and is definedas

Tr 3λ

k(kminus 1) (13)

where λ represents the number of triangles innetwork while k is the average degree of con-nectivity (weighted or unweighted) of a networkTransitivity is a global measure of the cluster-ing coefficient and is equal to it when the degreeof connectivity of all nodes is equal to oneanother

(iv) Characteristic path length (L) for an unweightednetwork L is given by

L 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdij (14)

with dij being the minimum amount of edges re-quired to connect nodes i and j and is replaced bythe shortest weighted path length dw

ij for theweighted characteristic path length Lw

(v) Global efficiency (G) this is calculated using theinverse of the shortest weighted or unweighted pathfor the network ie

G 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdminus1ij (15)

where dij is replaced by the shortest weightedpath length dw

ij for weighted global efficiencymeasure

(vi) Small-world features the work in [53] has shownthat human brain networks exhibit small-worldcharacteristics A small-world network is charac-terized by a high clustering coefficient and a smallaverage path length from one node to another[54] Here three small-world features are com-puted namely (i) the small-world characteristicslength

Ls L

Lrand (16)

(ii) the small-world clustering coefficient

Cs C

Crand (17)

and (iii) the small-worldness of a network [55]

S Cs

Ls

(18)

where Crand and Lrand are the corresponding clus-tering coefficient and characteristic path lengthvalues for a random network respectively

(vii) Direction of flow (DoF) As motif synchronizationalso provides the direction of information flow inthe brain network graph a simple feature is ex-plored here to represent the overall response of thebrain network as either receiving or transmittinginformation on average DoF is defined as

DoF 1113944ij

qij (19)

where qij is defined as the direction of informationflow with 1 representing information flowing from ito j minus1 representing information flow in the op-posite direction and 0 being no preferred in-formation flow direction

Table 1 provides a summary of the number of featuresextracted for each feature group and subgroup

24 Feature Selection Previous work has shown that motiffeatures convey complementary information to other am-plitude- and rate-of-change-based features [40 42] As suchwe explore the effects of combining the proposed motif-based features with the benchmark ones Given the smalldataset size however it is important to avoid issues withcurse of dimensionality and overfitting thus feature se-lection is required Here three feature selection strategieshave been explored

(1) ANOVA-based feature ranking and selection thisselection method is based on calculating the sig-nificance of the input features with respect to theoutput values and returning the ranked featuresaccording to their obtained p values

(2) Minimum redundancy maximum relevance(mRMR) feature selection the mRMR is a mutualinformation-based algorithm that optimizes twocriteria simultaneously the maximum-relevancecriterion (ie maximizes the average mutual in-formation between each feature and the targetvector) and the minimum-redundancy criterion(ie minimizes the average mutual informationbetween two chosen features) e algorithm findsnear-optimal features using forward selection withthe chosen features maximizing the combinedmax-min criteria Previous work showed mRMRpaired with a support vector machine (SVM)classifier [56] achieved the best performance inEEG-based emotion recognition tasks [57]


(3) Recursive feature elimination (RFE) given an ex-ternal estimator that assigns weights to features theleast important features are pruned from the currentset of features e procedure is recursively repeatedon the pruned set until the desired number of fea-tures to select is reached is technique considersthe interaction of features with the learning algo-rithm to give the optimal subset of features Sincerecursive training and feature elimination is re-quired this method takes a significant amount ofruntime

For the experiments herein 90 of the data is set aside forfeature selection and classifier training and the remaining 10is left aside for testinge split was performed with a randomseed of 0 using the scikit-learn function in Python e bestfeature selection algorithm and its corresponding optimalnumber of features are then selected by grid search Classifiertraining and different fusion schemes are described next

25 Classification SVMs have been widely used for affectivestate recognition [57] and are explored herein as well Giventheir widespread use a detailed description is beyond thescope of this paper and the interested reader is referred to[58] and references therein for more details Here SVMclassifiers are trained on two different binary classificationproblems namely discriminating between low and highvalence states and low and high arousal states For our studya radial basis function (RBF) kernel was used and imple-mented with the scikit-learn library in Python [59] As weare interested in exploring the benefits of the proposed motiffeatures and comparing them against benchmark featureswe do not perform classifer hyperparameter optimizationand use default parameters instead namely λSVM 1 andcRBF 001

Moreover as the DEAP database relies on 9-point scaleratings it has typically been the case where the midpoint isconsidered as a threshold where ratings greater than thethreshold are considered ldquohighrdquo and those below are con-sidered ldquolowrdquo As was recently emphasized in [4] however

subjects have their own internal biases thus leading tovarying scales for grading and consequently differentthresholds per participant For example as reported in [4]by using a midpoint threshold value of 5 a 6040 ratio ofhighlow levels was obtained across all participants In turnif an individualized threshold was used corresponding to thevalue in which an almost-balanced highlow ratio wasachieved per participant improved results were achieved[60] Figure 3 for example depicts the threshold found foreach participant for arousal and valence in this latter sce-nario As can be seen on average a threshold of 5 was mostoften selected though in a few cases much higher or muchlower values were found thus exemplifying the need for theindividualized approach used herein

26 Fusion Strategies Here we explore three different typesof fusion strategies to combine motif-based and benchmarkspectrum-based features which are described below

261 Feature Fusion As the name suggests this corre-sponds to the direct combination of motif and benchmarkfeatures prior to feature selection

262 Score-Level Fusion e weighted decision fusionmethod proposed in [61] has been used According to thistechnique the fusion classification probability px

0 forx ϵ [0 1] for each class x ϵ 1 2 can be denoted by

px0 1113944

N

i1αip

xi ti (20)

where i is the index of a particular feature group N is thetotal number of groups used and αi are the weights cor-responding to each group (1113936 N

i αi 1)e parameter ti is thetraining set performance of a particular feature group suchthat the fusion probabilities for all classes sum up to unityand is given by

ti Fi

1113936Ni αiFi

(21)

where F1 is the F1-score obtained on the training set using aparticular feature group e weight space was searched forbest performance as this is indicative of the contribution tothe outcome made by each of the feature groups

263 Output Associative Fusion Psychological evidence hassuggested a strong intercorrelation between the valence andarousal dimensions [62ndash65] As such the output associativefusion (OAF) method has been used to model the corre-lations for continuous prediction of valence and arousalscales [66] e OAF framework has been explored here andis depicted by the block diagram in Figure 4 As can be seenfirst individual classifiers make the valence and arousalpredictions for each individual feature group is is thenfollowed by a final prediction step which considers both thevalence and arousal dimensions in order to better predicteach of the two outputs

Table 1 Summary and grouping of features extracted

Feature name No offeatures Group

(Weighted) graphfeatures 20

Motif based features

(Unweighted) graphfeatures 20

Direction of flow 4Small-world features 12Permutation entropy 4Ordinal distancedissimilarity 48

Spectral band powerratio 5

Benchmark spectrum basedfeaturesShannon entropy 1

Spectral power 4Asymmetry index 48


27 Figure of Merit Balanced accuracy (BACC) has beenused as the performance metric as it takes into account classunbalances Balanced accuracy corresponds to the arith-metic mean of the classier sensitivity and specicity ie

BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)

with P TP + FN and N FP + TN and TP and FP cor-respond to true and false positives respectively while TNand FN correspond to true and false negatives

To test the signicance of the attained performancesagainst chance an independent one-sample t-test against arandom voting classier was used (ple 005) as suggested in[18] In order to have a more generalized performance of theclassier once the feature selection step is performedclassier training and testing are performed 100 times withdierent traintest partitioning is setup provides a moregeneralized performance of the features and their invarianceto the training set used e BACC values reported in thetables correspond to the mean plusmn the standard deviation of

0 5 10 15 20Subject number

Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal

Figure 3 reshold variation for each subject for valence (blue) and arousal (orange) dimensions

C2

C1

C3

C4

Benchmark

Motif

Val

Ar

Individual valencepredictions stage

Individual arousalpredictions stage

Finalvalence

predictions

Final arousalpredictions

Output associativefusion stage

Benchmark

Motif

Figure 4 Block diagram of OAF strategy for the two feature groups


all BACC values attained on the test set over all of the 100iterations

3 Results and Discussion

In this section we show and discuss the obtained results interms of impact of feature selection feature group andfusion strategy on overall performance

31 Feature Selection As mentioned previously three dif-ferent feature selection schemes were explored and testedherein Feature selection was implemented in the benchmarkfeatures alone proposed motif feature alone and in thecombined benchmark-motif set e optimal BACC valuesobtained are shown in Tables 2ndash4 respectively along withthe final number of features (nofs) used in the models

As can be seen for ANOVA-based feature selectionfewer than 10 features were used in the models for bothvalence and arousal dimensions with the benchmark fea-tures thus representing roughly one-sixth of the totalamount of available features For the motif group in turnroughly 40 were shown to be useful thus amounting toroughly one-third of the available feature pool Whencombining both feature sets the optimal model also reliedon roughly 40 features thus one quarter of the availablefeature pool

e mRMR algorithm in turn generally resulted infewer top features but with similar overall BACC thuscorroborating the results in [56 57] For the benchmarkfeature set for example BACC asymp 054 was achieved with justthree features for valence thus in line with the asymp 055achieved with ANOVA-selected features For arousal andmotif features similar BACC was achieved but relying onroughly half the number of features relative to ANOVA-based selection With the combined feature set in factimproved BACC was achieved for the arousal dimension butwith fewer than half the number of features chosen byANOVA

Lastly RFE selection typically resulted in the highestaccuracy with the best BACC vs nof tradeoff is is ex-pected as RFE considers the interaction of features amongthemselves and the final outcome Overall the best accuracywas achieved with the combined set followed closely by themodels trained on the proposed motif features esefindings corroborate the complementarity of the two dif-ferent feature types and show the importance of motiffeatures for affective state recognition

A one-way ANOVAwas computed between the differentpairs of feature selection algorithms (ANOVA vs mRMRANOVA vs RFE and RFE vs mRMR) for the benchmarkmotif and combined feature sets to assess the algorithmperformance For the benchmark feature set in the arousaldimension the three algorithms perform similarly with nostatistical differences observable However for the valencedimension the RFE performs significantly better than themRMR algorithm (pval lt 005) while there are no significantdifferences observed between RFE and ANOVA perfor-mances the RFE obtains a similar performance with fewer

features For the motif feature set in the arousal dimensionwe observe the RFE performs significantly better than bothANOVA (pval lt 001) and mRMR (pval asymp 001) In the va-lence dimension we observe a significant difference in al-gorithm performance between RFE and mRMR howeverthe performance of ANOVA is not significant compared toboth the algorithms However we again observe that RFEgives similar performance to ANOVA with half the numberof features thus being more efficient Finally for thecombined feature set in the arousal dimension both mRMRand RFE perform significantly better than ANOVA(pval lt 001) while there are no differences between mRMRand RFE performances with mRMR reaching equivalentperformance with fewer features than RFE In the valencedimension we observe ANOVA (pval asymp 005) and RFE(pval lt 001) perform significantly better than mRMR whilethere is no performance difference between ANOVA andRFE in this case It is interesting to note that the number offeatures for both ANOVA and RFE is almost the same Ingeneral we find the RFE gives significant or equal perfor-mance compared to ANOVA and mRMR with fewernumber of features For feature fusion the algorithm givingthe highest average performance has been considered thealgorithm of choice

Tables 5 and 6 in turn report the top 20 features used inthe models that achieved the best BACC for valence andarousal respectively As can be seen for valence (Table 5) thecβ and βθ power ratios showed to be important along withalpha-band spectral power is corroborates previous work

Table 2 Comparison of different feature selection algorithms andnumber of features (nof) for benchmark feature set

Feature groupsValence Arousal

BACC nof BACC nofANOVA 05490 9 05316 8mRMR 05404 3 05281 4RFE 05531 3 05318 15

Table 3 Comparison of different feature selection algorithms andnumber of features (nof) for motif-based feature set



Table 4 Comparison of different feature selection algorithms andnumber of features (nof) for combined benchmark-motif featureset




which has linked cβ and βθ to audio comprehension[67 68] and consequently to perceived valence in low-quality text-to-speech systems [5] For the motif-basedfeatures in turn small-worldness (c and β band) andweighted graph features (θ band) showed to be importantalongside PE for c and β bands Previous studies have in-dicated to a time-locked theta-band synchronization oc-curring during affective picture processing [69] related to thevalence dimension is synchronization seems to be cap-tured by motif-based graph-theoretic and ordinal similarityfeatures as eight of the top 20 features come from the θ band

Lastly for the combined feature set it can be seen that amix of benchmark and motif features are selected thus

exemplifying the complementarity of the two feature setsOver the entire nof 38 features used in the model 11 arebenchmark features and 27 are motif-based features Inparticular 17 of the top motif features showed importanceacross the motif and combined sets as well as all of the topbenchmark features across benchmark and combined setsAdditionally for the combined set 6 asymmetry features arealso in the top selected features of these 3 are from the sameelectrode pairs as the top ordinal dissimilarity measuresthus showing a complementary nature of the two featuresets e power ratios αθ and(c + β)θ also appear in thecombined feature sets From the motif feature sets apartfrom the overlapping features additional Dm and clusteringcoefficient features appear in the combined feature set alongwith two DoF features from the θ and c bands

For arousal (Table 6) and benchmark feature set almostall power ratios showed to be important alongside severalasymmetry index features particularly those in the frontaland parietal regions Such findings corroborate previousliterature showing the relationship between (i) arousal andfrontal asymmetry [29] in alpha band (eg [70]) and otherbands (eg [71]) (ii) an inherent asymmetry in the rightparietal-temporal regions responsible for modulating au-tonomic and behavioural arousal and (iii) arousal and EEGband power ratios [72]

For motif-based features in turn roughly half the topfeatures corresponded to ordinal distance dissimilaritymeasures thus corroborating the literature on EEG asym-metry and arousal [71 73] Moreover the majority of the topfeatures are from the beta and alpha bands (13 of the top 16)which have been linked to attention-based arousal changes[74] and to changes in visual selective attention [75 76]which is very closely linked to arousal [77]

Interestingly for the combined sets none of the topfeatures were from the benchmark feature set thus sug-gesting that the proposed motif features conveyed improvedarousal information relative to benchmark features emajority of the features corresponded to ordinal distancedissimilarity across all EEG bands Moreover the bestachieving model for motif only and combined feature setswere attained using different feature selection algorithms(RFE and mRMR respectively) Notwithstanding twofeatures coincided as being important namely PE(θ)Dm(T7 T8)(α) and a third showed similar behaviour (C(α)

and Cw(α)) thus suggesting their importance for arousalprediction In the combined set θ showed up in seven of thenof 17 thus also corroborating previous findings [71 73]Lastly most of ordinal dissimilarity features come fromfrontal parietal or temporal regions thus in line withprevious research connecting parietal-temporal regions withautonomic and behavioural arousal as well as frontal re-gions with arousal [78]

32 Individual Feature Groups So far we have explored theperformance achieved with benchmark motif and com-bined feature sets It is interesting however to gauge howeach individual feature subgroup contributes toward af-fective state recognition Table 7 reports the balanced

Table 5 Top 20 features used in the best valence models for thedifferent feature groups

Benchmark(nof 3 FSRFE)

Motif (nof 20FSRFE)

Combined (nof 38FSRFE)

cβ Tr (α) C (α)βθ PE (c) cβSpectral power (α) Gw (θ) (c + β)θ

kw (θ) αθCw (θ) Gw (θ)C (θ) PE (c)PE (β) Dm(P3 P4) (β)S (β) Dm(O1 O2) (θ)S (c) kw (θ)Ls (c) Trw (θ)

Dm(T7 T8) (c) C (θ)Dm(Fc5 Fc6) (β) Spectral power (α)

DoF (θ) C (β)Dm(P3 P4) (β) kw (β)Dm(O1 O2) (β) DoF (θ)Dm(F3 F4) (θ) Cw (θ)

Trw (θ) AI(C3 C4) (β)Ls (θ) AI(P3 P4) (c)

Dm(O1 O2) (θ) Dm(F3 F4) (θ)Dm(F3 F4) (β) Dm(T7 T8) (c)

Table 6 Top 20 features used in the best arousal models for thedifferent feature groups


Motif (nof 16FSRFE)

Combined (nof 17FSmRMR)

(α + β)c PE (β) Dm(O1 O2) (θ)(c + β)θ Tr (β) DoF (c)AI(O1 O2) (β) Cs (β) kw (θ)cβ Dm(T7 T8) (α) DoF (α)AI(P7 P8) (c) Lw (α) Dm(T7 T8) (β)AI(F3 F4) (β) Dm(FC1 FC2) (α) Dm(P3 P4) (β)βθ PE (θ) Dm(T7 T8) (α)Spectral power (β) Dm(P7 P8) (c) PE (θ)AI(Cp5 Cp6) (θ) Lw (c) Dm(F3 F4) (θ)AI(FC1 FC2) (α) Cw (β) Dm(C3 C4) (c)AI(P3 P4) (θ) Dm(C3 C4) (β) Cw (θ)AI(P3 P4) (β) C (α) DoF (θ)AI(C3 C4) (θ) Dm(Cp1 Cp2) (β) Ls (α)AI(Cp1 Cp2) (α) Dm(P3 P4) (α) Dm(Fc5 Fc6) (θ)AI(T7 T8) (c) Dm(F7 F8) (α) Cw (α)

k (α) Dm(Fc5 Fc6) (α)Dm(Fc5 Fc6) (c)


accuracy for each individual feature subgroup for the bestachieving model found after RFE feature selection

As can be seen for valence the weighted and unweightedgraph features achieve similar performances though themodel based on the former feature subgroup relies onnof 2 as opposed to nof 8 In fact all motif-basedfeatures achieved similar performance with small-worldness features being the only ones not significantlybetter than the benchmark (ie plt 001 and indicated by anasterisk in the table) For arousal in turn it is observed thatgraph and small-world feature subgroups do not signifi-cantly improve over the benchmark whereas other motiffeatures such as permutation entropy and ordinal distancedissimilarity do Overall models relying on these twofeature subgroups showed to provide the most discrimi-natory information for valence and arousal models

Additionally among the EEG features we observe thatSE θ and c spectral power never appear as top selectedfeaturesis could be due to the fact that power and entropymeasures are averaged over all electrodes thus removing anyspatial information relevant for the features Notwith-standing averaging ensures that the proposed features areinvariant and robust to the electrode set considered as seenwith the global graph-theoretic features using motif syn-chronization For valence in turn we observe that none ofthe AI features show up among the top in the EEG featureset alone scenario When using only motif features on theother hand seven Dm features (out of nof 20) are selectedthus suggesting that motif features may carry more relevantasymmetry signatures for the task at hand With the com-bined feature set it can be seen that proposed features fromall groups appear in the top list for both valence and arousal

33 Fusion Strategies As mentioned previously three fusionschemes were explored feature score and output associativefusion Tables 2ndash4 show the effects of feature fusion and thegains attained with the combined set relative to using only afeature group individually For the valence dimension forexample gains of 86 and 24 were achieved with featurefusion relative to using benchmark and motif feature alonerespectively As shown in Table 5 the model based on thecombined set relied on features from both feature groups thusemphasizing their complementarity for valence prediction

For arousal feature fusion resulted in more modest gainsrelative to the benchmark (ie 61) and to motif features

(26) Interestingly the best model relied on mRMR se-lected features which did not include benchmark ones esecond best model on the other hand was achieved withRFE feature selection and the top 20 features included sevenbenchmark ones (ie (α + β)c βθ AI(01 02) (β) AI(Fc1Fc2) (β) AI(C3 C4) (c) AI(F3 F4) (β) and AI(Fc1Fc2) (c)) three of which overlap with the top features se-lected from the benchmark alone set e remaining13 features were from the motif group nine of which showedto be top features selected in the motif alone setnamely PE(β) PE(θ) Lw(α) Lw(c) Cs(β) Dm(P7 P8)(c)Dm(Fc1 Fc2)(α) Dm(T7 T8)(α) and Dm(C3 C4)(β) Bycomparing the feature sets selected by mRMR and RFE itseems the former is capable of removing redundancies thatmay exist between Dm and AI asymmetry features butfavouring the motif ones as they provide maximum rele-vance Four features overlap between the two feature se-lection algorithms namely PE(θ) Dm(Fc1 Fc2)(α)Dm(T7 T8)(α) and Dm(C3 C4)(β) thus further suggest-ing their importance for the task at hand

For decision fusion in turn the weight space wassearched in steps of 01 and it was found that for valence thebenchmark feature set resulted in a weight of 02 (ie 08 formotif features) whereas a weight of 03 was found forarousal (ie 07 weight for motifs) Such findings highlightthe importance of motif features over the benchmark onesfor both valence and arousal prediction e BACC resultsshown in Table 8 show the effect of score-level fusion overfeature fusion As can be seen gains are attained only for thearousal dimension thus further suggesting the comple-mentarity of the two feature groups For comparison pur-poses a random voting classifier is also shown forcomparison and all attained BACCs are shown to be sig-nificantly better than chance (ple 001)

Lastly the output associative fusion method was out-performed by all other fusion methods despite showing to besignificantly better than chance Notwithstanding for thevalence dimension it achieved results similar to score-levelfusion without the need for an exhaustive search of weightsHere only two feature groups were explored thus suchadvantage may become more critical in more complex sce-narios involving additional feature groups (eg amplitudemodulation [4]) Overall feature-level fusion showed to bethe best strategy for valence and was observed to be signif-icantly better than score-level (pval asymp 001) and output as-sociative fusion (pval asymp 001) whereas score-level fusion forarousal was significantly better than both feature (pval lt 001)and output associative fusion (pval lt 001) In both cases theproposed motif features showed to provide important dis-criminatory information and to be complementary to existingbenchmark features

34 Comparison with Previous Work ere is increasedinterest in affective state recognition from EEG and dif-ferent methods have been recently proposed in the literaturemany of which have also relied on the DEAP database ework in [20] for example explored graph-theoretic featurescomputed from magnitude square coherence values Such

Table 7 Performance comparison of different individual featuregroups and subgroups

Feature (sub) groupValence Arousal

BACC nof BACC nofWeighted graph 05662lowast 2 05066 6Unweighted graph 05581lowast 8 05006 6Small world 05533 6 05208 2Other motif 05578lowast 9 05632lowast 12Spectral power AI 05400 15 05344 11Power ratio 05467 3 05000 1lowastCases where the results are significantly greater than a random votingclassifier


features were shown to outperform several other spectral-based and wavelet-basedmethods and on the DEAP datasetthey achieved an F1 score of 063 for valence and 060 forarousal using an SVM classifier For direct comparisons thebest models proposed herein achieved an F1 score of 05883for valence and 06960 for arousal thus representing a 16increase in arousal but a drop of 66 for valence It isimportant to emphasize however that the results in [20]relied on leave-one-sample-out (LOSO) cross validationthus the reported results are likely higher than what areachieved with the method described herein

More recently in turn the work in [4] proposed newamplitude modulation coupling features to gauge connec-tivity patterns as a function of valence and arousal BACCvalues of 0594 and 0598 were reported for valence andarousal respectively using SVM classifiers and feature fu-sion whereas somewhat lower values were attained withscore-level fusion for arousal (no changes seen for valence)e values reported in [4] were obtained using a LOSOcross-validation scheme Under the same testing setup ourproposed schemes achieve a BACC of 0614 and 0581 forvalence and arousal thus representing a 33 increase and a285 decrease in performance respectively It is importantto point out that motif-based methods did not rely onamplitude or rate of change information therefore fusingthem with amplitude modulation features might furtherimprove performance

35 Study Limitations is work has taken the first steps atgauging the advantages of motif-based features over exitingspectrum-based benchmarks To this end no optimizationwas done on the classifiers per se in order to directlycompare performances achieved with the same classifiersetup but with varying feature inputs As such it is expectedthat further gains may be observed not only with classifierhyperparameter optimization but also with more complexclassification methods or alternate fusion schemese workin [20] for example showed that relevance vector machines(RVMs) and fusion of RVMs outperformed SVMs especiallyfor the arousal dimension Recent work using deep neuralnetworks has also shown to be a promising route [79]Future work should explore these more complex machinelearning principles combined with motif-based features

4 Conclusion

In this work we propose the use of motif series and graphtheoretic features for improved valence and arousal levelpredictions Experiments on the widely used DEAP databaseshow the proposed motif features outperforming several

spectrum-based benchmark features Feature-level fusionshowed to provide important accuracy gains for bothemotional dimensions thus highlighting the complemen-tarity of the two feature groups for affective state recogni-tion Score-level fusion in turn provided furtherimprovements for arousal prediction Overall gains of 86for valence and 92 for arousal could be achieved with theproposed system relative to the benchmark and gains up to16 could be achieved relative to prior art

Data Availability

e DEAP database used to support the findings of thisstudy were supplied by I Patras under license and so cannotbe made freely available Requests for access to these datashould be made to I Patras (ipatrasqmulacuk) by fillingin and sending the end user license agreement at httpwwweecsqmulacukmmvdatasetsdeapdownloadhtml

Conflicts of Interest

e authors declare that they have no conflicts of interest

Acknowledgments

e authors are thankful to funding provided by the NaturalSciences and Engineering Research Council of Canada(RGPIN-2016-04175)

References

[1] J Makkonen I Avdouevski R Kerminen and A VisaldquoContext awareness in human-computer interactionrdquo inHuman-Computer Interaction InTech London UK 2009

[2] A Dix ldquoHuman-computer interactionrdquo in Encyclopedia ofDatabase Systems pp 1327ndash1331 Springer New York NYUSA 2009

[3] R W Picard S Papert W Bender et al ldquoAffective learning-amanifestordquo BT Technology Journal vol 22 no 4 pp 253ndash2692004

[4] A Clerico A Tiwari R Gupta S Jayaraman and T H FalkldquoElectroencephalography amplitude modulation analysis forautomated affective tagging of music video clipsrdquo Frontiers inComputational Neuroscience vol 11 p 115 2018

[5] R Gupta H J Banville and T H Falk ldquoMultimodal phys-iological quality-of-experience assessment of text-to-speechsystemsrdquo IEEE Journal of Selected Topics in Signal Processingvol 11 no 1 pp 22ndash36 2017

[6] E Ganglbauer J Schrammel S Deutsch and M TscheligildquoApplying psychophysiological methods for measuring userexperience possibilities challenges and feasibilityrdquo inWorkshop on User Experience Evaluation Methods in ProductDevelopment CiteSeer PA USA 2009

[7] E Modica G Cartocci D Rossi et al ldquoNeurophysiologicalresponses to different product experiencesrdquo ComputationalIntelligence and Neuroscience vol 2018 Article ID 961630110 pages 2018

[8] P Arico G Borghini G Di Flumeri et al ldquoAdaptive auto-mation triggered by eeg-based mental workload index apassive brain-computer interface application in realistic airtraffic control environmentrdquo Frontiers in Human Neurosci-ence vol 10 p 539 2016

Table 8 Performance comparison of different fusion methods anda random voting classifier with chance levels

Fusion methods Valence BACC Arousal BACCFeature 06010 05645Score 05875 05807OAF 05873 05568Random voting 05018 05028


[9] P Arico G Borghini G Di Flumeri N SciaraffaA Colosimo and F Babiloni ldquoPassive bci in operationalenvironments insights recent advances and future trendsrdquoIEEE Transactions on Biomedical Engineering vol 64 no 7pp 1431ndash1436 2017

[10] A R Damasio and S Sutherland ldquoDescartesrsquo error emotionreason and the human brainrdquo Nature vol 372 no 6503p 287 1994

[11] B De Martino D Kumaran B Seymour and R J DolanldquoFrames biases and rational decision-making in the humanbrainrdquo Science vol 313 no 5787 pp 684ndash687 2006

[12] G Loewenstein and J S Lerner ldquoe role of affect in decisionmakingrdquoHandbook of Affective Science vol 619 no 642 p 32003

[13] N Schwarz ldquoEmotion cognition and decision makingrdquoCognition amp Emotion vol 14 no 4 pp 433ndash440 2000

[14] S Poria E Cambria R Bajpai and A Hussain ldquoA review ofaffective computing from unimodal analysis to multimodalfusionrdquo Information Fusion vol 37 pp 98ndash125 2017

[15] J A Healey Wearable and automotive systems for affectrecognition from physiology Mass achusetts Institute ofTechnology PhD dissertation 2000

[16] P J Lang M K Greenwald M M Bradley and A O HammldquoLooking at pictures affective facial visceral and behavioralreactionsrdquo Psychophysiology vol 30 no 3 pp 261ndash273 1993

[17] J Kim and E Andre ldquoEmotion recognition based on phys-iological changes in music listeningrdquo IEEE Transactions onPattern Analysis and Machine Intelligence vol 30 no 12pp 2067ndash2083 2008

[18] S Koelstra C Muhl M Soleymani et al ldquoDEAP a databasefor emotion analysis using physiological signalsrdquo IEEETransactions on Affective Computing vol 3 no 1 pp 18ndash312012

[19] C Muhl B Allison A Nijholt and G Chanel ldquoA survey ofaffective brain computer interfaces principles state-of-the-art and challengesrdquo Brain-Computer Interfaces vol 1 no 2pp 66ndash84 2014

[20] R Gupta K Laghari H Banville and T H Falk ldquoUsingaffective brain-computer interfaces to characterize humaninfluential factors for speech quality-of-experience perceptionmodellingrdquo Human-centric Computing and InformationSciences vol 6 no 1 p 5 2016

[21] T O Zander and C Kothe ldquoTowards passive brain-computerinterfaces applying brain-computer interface technology tohuman-machine systems in generalrdquo Journal of Neural En-gineering vol 8 no 2 article 025005 2011

[22] H Ayaz B Onaral K Izzetoglu P A ShewokisR McKendrick and R Parasuraman ldquoContinuous moni-toring of brain dynamics with functional near infraredspectroscopy as a tool for neuroergonomic research empiricalexamples and a technological developmentrdquo Frontiers inHuman Neuroscience vol 7 p 871 2013

[23] R McKendrick R Parasuraman and H Ayaz ldquoWearablefunctional near infrared spectroscopy (fNIRS) and trans-cranial direct current stimulation (tDCS) expanding vistas forneurocognitive augmentationrdquo Frontiers in Systems Neuro-science vol 9 p 27 2015

[24] J J Kierkels M Soleymani and T Pun ldquoQueries and tags inaffect-based multimedia retrievalrdquo in Proceedings of IEEEInternational Conference on Multimedia and Expo (ICME2009) pp 1436ndash1439 2009

[25] D Heger F Putze and T Schultz ldquoOnline workload rec-ognition from eeg data during cognitive tests and human-machine interaction KI 2010 Advances in Artificial

Intelligencerdquo in Proceedings of Annual Conference on Arti-ficial Intelligence pp 410ndash417 Springer New York NY USA2010

[26] C A Kothe and S Makeig ldquoEstimation of task workload fromeeg data new and current tools and perspectivesrdquo in Pro-ceedings of 2011 Annual International Conference of the IEEEEngineering in Medicine and Biology Society (EMBC)pp 6547ndash6551 2011

[27] R Jenke A Peer and M Buss ldquoFeature extraction and se-lection for emotion recognition from eegrdquo IEEE Transactionson Affective Computing vol 5 no 3 pp 327ndash339 2014

[28] R J Davidson and A J Tomarken ldquoLaterality and emotionan electrophysiological approachrdquoin Handbook of Neuro-psychology vol 3 pp 419ndash441 Elsevier Amsterdam Neth-erlands 1989

[29] J A Coan and J J B Allen ldquoFrontal eeg asymmetry as amoderator and mediator of emotionrdquo Biological Psychologyvol 67 no 1-2 pp 7ndash50 2004

[30] A Kirke and E R Miranda ldquoCombining eeg frontal asym-metry studies with affective algorithmic composition andexpressive performance modelsrdquo in Proceedings of In-ternational Conference on Management Case Toronto Can-ada April 2011

[31] G Di Flumeri M T Herrero A Trettel et al ldquoEeg frontalasymmetry related to pleasantness of olfactory stimuli inyoung subjectsrdquo in Selected Issues in Experimental Economicspp 373ndash381 Springer New York NY USA 2016

[32] G Di Flumeri P Arico G Borghini et al ldquoEeg-basedapproach-withdrawal index for the pleasantness evaluationduring taste experience in realistic settingsrdquo in Proceedings of2017 39th Annual International Conference of the IEEE En-gineering in Medicine and Biology Society (EMBC)pp 3228ndash3231 Jeju Island Korea July 2017

[33] J Minguillon M A Lopez-Gordo and F Pelayo ldquoTrends ineeg-bci for daily-life requirements for artifact removalrdquoBiomedical Signal Processing and Control vol 31 pp 407ndash4182017

[34] J-P iran F Marques and H Bourlard Multimodal SignalProcessing Jeory and Applications for Human-ComputerInteraction Academic Press Cambridge MA USA 2009

[35] N Nicolaou and J Georgiou ldquoe use of permutation entropyto characterize sleep electroencephalogramsrdquo Clinical EEGand Neuroscience vol 42 no 1 pp 24ndash28 2011

[36] E Olofsen J W Sleigh and A Dahan ldquoPermutation entropyof the electroencephalogram a measure of anaesthetic drugeffectrdquo British Journal of Anaesthesia vol 101 no 6pp 810ndash821 2008

[37] X Li G Ouyang and D A Richards ldquoPredictability analysisof absence seizures with permutation entropyrdquo Epilepsy Re-search vol 77 no 1 pp 70ndash74 2007

[38] G Ouyang C Dang D A Richards and X Li ldquoOrdinalpattern based similarity analysis for eeg recordingsrdquo ClinicalNeurophysiology vol 121 no 5 pp 694ndash703 2010

[39] A Sengupta A Tiwari A Chaudhuri and A RoutrayldquoAnalysis of loss of alertness due to cognitive fatigue usingmotif synchronization of eeg recordsrdquo in Proceedings of 2016IEEE 38th Annual International Conference of the Engineeringin Medicine and Biology Society (EMBC) pp 1652ndash1655Orlando FL USA August 2016

[40] U Parlitz S Berg S Luther A Schirdewan J Kurths andN Wessel ldquoClassifying cardiac biosignals using ordinalpattern statistics and symbolic dynamicsrdquo Computers in Bi-ology and Medicine vol 42 no 3 pp 319ndash327 2012


[41] C Bandt and B Pompe ldquoPermutation entropy a naturalcomplexity measure for time seriesrdquo Physical Review Lettersvol 88 no 17 article 174102 2002

[42] K Keller A Unakafov and V Unakafova ldquoOrdinal patternsentropy and eegrdquo Entropy vol 16 no 12 pp 6212ndash62392014

[43] J A Russell ldquoA circumplex model of affectrdquo Journal ofPersonality and Social Psychology vol 39 no 6 pp 1161ndash1178 1980

[44] M M Bradley and P J Lang ldquoMeasuring emotion the self-assessment manikin and the semantic differentialrdquo Journal ofBehavior Jerapy and Experimental Psychiatry vol 25 no 1pp 49ndash59 1994

[45] M Soleymani J Lichtenauer T Pun and M Pantic ldquoAmultimodal database for affect recognition and implicittaggingrdquo IEEE Transactions on Affective Computing vol 3no 1 pp 42ndash55 2012

[46] G Borghini L Astolfi G Vecchiato D Mattia andF Babiloni ldquoMeasuring neurophysiological signals in aircraftpilots and car drivers for the assessment of mental workloadfatigue and drowsinessrdquo Neuroscience amp Biobehavioral Re-views vol 44 pp 58ndash75 2014

[47] B T Jap S Lal P Fischer and E Bekiaris ldquoUsing eeg spectralcomponents to assess algorithms for detecting fatiguerdquo ExpertSystems with Applications vol 36 no 2 pp 2352ndash2359 2009

[48] P Putman J van Peer I Maimari and S van der Werff ldquoEegthetabeta ratio in relation to fear-modulated response-inhibition attentional control and affective traitsrdquo Bi-ological psychology vol 83 no 2 pp 73ndash78 2010

[49] N Kannathal M L Choo U R Acharya and P K SadasivanldquoEntropies for detection of epilepsy in eegrdquo ComputerMethods and Programs in Biomedicine vol 80 no 3pp 187ndash194 2005

[50] R S Rosario P T Cardoso M A Muntildeoz P Montoya andJ G V Miranda ldquoMotif-synchronization a new method foranalysis of dynamic brain networks with eegrdquo Physica AStatistical Mechanics and its Applications vol 439 pp 7ndash192015

[51] M Rubinov and O Sporns ldquoComplex network measures ofbrain connectivity uses and interpretationsrdquo Neuroimagevol 52 no 3 pp 1059ndash1069 2010

[52] R Gupta and K Rehman Laghari ldquoRelevance vector classifierdecision fusion and eeg graph-theoretic features for automaticaffective state characterizationrdquo Neurocomputing vol 174pp 875ndash884 2016

[53] D S Falk and E Bullmore ldquoSmall-world brain networksrdquoNeuroscientist vol 12 no 6 pp 512ndash523 2016

[54] C Stam B Jones G Nolte M Breakspear and P ScheltensldquoSmall-world networks and functional connectivity in Alz-heimerrsquos diseaserdquo Cerebral Cortex vol 17 no 1 pp 92ndash992006

[55] C Lithari M A Klados C Papadelis C Pappas M Albaniand P D Bamidis ldquoHow does the metric choice affect brainfunctional connectivity networksrdquo Biomedical Signal Pro-cessing and Control vol 7 no 3 pp 228ndash236 2012

[56] H Peng F Long and C Ding ldquoFeature selection based onmutual information criteria of max-dependency max-relevance and min-redundancyrdquo IEEE Transactions on pat-tern analysis and machine intelligence vol 27 no 8pp 1226ndash1238 2005

[57] X-W Wang D Nie and B-L Lu ldquoEeg-based emotionrecognition using frequency domain features and supportvector machinesrdquo Neural Information Processing in Pro-ceedings of International Conference on Neural Information

Processing pp 734-743 Springer Berlin Heidelberg Ger-many 2011

[58] B Scholkopf and A J Smola Learning with Kernels SupportVector Machines Regularization Optimization and BeyondMIT Press Cambridge MA USA 2002

[59] F Pedregosa G Varoquaux A Gramfort et al ldquoScikit-learnmachine learning in pythonrdquo Journal of machine learningresearch vol 12 pp 2825ndash2830 2011


[61] S Koelstra and I Patras ldquoFusion of facial expressions and eegfor implicit affective taggingrdquo Image and Vision Computingvol 31 no 2 pp 164ndash174 2013

[62] A M Oliveira M P Teixeira I B Fonseca and M OliveiraldquoJoint model-parameter validation of self-estimates of valenceand arousal probing a differential-weighting model of af-fective intensityrdquo Proceedings of Fechner Day vol 22 no 1pp 245ndash250 2006

[63] N Alvarado ldquoArousal and Valence in the Direct Scaling ofEmotional Response to Film ClipsrdquoMotivation and Emotionvol 21 no 4 pp 323ndash348 1997

[64] R D Lane and L Nadel Cognitive Neuroscience of EmotionOxford University Press Oxford UK 1999

[65] P Lewis H Critchley P Rotshtein and R Dolan ldquoNeuralcorrelates of processing valence and arousal in affectivewordsrdquo Cerebral Cortex vol 17 no 3 pp 742ndash748 2006

[66] M A Nicolaou H Gunes and M Pantic ldquoContinuousprediction of spontaneous affect from multiple cues andmodalities in valence-arousal spacerdquo IEEE Transactions onAffective Computing vol 2 no 2 pp 92ndash105 2011

[67] O Ghitza ldquoLinking speech perception and neurophysiologyspeech decoding guided by cascaded oscillators locked to theinput rhythmrdquo Frontiers in Psychology vol 2 p 130 2011

[68] O Ghitza ldquoOn the role of theta-driven syllabic parsing indecoding speech intelligibility of speech with a manipulatedmodulation spectrumrdquo Frontiers in Psychology vol 3 p 2382012

[69] L Aftanas A Varlamov S Pavlov V Makhnev and N RevaldquoAffective picture processing event-related synchronizationwithin individually defined human theta band is modulatedby valence dimensionrdquo Neuroscience Letters vol 303 no 2pp 115ndash118 2001

[70] S Arndt J-N Antons R Gupta et al ldquoe effects of text-to-speech system quality on emotional states and frontal alphaband powerrdquo in Proceedings of 2013 6th International IEEEEMBS Conference on Neural Engineering (NER) pp 489ndash492San Diego CA USA 2013

[71] L I Aftanas A A Varlamov S V Pavlov V P Makhnevand N V Reva ldquoTime-dependent cortical asymmetries in-duced by emotional arousal eeg analysis of event-relatedsynchronization and desynchronization in individually de-fined frequency bandsrdquo International Journal of Psycho-physiology vol 44 no 1 pp 67ndash82 2002

[72] R J Barry A R Clarke S J Johnstone R McCarthy andM Selikowitz ldquoElectroencephalogram θβ ratio and arousal inattention-deficithyperactivity disorder evidence of in-dependent processesrdquo Biological Psychiatry vol 66 no 4pp 398ndash401 2009

[73] G G Knyazev ldquoMotivation emotion and their inhibitorycontrol mirrored in brain oscillationsrdquo Neuroscience amp Bio-behavioral Reviews vol 31 no 3 pp 377ndash395 2007


[74] FM Howells D J Stein and V A Russell ldquoPerceivedmentaleffort correlates with changes in tonic arousal during atten-tional tasksrdquo Behavioral and Brain Functions vol 6 no 1p 39 2010

[75] J J Foxe and A C Snyder ldquoe role of alpha-band brainoscillations as a sensory suppression mechanism during se-lective attentionrdquo Frontiers in Psychology vol 2 p 154 2011

[76] A Wrobel ldquoBeta activity a carrier for visual attentionrdquo ActaNeurobiologiae Experimentalis vol 60 no 2 pp 247ndash2602000

[77] J Coull ldquoNeural correlates of attention and arousal insightsfrom electrophysiology functional neuroimaging and psy-chopharmacologyrdquo Progress in Neurobiology vol 55 no 4pp 343ndash361 1998

[78] L J Metzger S R Paige M A Carson et al ldquoPtsd arousal anddepression symptoms associated with increased right-sidedparietal eeg asymmetryrdquo Journal of Abnormal Psychologyvol 113 no 2 p 324 2004

[79] F Movahedi J L Coyle and E Sejdic ldquoDeep belief networksfor electroencephalography a review of recent contributionsand future outlooksrdquo IEEE Journal of Biomedical and HealthInformatics vol 22 no 3 pp 642ndash652 2018


Computer Games Technology

International Journal of

Hindawiwwwhindawicom Volume 2018

Hindawiwwwhindawicom

Journal ofEngineeringVolume 2018

Advances in

FuzzySystems


Volume 2018


ReconfigurableComputing



Applied Computational Intelligence and Soft Computing

thinspAdvancesthinspinthinsp

thinspArtificial Intelligence

Hindawiwwwhindawicom Volumethinsp2018


Civil EngineeringAdvances in


Electrical and Computer Engineering

Journal of

Journal of

Computer Networks and Communications


Hindawi

wwwhindawicom Volume 2018

Advances in

Multimedia


Biomedical Imaging



Engineering Mathematics


RoboticsJournal of



Computational Intelligence and Neuroscience


Mathematical Problems in Engineering

Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018

Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom

The Scientific World Journal

Volume 2018


Human-ComputerInteraction

Advances in


Scientic Programming

Submit your manuscripts atwwwhindawicom

states [9 19 20] Within BCI-based affective computingmethods electroencephalography (EEG) has remained themost popular modality due to its noninvasiveness hightemporal resolution and portability [21] though alternativemodalities such as near-infrared spectroscopy (NIRS) areslowly emerging [5 22 23] Typically with EEG-basedsystems spectral power features have been widely used(eg [18 24ndash26]) including frontal interhemisphericasymmetry features [27ndash32] EEG signals however are verysensitive to artefacts such as eye blinks and musclemovement [33] To overcome such issues artefact removalalgorithms can be used Alternately new noise-robust fea-tures can be developed andor multimodal fusion strategiescan be explored [34]

In this paper focus is placed on the latter and motif-based features are proposed and tested alone or alongsidealternate complementary features Motif-based analysis hasshown to be useful in the past to recognize sleep states [35]as well as the effects of anesthesia [36] to detect seizures[37 38] and to measure alertness [39] Motif-based methodsare inherently robust to noise as they deal with the shape ofthe time series and are unaffected by the magnitude[38 40 41] To the best of our knowledge they have yet to beexplored for affective state monitoring thus this paper fillsthis gap In particular we compare the proposed featureswith spectral power and spectral asymmetry benchmarkfeatures Notwithstanding one main limitation of motiffeatures concerns the loss of both amplitude and rate-of-change information when time series are converted intomotif series [40 42] As such we also explore three differentfusion strategies to combine information from the proposedmotif features and classical benchmark features Experi-mental tests on a publicly available database [18] are per-formed which show the advantages of the proposed featuresover benchmark ones as well as the benefits of fusion foraffective state monitoring

e remainder of this paper is organized as followsSection 2 describes the materials and methods used in-cluding the database considered proposed and benchmarkfeatures fusion methods used and performance metricsused Section 3 then presents and discusses the results ob-tained and conclusions are drawn in Section 4

2 Materials and Methods

Here we describe the database used benchmark featuresproposed motif features as well as the feature selectionschemes employed classifiers and fusion schemes explored

21 DEAP Database is study relies on the publiclyavailable widely used DEAP (Dataset for Emotion Analysisusing EEG and physiological signals) database As detailed in[18] thirty-two healthy participants (50 females averageage 269 years) were recruited and consented to participatein the study irty-two channel EEG data were recordedusing a Biosemi ActiveTwo system (Amsterdam Nether-lands) at a sampling rate of 512Hz Electrodes were placedon the scalp according to the International 10ndash20 system

Participants were presented with 40 one-minute longmusic videos with varying emotional content ese videoclips were selected based on a previous analysis of severalhundred videos as they were shown to elicit the strongestreactions across the four quadrants in the valence-arousalspace (ie low valence low arousal low valence higharousal high valence low arousal and high valence higharousal) e valence-arousal space is a two dimensionalscale used to characterize emotions [43] Valence refers tothe (un)pleasantness of an event whereas arousal refers tothe intensity of the event ranging from very calming tohighly exciting Using this space various emotions can bemapped as shown in Figure 1 Prior to each video there wasa baseline period of five seconds where the participants wereasked to fixate at a cross in the middle of the screen Fol-lowing the presentation of each video participants wereasked to rate the music videos on discrete 9-point scales forvalence and arousal using the self-assessment manikins(SAM) [44] While other dimensional ratings such asdominance and liking were also collected these have notbeen explored herein

e EEG data are available for public download in rawformat or in preprocessed format which includes commonreferencing down-sampling to 128Hz bandpass filteringbetween 4 and 45Hz and eye blink artefact removal viaindependent component analysis Moreover only the lastthree seconds of the five-second baseline are available Sincethis is a standard pipeline for EEG processing the analysisreported herein is done on the preprocessed data Data persubject were epoched into forty 60 s long trials with a 3 s longprestimulus baseline e prestimulus baseline was thensubtracted from the preprocessed datae interested readercan refer to [18] for more details on the DEAP database andits data collection process

22 Benchmark Features As mentioned previously spectralpower features in different EEG bands have been widely usedfor affective state monitoring including for the DEAP data-base [18 45] Moreover an interhemispheric asymmetry inspectral power has also been been reported in the affectivestate literature [27 28 30ndash32] particularly in frontal brainregions [29 31] Typically EEG signals are band decomposedinto theta (4lt θlt 8 Hz) alpha (8lt αlt 13 Hz) beta(13lt βlt 30 Hz) and gamma (30lt clt 45 Hz) bands Here48 asymmetry index (AI) features (12 interhemisphereicelectrode pairs times 4 bands) were computed for the followingelectrodes pairs Fp2-Fp1 F3-F4 F7-F8 FC1-FC2 FC5-FC6C3-C4 T7-T8 Cp1-Cp2 Cp5-Cp6 P3-P4 P7-P8 andO1-O2

Moreover EEG band ratios have also been explored inthe past for tasks such as human mental state monitoringfatigue attention control and negative emotional responsemonitoring [46ndash48] thus are also included here as bench-mark features e ratios computed include cβ βθ

αθ (α + β)c and (c + β)θ e ratios are computed in-dividually over each electrode Lastly the Shannon entropy[49] has been used as a feature to measure the complexity ofthe EEG time series Shannon entropy can be calculated asfollows


SE minussumj




Xm(i)


(2)




PE minussumn

j




Dm(X Y)

n

nminus 1

radic

sumn

i

px(i)minuspy(i)( )2

radicradic

(4)


Valence

Aro

usal


Hig

h (e

xcite

d)Lo

w (c

alm

)

Neutral



Happy

Contented

Serene

Relaxed

Excited

AlertTense

Nervous

Upset

Bored

Depressed

Sad





i1Jτ0i sum

lm

i1Jτ1i sum

lm

i1Jτni

(5)

with

Jτi (i) 1 if XMi

YMi+τ

0 else (6)




Qxy max cXY cYX( )

lm

qXY 0 if cXY cYX


(7)





ki sumjϵNe

aij (8)


kwi sumjϵNe

wij (9)



C 1NesumNe

i

eiki ki minus 1( )

(10)


X (i) X (i + λ)

X (i + 2λ)

Time seriesX (t)



Cw

1

Ne

1113944

Ne

i

twi

kwi kw

i minus 1( 1113857 (11)


ti 12

1113944jhϵNe

wijwjhwhi1113872 111387313

(12)




Tr 3λ

k(kminus 1) (13)



L 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdij (14)




G 1

Ne Ne minus 1( 11138571113944

Ne





Ls L

Lrand (16)


Cs C

Crand (17)


S Cs

Ls

(18)



DoF 1113944ij

qij (19)
















px0 1113944

N

i1αip

xi ti (20)



ti Fi

1113936Ni αiFi

(21)














BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




SE minussumj




Xm(i)


(2)




PE minussumn

j




Dm(X Y)

n

nminus 1

radic

sumn

i

px(i)minuspy(i)( )2

radicradic

(4)


Valence

Aro

usal


Hig

h (e

xcite

d)Lo

w (c

alm

)

Neutral



Happy

Contented

Serene

Relaxed

Excited

AlertTense

Nervous

Upset

Bored

Depressed

Sad





i1Jτ0i sum

lm

i1Jτ1i sum

lm

i1Jτni

(5)

with

Jτi (i) 1 if XMi

YMi+τ

0 else (6)




Qxy max cXY cYX( )

lm

qXY 0 if cXY cYX


(7)





ki sumjϵNe

aij (8)


kwi sumjϵNe

wij (9)



C 1NesumNe

i

eiki ki minus 1( )

(10)


X (i) X (i + λ)

X (i + 2λ)

Time seriesX (t)



Cw

1

Ne

1113944

Ne

i

twi

kwi kw

i minus 1( 1113857 (11)


ti 12

1113944jhϵNe

wijwjhwhi1113872 111387313

(12)




Tr 3λ

k(kminus 1) (13)



L 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdij (14)




G 1

Ne Ne minus 1( 11138571113944

Ne





Ls L

Lrand (16)


Cs C

Crand (17)


S Cs

Ls

(18)



DoF 1113944ij

qij (19)
















px0 1113944

N

i1αip

xi ti (20)



ti Fi

1113936Ni αiFi

(21)














BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in






i1Jτ0i sum

lm

i1Jτ1i sum

lm

i1Jτni

(5)

with

Jτi (i) 1 if XMi

YMi+τ

0 else (6)




Qxy max cXY cYX( )

lm

qXY 0 if cXY cYX


(7)





ki sumjϵNe

aij (8)


kwi sumjϵNe

wij (9)



C 1NesumNe

i

eiki ki minus 1( )

(10)


X (i) X (i + λ)

X (i + 2λ)

Time seriesX (t)



Cw

1

Ne

1113944

Ne

i

twi

kwi kw

i minus 1( 1113857 (11)


ti 12

1113944jhϵNe

wijwjhwhi1113872 111387313

(12)




Tr 3λ

k(kminus 1) (13)



L 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdij (14)




G 1

Ne Ne minus 1( 11138571113944

Ne





Ls L

Lrand (16)


Cs C

Crand (17)


S Cs

Ls

(18)



DoF 1113944ij

qij (19)
















px0 1113944

N

i1αip

xi ti (20)



ti Fi

1113936Ni αiFi

(21)














BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




Cw

1

Ne

1113944

Ne

i

twi

kwi kw

i minus 1( 1113857 (11)


ti 12

1113944jhϵNe

wijwjhwhi1113872 111387313

(12)




Tr 3λ

k(kminus 1) (13)



L 1

Ne Ne minus 1( 11138571113944

Ne

j1inejdij (14)




G 1

Ne Ne minus 1( 11138571113944

Ne





Ls L

Lrand (16)


Cs C

Crand (17)


S Cs

Ls

(18)



DoF 1113944ij

qij (19)
















px0 1113944

N

i1αip

xi ti (20)



ti Fi

1113936Ni αiFi

(21)














BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in













px0 1113944

N

i1αip

xi ti (20)



ti Fi

1113936Ni αiFi

(21)














BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in





BACC sens + spec

2 (22)

where

sens TPP

spec TNN

(23)




Per s

ubje

ct th

resh

old

25 30 353

35

4

45

5

55

6

65

7

ValenceArousal


C2

C1

C3

C4

Benchmark

Motif

Val

Ar



Finalvalence

predictions



Benchmark

Motif

































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in


































Motif (nof 20FSRFE)










Motif (nof 16FSRFE)





















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




















4 Conclusion



Data Availability




Acknowledgments


References





























































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in





















































































Advances in

FuzzySystems


Volume 2018













Journal of

Journal of



Hindawi


Advances in

Multimedia


Biomedical Imaging





RoboticsJournal of









Volume 2018



Advances in




FusionofMotif-andSpectrum-RelatedFeaturesforImproved EEG...

Documents

Transcript of FusionofMotif-andSpectrum-RelatedFeaturesforImproved EEG...