Research Article...

Research ArticleEye-Tracking Analysis for Emotion Recognition

Paweł Tarnowski Marcin Kołodziej Andrzej Majkowski and Remigiusz Jan Rak

Institute of eory of Electrical Engineering Measurement and Information Systems Warsaw University of TechnologyWarsaw 00-662 Poland

Correspondence should be addressed to Paweł Tarnowski tarnowspeepwedupl

Received 11 December 2019 Revised 23 July 2020 Accepted 3 August 2020 Published 1 September 2020

Academic Editor Amparo Alonso-Betanzos

Copyright copy 2020 Paweł Tarnowski et al is is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work isproperly cited

is article reports the results of the study related to emotion recognition by using eye-tracking Emotions were evoked bypresenting a dynamic movie material in the form of 21 video fragments Eye-tracking signals recorded from 30 participants wereused to calculate 18 features associated with eye movements (fixations and saccades) and pupil diameter To ensure that thefeatures were related to emotions we investigated the influence of luminance and the dynamics of the presented movies reeclasses of emotions were considered high arousal and low valence low arousal and moderate valence and high arousal and highvalence A maximum of 80 classification accuracy was obtained using the support vector machine (SVM) classifier and leave-one-subject-out validation method

1 Introduction

11 Motivation Emotions play a very important role ineveryonersquos life People feel emotions during everyday tasksduring interpersonal communication when making deci-sions learning or during cognitive activities For severalyears a large increase in the number of studies on methodsof automatic recognition of emotions has been observedis is due to the increasing number and widespread use ofelectronic devices such as smartphones tablets and com-puters e issue of recognizing emotions is an interdisci-plinary problem and includes computer science psychologyand cognitive science e development of effective methodsfor recognizing emotions can not only improve the inter-action of people with machines but also contribute to thedevelopment of other areas such as psychology medicine[1] education [2] and entertainment [3]

12 State of the Art Automatic recognition of emotions inmedicine can be used to diagnose and treat diseases such asposttraumatic stress disorder [4] and depression [5] It is alsoused in diseases such as autism or Aspergerrsquos syndrome[6 7] Another application of emotion recognition is

education and supporting distance learning In [8] a speechsignal was used to recognize studentsrsquo emotions during thelesson In [9] a tool is presented to visualize the degree ofattention and involvement of students using the mea-surement of electrodermal skin activity Emotion recogni-tion systems can improve teacher-student interaction and asa result lead to better learning results Currently due to therapidly growing number of multimedia materials appearingon the Internet numerous studies are carried out related totheir automatic labeling Movies or music is often designedto evoke specific emotions hence the automatically gener-ated labels should contain information about the emotionalcharacteristics of the material [10] In [11] an EEG signalwas used to label movies in terms of evoked emotions Torecognize emotions the features of the signal registered for agroup of people watching the same set of movies were usedSimilar studies are described in [12 13] using in addition tothe EEG signal facial expressions

Emotions are felt by a person as certain reactions of thebody to a given situation or stimulus e occurrence ofthese reactions is the basis for the operation of systems forrecognizing the emotional state of a person Facial expres-sions [12 14 15] speech signal recordings [16 17] and brainactivity and changes in other human physiological

HindawiComputational Intelligence and NeuroscienceVolume 2020 Article ID 2909267 13 pageshttpsdoiorg10115520202909267

parameters are subject to testing In the case of brain activitysignals from the central nervous system are recorded usingelectroencephalography (EEG) [12 14 15] as well asmedical magnetic resonance imaging (fMRI) [18] Otherphysiological signals used include electrocardiography(ECG) [19ndash21] electromyography (EMG) [22] electroder-mal activity (EDA) [19 20 23] heart rate [24ndash26] respi-ration rate and depth [24 27] and arterial pressure [24]Eye-tracking [28ndash32] and pupil width [33ndash36] are also usedto recognize emotions

Interest in eye movement research dates back to thenineteenth century when the method of direct observationwas useden invasive eye-trackers were constructed usinga special type of lens placed on the eyeball At presentnoncontact devices which use infrared cameras are mostcommonly used for eye-tracking e development of suchdevices increased the popularity of eye movement researchamong scientists from many different fields Eye-trackingresearch is applied in areas such as cognitive psychologyneuropsychology usability testing or marketing [37] eeyeball movement is examined while viewing the presentedmaterials eg photos or movies and while reading playingbrowsing the web or using graphical user interfaces Fewattempts have also been made to use eye-tracking to rec-ognize emotions [32] In [28] features associated with eyemovements were used to recognize three classes of emotionse classification accuracy obtained was 778 In [38] forthree classes of emotions recognized using eye-trackingmethods the classification accuracy was 688 In [30] theuse of features such as fixation duration amplitude andduration of saccades and blinking frequency for classifi-cation of four classes of emotions was described achieving6782 accuracy In all mentioned studies emotions wereevoked by presenting movies

e eye-tracking often includes measuring the diameterof the pupil which is responsible for regulating the amountof light falling on the retina In addition to light intensity thepupil response is also influenced by other factors esefactors include the emotions experienced by oneself [39 40]To date studies have shown that the pupil diameter increaseswhen feeling excited [34 36] In [29] an attempt to rec-ognize three classes of emotions (positive negative andneutral) evoked by movies using pupil width was presentedClassification accuracy of 589was achievede study [41]describes research on recognizing emotions caused byplaying a video game A set of features related to pupil widthwere used for the classification For three classes of emo-tions the classification accuracy achieved was 76 and614 respectively for the arousal and the valence scalese work [35] presents the use of pupillometry to measureemotional arousal in videos

13 e Aim of is Paper e purpose of the presentedresearch is to analyze whether it is possible to recognizeemotions by using eye-tracking signal Emotions have beencaused by the presentation of video material For the presentresearch an experiment was designed during which a groupof 30 participants watched a set of 21 fragments of movies

intended to evoke emotions During movie presentationeye-tracking data were recorded As features we used eye-movement specific elements such as fixations and saccadesFeatures related to the pupil diameter were also calculatedWe recognized three classes of emotions high arousal andlow valence low arousal and moderate valence and higharousal and high valence ree classifiers were tested SVMLDA and k-NN e leave-one-subject-out method wasused to assess the quality of the classification Individualsuccessive stages of the research were the following

(i) acquisition of eye-tracking data while evokingemotions

(ii) performing signal preprocessing(iii) removal of the effect of luminance on pupil width(iv) calculation of eye-tracking features related to eye

movements and pupil width(v) classification of emotions using SVM LDA and k-

NN

Videomaterial is characterized by fast scene changes andmoving objects e dynamics of the movies can greatlyinfluence the eye-tracking features and the possibility to usethem for emotion recognition Innovative in our research isthat we examined the effects of the dynamics of movies onclassification accuracy Further when measuring the pupildiameter the impact of luminance and lighting conditions isvery important e effect of luminance on pupil width wasremoved using linear regression

2 Materials and Methods

21 Evoking Emotions An important element of the studywas evoking emotions in participants of the experimentEmotions are most often caused by the presentation ofvisual stimuli (images and movies) [34 38 42 43] andsound stimuli (music and sounds) [36 44] In our ex-periment emotions were evoked by a video presentationon a monitor screen e presented material comprisedshort movies along with the soundtrack Twenty-onemovies were used in the experiment ey elicited sixbasic emotions as defined in [45] that is happinesssadness anger surprise disgust and fear Several othermovies were selected to be as neutral as possible and didnot cause any of the above emotions e selection ofmovies that cause a specific emotion is a difficult taskerefore a survey was conducted to ensure that a movieaffects the examined participant in the intended way equestionnaire was designed to examine the feelings andemotions experienced after seeing individual pieces ofmovies Each participant was asked the following 3questions about each movie

(1) Evaluate the emotions you experienced whilewatching a movie from 1 (unpleasant-low valence) to9 (pleasant-high valence)

(2) Rate the intensity of the emotions you experiencedwhile watching a movie from 1 (nonstimulating-lowarousal) to 9 (highly stimulating-high arousal)

2 Computational Intelligence and Neuroscience

(3) Choose the type of emotion you felt while watchingthe movie neutral happiness sadness anger sur-prise disgust and fear A participant also had theopportunity to describe hisher feelings

e survey among the participants allowed creating acircumplex model of emotions [46 47] In this model thehorizontal axis represents emotion valence whereas thevertical axis represents arousal Each emotion can be con-sidered as a linear combination of these dimensions Figure 1shows the distribution of emotions caused by each movie(numbered from 1 to 21) used in the experiment on thevalence-arousal plane e distribution is based on theanswers given by the participants in surveys

e distribution of emotions resembles the shape ofletter ldquoVrdquo this is because movies that are very pleasant orvery unpleasant are rated as stimulating at the same timeNeutral materials are generally rated as nonstimulating [48]ese tendencies are also reported in previous works[49 50] We created three classes of emotions from thedistribution of movies on the plane Each class included twomovies Class C1 covered videos that were rated in surveys asvery pleasant and very stimulating (high arousal and highvalence) Class C2 included neutral low stimulating movies(low arousal and moderate valence) Class C3 contained veryunpleasant and highly stimulating movies (high arousal andhigh valence) Only these 6 movies were further consideredin the experiment Table 1 presents the mean and medianvalues of valence and arousal assigned to the selected moviesby the participants Movies 2 and 12 presented dancingwomen with rhythmic music Movie 3 presented a birdrsquoseye view of the highway with calm music Movie 18 pre-sented the weather forecast Movie 13 included scenes ofviolence against a woman while movie 20 showed am-putation of a finger

22 Participants of the Experiment A specific group ofpeople selected based on age gender and education wereinvited to participate in the experiment Participants wererecruited through an advertisement on the website of Facultyof Electrical Engineering Warsaw University of TechnologyFinally 30 volunteers took part in the experiment Allparticipants were male third-year students with an averageage of 2125plusmn 074 years ey were informed about theoverall purpose and organization of the experiment andagreed to participate in it Volunteers were not paid forparticipating in the experiment

23 Experimental Procedure e experiment was carriedout in a specially prepared room and its temperature andlighting were the same for all participants e test standconsisted of a PC EyeTribe eye-tracker and two CreativeInspire T10 speakers e experiment always started in themorning between 700 and 800 AM e participants wereasked to take a comfortable position on the armchair thatsupported their hands and back in front of the computermonitor Next eye-tracker calibration was performed eperson supervising the experiment then left the room and

the participant alone watched the video materialemoviesshown had a soundtracke sound volume was identical forall participants We developed our own software responsiblefor controlling the course of the experiment which enabledsimultaneous display of movies and eye-tracking signalacquisition [51] e movie prepared for the experimentlasted 8min 10 s Different clips of the video material thatevoked different emotions were separated by a sequence of5 s of black screen and 5 s of colour animated curves thatresembled a screen saver e break between the videos wasintended to curtail the previous emotions and prepare theparticipant for the next part of the movie Each participantwatched the same video material with a random order of 21short movies associated with particular emotions Figure 2shows a typical sequence of the presented movies After thevideo material was played each participant had to answerthe questions in a survey on emotions caused by movies

24 Acquisition of Eye-Tracking Data e EyeTribe [52]optical eye-tracker was used to register oculographic signale EyeTribe calculates a pair of (X Y) coordinates of theuserrsquos gaze point with respect to a screen the user is lookingat e coordinates are expressed in pixels Accuracy ofcalculating userrsquos eye gaze coordinates is approximately 1degSuch an accuracy can be obtained after proper calibration ofthe device Calibration using 9 measurement points wasperformed for each participant e EyeTribe also providesinformation about the occurrence of eye fixations andmeasures the pupil diameter of both eyes e eye-trackingsignal was registered with a frequency of 60Hz

25 Signal Preprocessing Some of the participantsrsquo eyemovement information was lost for example duringblinking or during other eye or head movements Data froma participant were rejected completely when the number oflost samples was greater than 25us data recorded for 5participants were rejected Further analysis was performedon data recorded for the remaining 25 participants Sampleslost due to for example blinking were supplemented bylinear interpolation e signal was also filtered with 4thorder low-pass Butterworth filter with a cutoff frequencyequal to 5Hz Filtration was designed to eliminate high-frequency noise Figure 3(a) shows the recorded eye-tracking signal (X and Y coordinates of the gaze pointexpressed in pixels) (B) interpolated signal and (C) signalafter low-pass filtering

26 Feature Extraction e analysis of eye-tracking datainvolved calculating the features of the signal that dependson the emotional content of a movie Raw data points wereprocessed into eyemovements such as fixations and saccades[37] ere are many algorithms to calculate fixation thesealgorithms are based on the speed of eye movement (I-VT)and data scattering (I-DT) or are related to areas of interest(AOI-I) [53] e EyeTribe eye-tracker has a built-in algo-rithm that classifies a given sample of a registered signal andinforms one whether a given signal sample represents a

Computational Intelligence and Neuroscience 3

fixation Figure 4 shows a fragment of the registered visualcoordinates along with the marked fixations ere are 6fixations identified by the eye-tracker in the shown signalfragment

Features were calculated in time windows covering eachof the movie fragments Fixations were used to calculatefeatures such as the number of fixations mean value var-iance skewness and kurtosis of duration of fixations efixation durations were expressed as the number of samplesBased on fixations the overall fixation vector (OFV) was alsocalculated is feature considers the number position andduration of fixations It is described by the following [37 54]

OFV 1113944N

i1tivi (1)

where vi (xi minus xc yi minus yc) is a vector with an origin in thecenter of the screen (xc yc) and the end at the fixation point(xi yi) ti is the duration of the i-th fixation Pairs (xc yc)

and (xi yi) are coordinates expressed as pixels of thecenter of the screen and the fixation point respectivelyMean value variance skewness and kurtosis of amplitudesand the duration of the saccades were also calculated eduration of saccades was expressed as the number ofsamples while amplitude was expressed as pixels Somefeatures such as the number of fixations depend on the

duration of the movie for which they were calculated Tonormalize these features they were divided by the length ofthe movie expressed as the number of samples ereforethese features were not correlated with the length of theanalysed part of the movie

e pupil diameter enables the detection of changes inthe intensity of human emotions at a given moment[34 36] In addition to eye movement the EyeTribe eye-tracker allows one to measure the pupil diameter Meanvalue variance skewness and kurtosis of the pupil di-ameter which were measured during watching a givenmovie were calculated Table 2 lists the calculated featurese features were grouped according to the type of eyemovement

27Classification e classification was performed using 18eye-tracking features taken from 25 participants e fea-tures were calculated for a whole movie associated with onetype of emotion Each emotion was evoked by two differentmovies Features were standardized by centering them tohave a mean value equal to zero and by scaling them to havestandard deviation equal to one (z-score) for each partici-pant erefore for 25 participants the total number ofexamples related to one emotion was 50 (2 moviestimes 25volunteers) Hence to classify emotions in pairs and intriplets the number of examples was 100 and 150respectively

As classifiers we used linear discriminant analysis(LDA) support vector machine with a square kernel(Quadratic SVM) and the nearest neighbour classifier (k-NN) with k 11 [55]e parameters of individual classifierswere selected experimentally based on the accuracy of theclassification Classification was performed in the user-in-dependent mode Training dataset included examples from24 participants and testing dataset included examples from 1remaining participant so leave-one-out method was used fortests e division of the data into training and testing setswas repeated 25 times is enabled calculation of the av-erage classification accuracy Examples that were used in thetraining process were not used again during testing theclassifier

3 Results

Classification was performed for three pairs and for all threeclasses of emotions together Classification accuraciesachieved for each case are shown in Table 3 e highestaccuracy of 80 of the classification of three classes ofemotions was obtained using the SVM classifier Whenrecognizing emotions in pairs it turned out that the easiest isto distinguish between classes C1 and C2 On the other handthemost difficult is to distinguish between classes C2 and C3Tables 4ndash6 show the confusion matrixes for the classificationof all three classes for SVM LDA and k-NN classifiersrespectively

en for each classifier parameters precision (P) recall(R) specificity (S) and F1-score (F1) were calculated for allthree classes of emotions (Table 7)

Aro

usal

9

8

7

6

5

4

3

2

1 2 3 4 5 6 7Valence

8 91

C3

C1

C2

13

204

1710

515

7

916

183

114

11

6

19

8

21

122

Figure 1 Distribution of emotions caused by movies (numberedfrom 1 to 21) on the valence-arousal plane along with createdclasses (marked red)

Table 1 Evaluation of the selected movies

Class Videonumbers

Duration(s)

Arousal ValenceMean Median Mean Median

C1 2 10 73 8 81 912 13 72 75 79 85

C2 3 5 19 1 56 518 10 13 1 48 5

C3 13 10 66 65 28 220 10 64 6 33 25


Confusion matrixes analysis showed that the mostclassification errors appear for the C3 class is is alsoconfirmed by the values of precision recall and F1-score

To answer the question of which features associated witheye movement can be a reliable indicator of emotionsstatistical tests were performed One-way analysis of vari-ance (ANOVA) was used to determine whether featuresshow a significant difference between the three classesBefore the ANOVA test was performed it was checkedwhether the features satisfy normal distributions usingKolmogorovndashSmirnov test Distributions of the featureswere close to normal Bonferroni post-hoc test for evaluatingfeatures among the groups was also made Table 8 presentsthe results of the ANOVA and Bonferroni tests for eachfeature e Bonferroni test showed that 9 features are

suitable for distinguishing between classes C1 and C2 8features for distinguishing between C1 and C3 and only 5features for distinguishing between C2 and C3 is result isin line with previous observations that is the lowest clas-sification accuracy was obtained when distinguishing be-tween classes C2 and C3

Mean values and standard deviations of all features forthree classes of emotions are presented in Table 9

e participantrsquos pupil diameter is one of the statis-tically significant features e pupil diameter reached thesmallest value when experiencing neutral emotions (lowarousal) e average pupil diameters for C1 C2 and C3classes were 085 -074 and -043 (z-score standardized)respectively A similar trend was observed in otherstudies [28 29 34 36] Figure 5 shows the distribution of

Blac

ksc

reen

Dist

ract

or

Vid

eo cl

ipno

1

Time

Vid

eo cl

ipno

21

Emot

iona

lsta

tese

lf-as

sess

men

t

Break Emotion elicitation Break Emotion

elicitation

5s 5s 5s 5s5 ndash 717s 5 ndash 717s

Dist

ract

or

Blac

ksc

reen

Questionnaire

Figure 2 Typical sequence of the presented movies

1000

135 140 145Time (s)

X coordinateY coordinate

150 155

Pixe

ls

500

0

(a)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(b)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(c)

Figure 3 Preprocessing of eye-tracking signal (a) pure signal (b) signal after interpolation and (c) signal after low-pass filtering


the two features with the smallest p-values (average pupildiameter and average saccade amplitude) for the threeclasses of emotions

4 Discussion

e obtained accuracy of emotion classification indicatesthat it is possible to distinguish emotions using eye-tracking However when using eye-tracking and pupil

width some additional factors that may affect the reli-ability of the results should be considered e changes inthe pupil diameter are largely dependent on the lightingconditions and the luminance of the movie [29 56] eeffect of luminance is so large that it is impossible to usethe pupil diameter directly to recognize emotions withoutprior processing it We have assumed that pupil changesare partly due to changes in the luminance of the movieand partly due to the experienced emotions ereforethe effect caused by the luminance of the movie had to beremoved For this purpose the luminance of the moviewas assessed by calculating for each of its frames the Vcomponent in the HSV colour space e process ofremoving the movie luminance effect from the mea-surements of the pupil diameter is presented in Figure 6Figure 6(a) shows a fragment of the luminance of themovie Linear regression was used to model the rela-tionship between the pupil diameter and luminance of amovie For each participant a linear model of the in-fluence of luminance on the pupil diameter was calculated(coefficients b0 b1) using

y1

y2

⋮

yn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

1 x1

1 x2

⋮ ⋮

1 xn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

b0

b1

⎡⎣ ⎤⎦ (2)

Pixe

ls

1000

3311 2 3 4 5 6

332 333 334 335 336 337Time (s)

338 339 340

500

0

XY

Figure 4 A fragment of an eye-tracking (fixations saccades and coordinates of the ldquosightrdquo)

Table 2 List of calculated features

Signal type Features No

Fixation Number of fixations 1Overall fixation vector 2

Fixation duration

Mean 3Variance 4Skewness 5Kurtosis 6

Saccade amplitude


Saccade duration


Pupil diameter


Table 3 Classification accuracies

Classes SVM LDA k-NN MeanC1 vs C2 vs C3 080 073 065 073C1 vs C2 092 085 086 088C1 vs C3 083 084 075 081C2 vs C3 075 078 070 074

Table 4 Confusion matrix for SVM classifier

Predicted classesTrue classes

C1 C2 C3C1 45 3 7C2 1 42 10C3 4 5 33

Table 5 Confusion matrix for LDA classifier


C1 C2 C3C1 41 6 7C2 1 34 5C3 8 10 38

Table 6 Confusion matrix for k-NN classifier


C1 C2 C3C1 48 13 20C2 0 33 13C3 2 4 17


where y is the average pupil diameter of both eyes recordedfor a participant and x is the luminance value calculated for amovie frame

en the estimated value of the pupil diameter yest wascalculated from

yest b0 + b1x (3)

is caused the pupil diameter to be independent ofluminance changes ((4)) We assume that yemo depends onthe felt emotions

Table 7 Precision recall specificity and F1-score for all classifiers

ClassesSVM LDA k-NN

P R S F1 P R S F1 P R S F1C1 082 090 090 086 076 082 087 079 059 096 067 073C2 079 084 089 081 085 068 094 076 072 066 087 069C3 079 066 091 072 068 076 082 072 074 034 094 047Mean 080 080 090 080 076 075 088 076 068 065 083 063

Table 8 Results of ANOVA and Bonferroni post-hoc tests

FeatureANOVA Bonferroni multiple comparison test

p plt 005 C1 vs C2 p C1 vs C3 p C2 vs C3 p

Number of fixations 032 minus 100 100 040Overall fixation vector 064 minus 100 100 100Average duration of fixation 001 + 100 001 002Variance of fixation duration 006 minus 100 006 043Skewness of fixation duration 000 + 001 100 000Kurtosis of fixation duration 000 + 001 100 001Average saccade amplitude 000 + 000 065 000Variance of the saccade amplitudes 000 + 000 002 096Skewness of the saccade amplitudes 000 + 000 001 100Kurtosis of the saccade amplitudes 000 + 000 000 100Average duration of saccades 086 minus 100 100 100Variance of saccade durations 056 minus 100 089 100Skewness of saccade durations 000 + 000 001 099Kurtosis of saccade durations 000 + 000 000 071Average pupil diameter 000 + 000 000 035Variance of the pupil diameter 004 + 029 100 003Skewness of the pupil diameter 009 minus 100 023 012Kurtosis of the pupil diameter 005 minus 051 004 082

Table 9 Mean values and standard deviations of all features for three classes of emotions

No FeatureC1 C2 C3

Mean Std Mean Std Mean Std1 Number of fixations minus033 050 minus029 088 017 1082 Overall fixation vector minus031 047 minus016 105 009 0913 Average duration of fixation 011 095 minus050 085 018 1114 Variance of fixation duration 017 104 minus042 063 015 1145 Skewness of fixation duration 018 076 003 123 minus013 1006 Kurtosis of fixation duration minus031 048 100 100 minus011 0907 Average saccade amplitude minus033 051 048 088 027 1528 Variance of the saccade amplitudes minus037 060 017 096 017 0889 Skewness of the saccade amplitudes minus044 059 024 085 011 08610 Kurtosis of the saccade amplitudes 012 093 009 126 001 10711 Average duration of saccades 015 107 008 122 minus008 09712 Variance of saccade durations 031 089 minus036 069 minus020 08313 Skewness of saccade durations 033 103 minus043 054 minus024 07714 Kurtosis of saccade durations 020 074 004 093 017 09815 Average pupil diameter 085 083 minus073 117 minus043 08316 Variance of the pupil diameter minus004 068 minus031 069 011 10417 Skewness of the pupil diameter minus006 066 minus001 072 minus035 10618 Kurtosis of the pupil diameter 009 066 minus013 081 minus031 089


yemo y minus yest (4)

Figure 6(b) shows the registered pupil diameter value forparticipant S06 and its estimated (as a response for movieluminance) value yest 2984 minus 1123x Figure 6(c) showsthe pupil diameter of the same participant after removing theeffect of luminance e mean correlation between the pupildiameter and luminance for all volunteers was minus049 isvalue was reduced to 0006 after subtracting the estimatedluminance effect

We also carried out additional tests to assess whether themethod of removing the influence of luminance is suffi-ciently effective First we tested the approach taken from aprior publication [23] in which neighbourhood of the gazeposition was used to calculate the luminance instead of theaverage luminance of the whole frame e neighbourhoodhad a width of 2 degrees (of visual angle) around the gazepoint which corresponds to a circle with a diameter of 68pixels (distance from the monitor 60 cm resolution1920times1080 pixels) Sample frames from movies withmarked gaze positions are shown in Figure 7

Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3

35325215105Average saccade amplitude

C1C2C3

0ndash05ndash1ndash15

Figure 5 e distribution of the average pupil diameter and the average saccade amplitude for the three classes of emotions

08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)

Acquired pupil diameterEstimated pupil response

50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)

Figure 6 Removing the effect of movie luminance (a) calculated luminance of a movie (b) blue line registered pupil diameter for aparticipant red line estimated pupil diameter (as a response for movie luminance) (c) pupil diameter after removing the luminance effect


e average framersquos luminance was 04 for the leftframe and 019 for the right one e luminance in theneighbourhood of the gaze positions was 008 and 071respectively In both cases the average luminance of theframe and the luminance of the neighbourhood of thegaze positions are significantly different from each otherIn the frame on the left the participant looked at one ofthe darker points while in the frame on the right helooked at a bright point One however cannot ignore theluminance of the rest of the frame because it is alsoimportant especially when there is no other light sourceduring the presentation of the movies or when the light inthe room is very dim Figure 8 shows the changes in thepupil diameter as determined using both of the above-mentioned methods e correlation between the pupildiameter determined by the average luminance of theentire frame and using the luminance in the neigh-bourhood of the gaze position was 087 A very strongcorrelation indicates that both methods work similarlyFor each participant the correlation of the mean pupildiameter for each movie related to one emotion with theaverage value of the movie luminance (ie without re-moving the influence of luminance) was also calculatede same correlations were calculated after removing theinfluence of luminance (by using the average luminanceof the entire frame) e median of correlation valuescalculated for all participants was minus084 in the first caseand minus030 in the second case e large difference in thesevalues indicates that the method proposed by us forremoving the effect of luminance is sufficiently effective

In [28 30 38] attempts were made to use eye-tracking to recognize emotions caused by moviesHowever these works did not consider the possibleimpact of movie dynamics on eye-tracking features efeatures discussed in this article which are associatedwith eye movements may also be related to factors otherthan emotions Hence we tested the relationship of theeye-tracking features with the dynamics of the presentedmovies Movies can be characterized by frequent or rarescene changes and moving or static objects Eye move-ments such as fixation and saccades depend largely on thedynamics of the movie scene is impact cannot beignored when analyzing the signal for emotion

recognition We calculated a D index describing thedynamics of the video clip

Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)

where F is the matrix representing video frame of size N times

M in grayscale and i is the frame number Figure 9 depictsthe variation of the D index for each of the movies shownus each movie was rated for the speed at which scenes orobjects changed between frames

e obtained classification results indicate that the mostdifficult emotions to distinguish were negative highlystimulating emotions from neutral low stimulating ones(class C2 vs C3) ese two classes contain two pairs ofmovies 3 and 18 with 13 and 20 According to thecalculated dynamics index these movies are characterizedby small dynamics eir dynamics are definitely smallerthan those of movies 2 and 12 in class C1 e abovetendency may indicate that the classification result to someextent also depends onmovie dynamics and not just only onits emotional content e distinction between the twoclasses of movies of similar dynamics seems to be moredifficult than that between the movies of different dynamicsTo test the effect of video dynamics on the obtained results acorrelation of the coefficient D with the used features av-eraged for all participants was calculated e correlationsare shown in Table 10 We obtained the highest values of theaverage correlation (ge04) for the following features1mdashnumber of fixations 13mdashskewness of the saccadedurations and 15mdashaverage pupil diameter e ANOVAtest showed that the average pupil diameter and skewness ofthe saccade durations were identified as statistically signif-icant for emotion recognition (Table 8) is may indicate arelationship between the effectiveness of emotion recogni-tion and movie dynamics e correlation between theparticipantsrsquo ratings of movies used in the experiment withtheir dynamics was 045 for the arousal ese values arestatistically significant and may indicate that stimulatingvideos are characterized by high dynamics (2 12 15 19and 21) and nonstimulating videos by low dynamics (13 14 and 18) Dependence of arousal parameter on themovie dynamics is shown in Figure 10

(a) (b)

Figure 7 Two sample frames from movies with one participantrsquos gaze positions


Our database contains only a few specific movies that arestimulating and not dynamicese are for example moviesshowing finger amputation (movie 20) or tooth extraction(movie 10) In these movies the camera does not change itsposition and the scenes are static erefore the dynamicindex for these movies is low ese movies are rated asstimulating because they cause strong negative emotionsrelated to disgust Movies 4 and 17 depict horror scenesAfter a few static scenes (low dynamics) terrible charactersappear in these movies thereby causing high arousal

e weakness of the work is the possible impact of moviedynamics on the results achieved However when recog-nizing the emotions evoked by movies with similar dy-namics a satisfactory classification accuracy of 78 wasachieved using the LDA classifier In the future it is worthconducting additional research on the impact of moviedynamics

e obtained results were compared with the resultsfrom other publications in which eye-tracking was usedHowever it should be borne in mind that accurate com-parison is difficult because each study used a different way ofevoking emotions a different set of eye-tracking featuresand different research groupe comparison is presented inTable 11 e conducted comparison indicates that the re-sults obtained by us are slightly better than those in similarworks e choice of a narrow group taking part in the

Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)

Pupil diameter using average frame luminancePupil diameter using gaze point neighborhood

3020100

Figure 8 Changes in pupil eye diameter of one of the participantsafter removing the influence of luminance calculated using twomethods

Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0

Figure 9 Movie dynamics index

Table 10 Correlation of the coefficient D with the used featuresaveraged for all participants

No Feature Movie dynamics1 Number of fixations 0432 Overall fixation vector minus0343 Average duration of fixation minus0324 Variation of fixation durations minus0385 Skewness of fixation durations 0286 Kurtosis of fixation durations 0307 Average amplitude of the saccades minus0358 Variation of saccade amplitudes minus0309 Skewness of saccade amplitudes minus01010 Kurtosis of saccade amplitudes minus01711 Average duration of saccades 00712 Variance of the saccade durations 01013 Skewness of the saccade durations 04014 Kurtosis of the saccade durations 03915 Average pupil diameter 06216 Pupil diameter variance 01317 Pupil diameter skewness 01618 Pupil diameter kurtosis 025

8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal

Figure 10 Dependence of arousal parameter on the moviedynamics


experiment (young men) may affect the result It is possiblethat in such a coherent group of people emotions wereevoked more precisely

5 Conclusions

e classification results confirm the ability of recognizingemotions using eyemovement features and pupil diametereuse of an eye-tracking requires however the elimination offactors that may affect this classification ese factors includethe effect of luminance on changes in the pupil diameter Itshould be ensured that the lighting conditions remain the samethroughout the experiment e influence of luminance of thepresented materials should also be compensated is could beachieved by implementing appropriate methods such as re-gression or principal component analysis

Another factor that should be considered is the dy-namics of the presented material It seems that the dy-namics of the movie can affect the accuracy of the emotionclassification using the eye-tracking features On the otherhand the dynamics of movie is related in some way to feltemotions Research shows that high-dynamic movies havea stimulating effect When recognizing the emotionsevoked by movies with similar dynamics a satisfactoryclassification accuracy of 78 using LDA was achievedDuring recognition of three classes of emotions we ob-tained the maximum classification accuracy of 80 It isworth emphasizing that all results were obtained using theleave-one-subject-out validation method is implies thatthe presented user-independent method of emotion clas-sification based on eye-tracking features can be success-fully used in practice

Data Availability

e data used to support the findings of this study areavailable from the corresponding author upon request

Conflicts of Interest

e authors declare that there are no conflicts of interestregarding the publication of this paper

Acknowledgments

is work was supported by the statutory activity of Elec-trical Faculty at Warsaw University of Technology underDeanrsquos Grant in 2017

References

[1] E Bal E Harden D Lamb A V Van Hecke J W Denverand S W Porges ldquoEmotion recognition in children withautism spectrum disorders relations to eye gaze and auto-nomic staterdquo Journal of Autism and Developmental Disordersvol 40 no 3 pp 358ndash370 2010

[2] S B Daily M T James D Cherry et al ldquoAffective com-puting historical foundations current applications and fu-ture trendsrdquo in Emotions and Affect in Human Factors andHuman-Computer Interaction pp 213ndash231 Elsevier Aca-demic Press San Diego CA USA 2017

[3] K Karpouzis and G N Yannakakis Emotion in GamesVol 4 Springer International Publishing Cham Switzerland2016

[4] C Holmgard G N Yannakakis K-I Karstoft andH S Andersen ldquoStress detection for PTSD via the StartleMartgamerdquo in Proceedings of the 2013 Humaine AssociationConference on Affective Computing and Intelligent Interactionpp 523ndash528 Geneva Switzerland September 2013

[5] A Pampouchidou P G Simos K Marias et al ldquoAutomaticassessment of depression based on visual cues a systematicreviewrdquo IEEE Transactions on Affective Computing vol 10no 4 pp 445ndash470 2019

[6] B Li ldquoA facial affect analysis system for autism spectrumdisorderrdquo in Proceedings of the 2019 IEEE InternationalConference on Image Processing (ICIP) pp 4549ndash4553 TaipeiTaiwan September 2019

[7] S Sarabadani L C Schudlo A-A Samadani and A KushkildquoPhysiological detection of affective states in children withautism spectrum disorderrdquo IEEE Transactions on AffectiveComputing p 1 2018

[8] L Cen F Wu Z L Yu and F Hu ldquoA real-ime speechemotion recognition system and its application in onlinelearningrdquo in Emotions Technology Design and LearningS Y Tettegah and M Gartmeier Eds pp 27ndash46 AcademicPress San Diego DA USA 2016

[9] S S S Darnell ldquoEngageMErdquo in Proceedings of the ExtendedAbstracts of the 32nd Annual ACM Conference on Human

Table 11 Comparison of our results with other studies

Works Classes User-independentmethod

Accuracy()

Our research 3 (high arousal and high valence high arousal and low valence low arousal andmoderate valence) + 8000

Soleymani et al[57] 3 (calm medium aroused and activated) + 7110

Soleymani et al[57] 3 (unpleasant neutral and pleasant) + 6660

Zheng et al [30] 4 (sad feeling fear happy and neutral) minus 6782Lu et al [28] 3 (positive neutral and negative) minus 7780Soleymani et al[38] 3 (calm medium aroused and excited) + 6350


Zhao et al [31] 5 (happy sad feeling fear disgusted and neutral) minus 5981


Factors in Computing Systems - CHI EA lsquo14 pp 359ndash362Toronto Canada 2014

[10] M Soleymani M Larson T Pun and A Hanjalic ldquoCorpusdevelopment for affective video indexingrdquo 2012 httpsarxivorgabs12115492

[11] Y Ding X Hu Z Xia Y-J Liu and D Zhang ldquoInter-brainEEG feature extraction and analysis for continuous implicitemotion tagging during video watchingrdquo IEEE Transactionson Affective Computing p 1 2018

[12] M Soleymani S Asghari-Esfeden Y Fu and M PanticldquoAnalysis of EEG signals and facial expressions for continuousemotion detectionrdquo IEEE Transactions on Affective Com-puting vol 7 no 1 pp 17ndash28 2016

[13] S Koelstra and I Patras ldquoFusion of facial expressions andEEG for implicit affective taggingrdquo Image and Vision Com-puting vol 31 no 2 pp 164ndash174 2013

[14] P Tarnowski M Kołodziej A Majkowski and R J RakldquoEmotion recognition using facial expressionsrdquo ProcediaComputer Science vol 108 pp 1175ndash1184 2017

[15] P Tarnowski M Kolodziej A Majkowski and R J RakldquoldquoAnthropometric facial features in emotion recognitionrdquo inProceedings of 2018 19th International Conference Computa-tional Problems of Electrical Engineering CPEE 2018 BanskaStiavnica Slovakia September 2018

[16] W Dai D Han Y Dai and D Xu ldquoEmotion recognition andaffective computing on vocal social mediardquo Information ampManagement vol 52 no 7 pp 777ndash788 2015

[17] A Majkowski M Kolodziej R J Rak and R KorczyήskildquoClassification of emotions from speech signalrdquo in Proceed-ings of the 2016 Signal Processing Algorithms ArchitecturesArrangements and Applications (SPA) pp 276ndash281 PoznanPoland September 2016

[18] J Han X Ji X Hu L Guo and T Liu ldquoArousal recognitionusing audio-visual features and FMRI-based brain responserdquoIEEE Transactions on Affective Computing vol 6 no 4pp 337ndash347 2015

[19] A Goshvarpour A Abbasi and A Goshvarpour ldquoAn ac-curate emotion recognition system using ECG and GSRsignals and matching pursuit methodrdquo Biomedical Journalvol 40 no 6 pp 355ndash368 2017

[20] P Das A Khasnobish and D N Tibarewala ldquoEmotionrecognition employing ECG and GSR signals as markers ofANSrdquo in Proceedings of the 2016 Conference on Advances inSignal Processing (CASP) pp 37ndash42 Pune India June 2016

[21] Y-L Hsu J-S Wang W-C Chiang and C-H HungldquoAutomatic ECG-based emotion recognition in music lis-teningrdquo IEEE Transactions on Affective Computing vol 11no 1 pp 85ndash99 2017

[22] V Kehri R Ingle S Patil and R N Awale ldquoAnalysis of facialEMG signal for emotion recognition using wavelet packettransform and SVMrdquo in Proceedings of the Machine Intelli-gence and Signal Analysis pp 247ndash257 Singapore 2019

[23] X Sun T Hong C Li and F Ren ldquoHybrid spatiotemporalmodels for sentiment classification via galvanic skin re-sponserdquo Neurocomputing vol 358 pp 385ndash400 2019

[24] S Hassani I Bafadel A Bekhatro E Al Blooshi S Ahmedand M Alahmad ldquoPhysiological signal-based emotion rec-ognition systemrdquo in Proceedings of the 2017 4th IEEE Inter-national Conference on Engineering Technologies and AppliedSciences (ICETAS) pp 1ndash5 Salmabad Bahrain November2017

[25] R Rakshit V R Reddy and P Deshpande ldquoEmotion de-tection and recognition using HRV features derived fromphotoplethysmogram signalsrdquo in Proceedings of the 2nd

workshop on Emotion Representations and Modelling forCompanion Systems pp 1ndash6 Tokyo Japan November 2016

[26] J A Domınguez-Jimenez K C Campo-LandinesJ C Martınez-Santos E J Delahoz and S H Contreras-Ortiz ldquoA machine learning model for emotion recognitionfrom physiological signalsrdquo Biomedical Signal Processing andControl vol 55 Article ID 101646 2020

[27] L Mirmohamadsadeghi A Yazdani and J-M Vesin ldquoUsingcardio-respiratory signals to recognize emotions elicited bywatching music video clipsrdquo in Proceedings of the 2016 IEEE18th International Workshop on Multimedia Signal Processing(MMSP) Montreal Canada September 2016

[28] Y Lu W-L Zheng B Li and B-L Lu ldquoCombining eyemovements and EEG to enhance emotion recognitionrdquo inProceedings of the 24th International Conference on ArtificialIntelligence pp 1170ndash1176 Buenos Aires Argentina 2015

[29] W L Zheng B N Dong and B L Lu ldquoMultimodal emotionrecognition using EEG and eye tracking datardquo in Proceedings ofthe 2014 36th Annual International Conference of the IEEEEngineering in Medicine and Biology Society pp 5040ndash5043Chicago IL USA August 2014

[30] W-L Zheng W Liu Y Lu B-L Lu and A CichockildquoEmotionMeter a multimodal framework for recognizinghuman emotionsrdquo IEEE Transactions on Cybernetics vol 49no 3 pp 1110ndash1122 2019

[31] L-M Zhao R Li W-L Zheng and B-L Lu ldquoClassificationof five emotions from EEG and eye movement signalscomplementary representation propertiesrdquo in Proceedings ofthe 2019 9th International IEEEEMBS Conference on NeuralEngineering (NER) pp 611ndash614 San Francisco CA USAMarch 2019

[32] J Z Lim J Mountstephens and J Teo ldquoEmotion recognitionusing eye-tracking taxonomy review and current challengesrdquoSensors vol 20 no 8 p 2384 2020

[33] T Zhang A El Ali C Wang X Zhu and P Cesar ldquoCorrFeatcorrelation-based feature extraction algorithm using skinconductance and pupil diameter for emotion recognitionrdquo inProceedings of the 2019 International Conference on Multi-modal Interaction pp 404ndash408 Suzhou China October 2019

[34] M M Bradley L Miccoli M A Escrig and P J Lang ldquoepupil as a measure of emotional arousal and autonomic ac-tivationrdquo Psychophysiology vol 45 no 4 pp 602ndash607 2008

[35] P Raiturkar A Kleinsmith A Keil A Banerjee and E JainldquoDecoupling light reflex from pupillary dilation to measureemotional arousal in videosrdquo in Proceedings of the ACMSymposium on Applied Perception pp 89ndash96 Anaheim CAUSA July 2016

[36] T Partala and V Surakka ldquoPupil size variation as an indi-cation of affective processingrdquo International Journal of Hu-man-Computer Studies vol 59 no 1-2 pp 185ndash198 2003

[37] K Holmqvist M Nystrom R Anderson R DewhurstH Jarodzka and J van der Weijer Eye Tracking A Com-prehensive Guide toMethods andMeasures Oxford UniversityPress New York NY USA 2011

[38] M Soleymani J Lichtenauer T Pun and M Pantic ldquoAmultimodal database for affect recognition and implicittaggingrdquo IEEE Transactions on Affective Computing vol 3no 1 pp 42ndash55 2012

[39] E H Hess and J M Polt ldquoPupil size as related to interest valueof visual stimulirdquo Science vol 132 no 3423 pp 349-3501960

[40] W L Libby B C Lacey and J I Lacey ldquoPupillary and cardiacactivity during visual attentionrdquo Psychophysiology vol 10no 3 pp 270ndash294 1973


[41] A Alhargan N Cooke and T Binjammaz ldquoAffect recog-nition in an interactive gaming environment using eyetrackingrdquo in Proceedings of the 2017 Seventh InternationalConference on Affective Computing and Intelligent Interaction(ACII) pp 285ndash291 San Antonio TX USA October 2017

[42] S Koelstra C Muhl M Soleymani et al ldquoDEAP a databasefor emotion analysis using physiological signalsrdquo IEEETransactions on Affective Computing vol 3 no 1 pp 18ndash312012

[43] A G Money and H Agius ldquoAnalysing user physiologicalresponses for affective video summarisationrdquo Displaysvol 30 no 2 pp 59ndash70 2009

[44] M Bradley and P Lang ldquoe international affective digitizedsounds (2nd edition IADS-2) affective ratings of sounds andinstruction manualrdquo Technical report B-3 University ofFlorida Gainesville FL USA 2007

[45] P Ekman W V Friesen M OrsquoSullivan et al ldquoUniversals andcultural differences in the judgments of facial expressions ofemotionrdquo Journal of Personality and Social Psychology vol 53no 4 pp 712ndash717 Oct 1987

[46] J Posner J A Russell and B S Peterson ldquoe circumplexmodel of affect an integrative approach to affective neuro-science cognitive development and psychopathologyrdquo De-velopment and Psychopathology vol 17 no 3 pp 715ndash7342005

[47] Y Nakayama Y Takano M Matsubara K Suzuki andH Terasawa ldquoe sound of smile auditory biofeedback offacial EMG activityrdquo Displays vol 47 pp 32ndash39 2017

[48] J A Coan and J J B Allen Handbook of Emotion Elicitationand Assessment Oxford University Press New York NYUSA 2007

[49] R Dietz and A Lang ldquoAEligffective agents - Dietz amp Langrdquo inProceedings of the ird International Cognitive TechnologyConference San Francisco CA USA 1999

[50] M Horvat M Vukovic and Z Car ldquoEvaluation of keywordsearch in affective multimedia databasesrdquoldquoEvaluation ofkeyword search in affective multimedia databasesrdquo inTransactions on Computational Collective Intelligence XXISpecial Issue on Keyword Search and Big Data N T NguyenR Kowalczyk and P Rupino da Cunha Eds pp 50ndash68Springer Berlin Heidelberg Berlin Heidelberg 2016

[51] P Tarnowski M Kołodziej A Majkowski and R J Rak ldquoAsystem for synchronous acquisition of selected physiologicalsignals aimed at emotion recognitionrdquo Przeglad Elek-trotechniczny vol 1 no 12 pp 329ndash333 2016

[52] ldquoBasics eyetribe-docsrdquo httpstheeyetribecomdevtheeyetribecomdevtheeyetribecomgeneralindexhtml

[53] D D Salvucci and J H Goldberg ldquoIdentifying fixations andsaccades in eye-tracking protocolsrdquo in Proceedings of the 2000Symposium on Eye Tracking Research amp Applicationspp 71ndash78 New York NY USA 2000

[54] C-F Chi and F-T Lin ldquoA new method for describing searchpatterns and quantifying visual load using eye movementdatardquo International Journal of Industrial Ergonomics vol 19no 3 pp 249ndash257 1997

[55] M R Ogiela and R Tadeusiewicz ldquoModern computationalintelligence methods for the interpretation of medical imagesin Studies in Computational Intelligencerdquo vol 84 SpringerBerlin Germany 2008

[56] C J Ellis ldquoe pupillary light reflex in normal subjectsrdquoBritish Journal of Ophthalmology vol 65 no 11 pp 754ndash7591981

[57] M Soleymani M Pantic and T Pun ldquoMultimodal emotionrecognition in response to videosrdquo IEEE Transactions onAffective Computing vol 3 no 2 pp 211ndash223 2012


parameters are subject to testing In the case of brain activitysignals from the central nervous system are recorded usingelectroencephalography (EEG) [12 14 15] as well asmedical magnetic resonance imaging (fMRI) [18] Otherphysiological signals used include electrocardiography(ECG) [19ndash21] electromyography (EMG) [22] electroder-mal activity (EDA) [19 20 23] heart rate [24ndash26] respi-ration rate and depth [24 27] and arterial pressure [24]Eye-tracking [28ndash32] and pupil width [33ndash36] are also usedto recognize emotions

Interest in eye movement research dates back to thenineteenth century when the method of direct observationwas useden invasive eye-trackers were constructed usinga special type of lens placed on the eyeball At presentnoncontact devices which use infrared cameras are mostcommonly used for eye-tracking e development of suchdevices increased the popularity of eye movement researchamong scientists from many different fields Eye-trackingresearch is applied in areas such as cognitive psychologyneuropsychology usability testing or marketing [37] eeyeball movement is examined while viewing the presentedmaterials eg photos or movies and while reading playingbrowsing the web or using graphical user interfaces Fewattempts have also been made to use eye-tracking to rec-ognize emotions [32] In [28] features associated with eyemovements were used to recognize three classes of emotionse classification accuracy obtained was 778 In [38] forthree classes of emotions recognized using eye-trackingmethods the classification accuracy was 688 In [30] theuse of features such as fixation duration amplitude andduration of saccades and blinking frequency for classifi-cation of four classes of emotions was described achieving6782 accuracy In all mentioned studies emotions wereevoked by presenting movies

e eye-tracking often includes measuring the diameterof the pupil which is responsible for regulating the amountof light falling on the retina In addition to light intensity thepupil response is also influenced by other factors esefactors include the emotions experienced by oneself [39 40]To date studies have shown that the pupil diameter increaseswhen feeling excited [34 36] In [29] an attempt to rec-ognize three classes of emotions (positive negative andneutral) evoked by movies using pupil width was presentedClassification accuracy of 589was achievede study [41]describes research on recognizing emotions caused byplaying a video game A set of features related to pupil widthwere used for the classification For three classes of emo-tions the classification accuracy achieved was 76 and614 respectively for the arousal and the valence scalese work [35] presents the use of pupillometry to measureemotional arousal in videos

13 e Aim of is Paper e purpose of the presentedresearch is to analyze whether it is possible to recognizeemotions by using eye-tracking signal Emotions have beencaused by the presentation of video material For the presentresearch an experiment was designed during which a groupof 30 participants watched a set of 21 fragments of movies

intended to evoke emotions During movie presentationeye-tracking data were recorded As features we used eye-movement specific elements such as fixations and saccadesFeatures related to the pupil diameter were also calculatedWe recognized three classes of emotions high arousal andlow valence low arousal and moderate valence and higharousal and high valence ree classifiers were tested SVMLDA and k-NN e leave-one-subject-out method wasused to assess the quality of the classification Individualsuccessive stages of the research were the following

(i) acquisition of eye-tracking data while evokingemotions

(ii) performing signal preprocessing(iii) removal of the effect of luminance on pupil width(iv) calculation of eye-tracking features related to eye

movements and pupil width(v) classification of emotions using SVM LDA and k-

NN

Videomaterial is characterized by fast scene changes andmoving objects e dynamics of the movies can greatlyinfluence the eye-tracking features and the possibility to usethem for emotion recognition Innovative in our research isthat we examined the effects of the dynamics of movies onclassification accuracy Further when measuring the pupildiameter the impact of luminance and lighting conditions isvery important e effect of luminance on pupil width wasremoved using linear regression

2 Materials and Methods

21 Evoking Emotions An important element of the studywas evoking emotions in participants of the experimentEmotions are most often caused by the presentation ofvisual stimuli (images and movies) [34 38 42 43] andsound stimuli (music and sounds) [36 44] In our ex-periment emotions were evoked by a video presentationon a monitor screen e presented material comprisedshort movies along with the soundtrack Twenty-onemovies were used in the experiment ey elicited sixbasic emotions as defined in [45] that is happinesssadness anger surprise disgust and fear Several othermovies were selected to be as neutral as possible and didnot cause any of the above emotions e selection ofmovies that cause a specific emotion is a difficult taskerefore a survey was conducted to ensure that a movieaffects the examined participant in the intended way equestionnaire was designed to examine the feelings andemotions experienced after seeing individual pieces ofmovies Each participant was asked the following 3questions about each movie

(1) Evaluate the emotions you experienced whilewatching a movie from 1 (unpleasant-low valence) to9 (pleasant-high valence)

(2) Rate the intensity of the emotions you experiencedwhile watching a movie from 1 (nonstimulating-lowarousal) to 9 (highly stimulating-high arousal)














OFV 1113944N

i1tivi (1)







3 Results



Aro

usal

9

8

7

6

5

4

3

2

1 2 3 4 5 6 7Valence

8 91

C3

C1

C2

13

204

1710

515

7

916

183

114

11

6

19

8

21

122



Class Videonumbers

Duration(s)


C1 2 10 73 8 81 912 13 72 75 79 85

C2 3 5 19 1 56 518 10 13 1 48 5

C3 13 10 66 65 28 220 10 64 6 33 25







Blac

ksc

reen

Dist

ract

or

Vid

eo cl

ipno

1

Time

Vid

eo cl

ipno

21

Emot

iona

lsta

tese

lf-as

sess

men

t


elicitation


Dist

ract

or

Blac

ksc

reen

Questionnaire


1000

135 140 145Time (s)


150 155

Pixe

ls

500

0

(a)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(b)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(c)




4 Discussion



y1

y2

⋮

yn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

1 x1

1 x2

⋮ ⋮

1 xn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

b0

b1

⎡⎣ ⎤⎦ (2)

Pixe

ls

1000

3311 2 3 4 5 6

332 333 334 335 336 337Time (s)

338 339 340

500

0

XY





Fixation duration


Saccade amplitude


Saccade duration


Pupil diameter






C1 C2 C3C1 45 3 7C2 1 42 10C3 4 5 33



C1 C2 C3C1 41 6 7C2 1 34 5C3 8 10 38



C1 C2 C3C1 48 13 20C2 0 33 13C3 2 4 17




yest b0 + b1x (3)



ClassesSVM LDA k-NN







No FeatureC1 C2 C3






Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()








































































OFV 1113944N

i1tivi (1)







3 Results



Aro

usal

9

8

7

6

5

4

3

2

1 2 3 4 5 6 7Valence

8 91

C3

C1

C2

13

204

1710

515

7

916

183

114

11

6

19

8

21

122



Class Videonumbers

Duration(s)


C1 2 10 73 8 81 912 13 72 75 79 85

C2 3 5 19 1 56 518 10 13 1 48 5

C3 13 10 66 65 28 220 10 64 6 33 25







Blac

ksc

reen

Dist

ract

or

Vid

eo cl

ipno

1

Time

Vid

eo cl

ipno

21

Emot

iona

lsta

tese

lf-as

sess

men

t


elicitation


Dist

ract

or

Blac

ksc

reen

Questionnaire


1000

135 140 145Time (s)


150 155

Pixe

ls

500

0

(a)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(b)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(c)




4 Discussion



y1

y2

⋮

yn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

1 x1

1 x2

⋮ ⋮

1 xn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

b0

b1

⎡⎣ ⎤⎦ (2)

Pixe

ls

1000

3311 2 3 4 5 6

332 333 334 335 336 337Time (s)

338 339 340

500

0

XY





Fixation duration


Saccade amplitude


Saccade duration


Pupil diameter






C1 C2 C3C1 45 3 7C2 1 42 10C3 4 5 33



C1 C2 C3C1 41 6 7C2 1 34 5C3 8 10 38



C1 C2 C3C1 48 13 20C2 0 33 13C3 2 4 17




yest b0 + b1x (3)



ClassesSVM LDA k-NN







No FeatureC1 C2 C3






Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()

































































Blac

ksc

reen

Dist

ract

or

Vid

eo cl

ipno

1

Time

Vid

eo cl

ipno

21

Emot

iona

lsta

tese

lf-as

sess

men

t


elicitation


Dist

ract

or

Blac

ksc

reen

Questionnaire


1000

135 140 145Time (s)


150 155

Pixe

ls

500

0

(a)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(b)

135 140 145Time (s)

150 155

140012001000

800600400200

Pixe

ls


(c)




4 Discussion



y1

y2

⋮

yn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

1 x1

1 x2

⋮ ⋮

1 xn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

b0

b1

⎡⎣ ⎤⎦ (2)

Pixe

ls

1000

3311 2 3 4 5 6

332 333 334 335 336 337Time (s)

338 339 340

500

0

XY





Fixation duration


Saccade amplitude


Saccade duration


Pupil diameter






C1 C2 C3C1 45 3 7C2 1 42 10C3 4 5 33



C1 C2 C3C1 41 6 7C2 1 34 5C3 8 10 38



C1 C2 C3C1 48 13 20C2 0 33 13C3 2 4 17




yest b0 + b1x (3)



ClassesSVM LDA k-NN







No FeatureC1 C2 C3






Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()





























































4 Discussion



y1

y2

⋮

yn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

1 x1

1 x2

⋮ ⋮

1 xn

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦

b0

b1

⎡⎣ ⎤⎦ (2)

Pixe

ls

1000

3311 2 3 4 5 6

332 333 334 335 336 337Time (s)

338 339 340

500

0

XY





Fixation duration


Saccade amplitude


Saccade duration


Pupil diameter






C1 C2 C3C1 45 3 7C2 1 42 10C3 4 5 33



C1 C2 C3C1 41 6 7C2 1 34 5C3 8 10 38



C1 C2 C3C1 48 13 20C2 0 33 13C3 2 4 17




yest b0 + b1x (3)



ClassesSVM LDA k-NN







No FeatureC1 C2 C3






Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()






























































yest b0 + b1x (3)



ClassesSVM LDA k-NN







No FeatureC1 C2 C3






Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()































































Aver

age p

upil

diam

eter

ndash3

ndash2

ndash1

0

1

2

3


C1C2C3



08

0 10 20 30 40Time (s)

50 60 70 80 90

Lum

inan

ce

060402

0

(a)

0 10 20 30 40Time (s)


50 60 70 80 90

35

Lum

inan

ce

30

25

20

(b)

0 10 20 30 40Time (s)

50 60 70 80 90

10

5

0

ndash5Pupi

l dia

met

er

(c)






Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()































































Di 1113944N

j

1113944

M

k

Fi+1 minus Fi( 1113857 (5)




(a) (b)






Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()































































Pupi

l dia

met

er

10

8

6

4

2

0

ndash2

ndash4

ndash6

ndash8100908070605040

Time (s)


3020100


Var (D

)

2

C1C2C3

18

times1014

16

14

12

1

08

06

1 2 3 4 5 6 7 8 9 10Number of videos

11 12 13 14 15 16 17 18 19 20 21

04

02

0




8

7

6

5

183

114

11

9

7

58

4

101720

132

21

19 15

12

616

4

0 02 04 06 08Var (D)

1 12 14 16 18 2times1014

3

2

1

Aro

usal




5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()





























































5 Conclusions



Data Availability




Acknowledgments


References












Accuracy()




























































Research Article...

Documents

Transcript of Research Article...