CogniMeter: EEG-Based Brain StatesMonitoring

Xiyuan Hou1(B), Yisi Liu1, Wei Lun Lim2, Zirui Lan2, Olga Sourina1,Wolfgang Mueller-Wittig1, and Lipo Wang2

1 Fraunhofer IDM@NTU, Nanyang Technological University, Singapore, Singapore{houxy,liuys,eosourina,askwmwittig}@ntu.edu.sg

2 School of Electrical and Electronic Engineering,Nanyang Technological University, Singapore, Singapore{wllim2,lanz0001}@e.ntu.edu.sg, elpwang@ntu.edu.sg

Abstract. Electroencephalogram (EEG) techniques are traditionallyused in the medical field. Recent research work focuses on applying thesetechniques to daily life with wireless and relatively low-price EEG devicesavailable in the market. As a result, applications such as neurofeedbacktraining, neuromarketing, emotion, stress, mental workload recognition,etc. using EEG techniques on healthy adults have been developed. Sincethe EEG measures and records electrical activity in the brain, it is possiblefor it to reflect a person’s brain states. In this paper, we describe a novelbrain computer interface calledCogniMeter integrated with proposed real-time emotion, mental workload, and stress recognition algorithms. Withthis system, we can assess human emotions, mental workload, and stress inreal time. This work can be applied as a human study tool in many fields.For example, the wellbeing of users within a system or workers in industrycan be monitored to improve their protection from overly stressful work-load conditions. In research, brain state monitoring can be applied in sim-ulation scenarios during human factor study experiments. In marketing,a person’s emotional response toward products or advertisements can bestudied using EEG-based brain states monitoring.

Keywords: Visual interface · EEG · Stress · Mental workload · Emo-tions

1 Introduction

With an increasing number of wearable devices (Apple Watch, Fitbit, and Jow-bone) becoming available in the commercial market, it becomes popular to usesuch wearable devices and mobile phones to monitor our daily physiologicalstates based on number of walking steps, heart rate, calories consumed, andsleep pattern. All signals read by these wearable devices come either from themuscle activities or movement of the user. This however, limits the ability ofthese devices to monitor people’s mental states. A brain computer interface ismore suitable for such analysis since Electroencephalogram (EEG) signals arec© Springer-Verlag Berlin Heidelberg 2016M.L. Gavrilova et al. (Eds.): Trans. on Comput. Sci. XXVIII, LNCS 9590, pp. 108–126, 2016.DOI: 10.1007/978-3-662-53090-0 6

CogniMeter: EEG-Based Brain States Monitoring 109

directly captured from brain activity. A number of EEG-based methods and cor-responding applications have been designed and implemented in order to recog-nize the user’s brain states [1–3]. However, most of the contemporary worksneither process EEG data in real time nor provide intuitive visualization toolsfor data analysis and decision making.

The EEG-based visualization of brain states can provide additional insightsfor human factor study experiments. In experimental design and analysis, thevisualization tool can enable researchers to directly observe the relationshipbetween task variables and subject’s states. In [40], an EEG-based brain statesmonitoring system is proposed to monitor emotion, mental workload and stressin real time simultaneously. The stress level is inferred from emotion and work-load recognition. In this paper, we describe a new version of the brain statesmonitoring system CogniMeter integrated with updated mental workload andstress recognition algorithms.

This paper is structured as follows: Sect. 2 introduces related work on EEGvisualization tools and EEG-based emotion, mental workload and stress recog-nition algorithms. Section 3 presents the methodology used to implement thesealgorithms. Section 4 introduces the structure of the proposed visualization sys-tem and details of the visualization tools used to monitor brain states from EEGsignals. Section 5 gives the conclusion.

2 Related Work

2.1 Visualization Tools

There are a number of tools for analysis and visualization of EEG data such asEEGLab [4], Brainstorm [5], and ELAN [6]. These systems provide graphic userinterfaces which enable users to interactively process high-density EEG data.For intuitive observation of brain activity, the amplitudes of EEG signals canbe mapped to a 2-D or 3-D model of the scalp according to the EEG channelpositions. These visualization tools are able to process EEG data epochs usingspectral analysis or independent component analysis [4]. In [7], a blobby modelis implemented for EEG signal visualization onto a 3D head model.

Because EEG signals are non-stationary and high-dimensional, machinelearning methods can be used for efficient real-time analysis of brain activitywith specific tasks [8]. In this way, visualization tools can provide further inter-pretation of mental states from EEG data instead of showing only amplitudes ofdifferent channels. In [9], an EEG-based workload gauge is implemented and theworkload level is monitored when subjects are doing cognitive and operationalAir Traffic Controller’s (ATC) tasks. It is critical to reliably measure the brainstates and performance of the controllers/pilots when the mode of automationis changed or when new tasks appear. During simulation of the ATC task, theworkload gauge is updated with the recognized level of workload (three workloadlevels). Other gauges depict dynamic changes in EEG signals and show spectrumpowers in the form of brain maps simultaneously. In [8], an EEG-based mentaltext entry system Hex-o-Spell is proposed. By imagining left/right hand move-ment, the user can generate different brain patterns. These patterns are used to

110 X. Hou et al.

control visual text input. The visualization tools include a text dial, an arrow,and a bar. The arrow’s orientaion and the graphic bar’s length are controlled bythe brain states.

2.2 Emotion

Emotions can be defined from a dimensional perspective where arousal, valence,and dominance dimensions are considered [10]. In the dimensional model, thearousal dimension ranges from not aroused to excited, the valence dimensionranges from negative to positive, and the dominance dimension ranges from beingcontrolled to being in control. The dimensional model is preferable for emotionrecognition because it allows the location of discrete emotions in the dimensionalspace. Even feelings that cannot be described in words can be located using thedimensional model [11]. Emotions can be induced by different kinds of stimulisuch as audio, visual and combined ones [12]. Different algorithms have beenproposed for EEG-based emotion recognition. [3] extracted power features fromEEG data and used Support Vector Machine (SVM) as a classifier. 82.37 %accuracy for distinguishing four emotions was achieved with 32 channels. In[13], Short Time Fourier Transform (STFT) as a feature extraction method andSVM as a classifier were applied and a mean accuracy of 62.07 % was obtainedwith 16 channels. However, all these accuracies were achieved for off-line emotionrecognition only. As the EEG signal is nonlinear and chaotic, traditional featuresmay be hard to capture the nonlinear property of EEG. The fractal dimension(FD) can reflect changes of the EEG signal during different mental tasks in realtime [15]. In [16], FD, statistical and Higher Order Crossings (HOC) featureswere used for real-time EEG-based emotion recognition.

2.3 Mental Workload

Mental workload is described as a noticeable relationship between the humancognitive capacity and the effort required to process a particular function [17].There are mainly three broad categories for workload definition: physiological,subjective, and cognitive workloads [18]. In this research, we are interested incognitive workload which indicates the capability of a person to complete a taskwith some amount of mental effort. Performance in the task ascertains the cog-nitive workload [19]. In [8], mental workload is evaluated in online EEG monitor-ing during a security surveillance task. By comparing the mental workload indexwith the error rate of the subjects, the correlation coefficient is approximately0.7, which indicates that when the workload increases, people have a tendencyto make more errors. The significant positive correlation between workload andtheta band power has been proven in [20–22]. In experiment [20], it is shownthat the EEG theta band power increases when the workload is induced witha mental arithmetic task. In [21], the driver’s mental workload is significantlycorrelated with theta band power and alpha band power. In different drivingtasks, the frontal theta activity shows significant increase when working mem-ory load increases. In another experiment studying the workload and fatigue in

CogniMeter: EEG-Based Brain States Monitoring 111

aircraft pilots [22], increased EEG theta band power and decreased alpha bandpower was observed during high mental workload as compared with low mentalworkload conditions. Additionally, in [22] it is shown that when the pilots havehigh mental workload and mental fatigue, their EEG theta band power, as wellas delta and alpha band power increases.

2.4 Stress

Stress is a human state which is caused by a number of reasons, including highmental workload, emotions, or environmental influences. The stress can be mea-sured and assessed from physiological variables including EEG [1,2,23,24], bloodpressure [25], heart rate variability [26], and skin conductance level [27]. EEG canbe used to detect human stress levels. In work [25], the experiment shows that thestress is positively correlated with beta EEG power at the anterior temporal lobe.In [2], higher order spectra features are used for stress recognition. The SVMwith RBF kernel is chosen as a classifier and the accuracy calculated with 5-foldcross validation for recognition of two stress states is 79.2 %. In [23], featuressuch as Gaussian mixtures of EEG spectrogram, fractal dimension and magni-tude square coherence estimation are used in the stress recognition algorithm.The classification of two levels of mental stress is done by k-Nearest Neighbor(k-NN) and SVM classifiers, and the best accuracy is 90 %. However, neither [2]nor [23] used a standard stressor to induce stress in the experiments. In [1], aStroop color-word test is used to induce stress. The discrete cosine transform isapplied to reduce the data size and extract features from the frequency domain.Classification is implemented with an artificial neural network, linear discrim-inant analysis and k-NN. The best classification result for two stress states is72 % with k-NN. In [24], the band power of theta, alpha, and beta are used asfeatures in logistic regression and are fed into the k-NN classifier. The resultsshow a median accuracy of 73.96 % for the recognition of relaxed and stressedstates.

3 Methodology

3.1 EEG Device

The Emotiv Headset [28] is used to capture the users’ EEG signals wirelesslywith the USB receiver. It has 14 channels located at AF3, F7, F3, FC5, T7, P7,O1, O2, P8, T8, FC6, F4, F8, and AF4 as shown in Fig. 1.

3.2 Feature Extraction

In our real-time EEG-based brain states monitoring system CogniMeter, emo-tion, mental workload and stress are recognized using machine learning methods.The statistical and FD features are used in real-time EEG-based brain statesrecognition.

112 X. Hou et al.

Fig. 1. Location of 14 electrodes of Emotiv EEG device.

Fractal Dimension Feature. FD measures the complexity and irregularity ofa time series [29]. It can be used as an index for characterizing the complexitiesof EEG signals. For a regular signal, the fractal dimension value is low. If thesignal becomes irregular, the fractal dimension value increases accordingly. Wangand Sourina [30] proposed to use Higuchi fractal dimension to recognize differentarithmetic mental tasks from EEG. It is also used in EEG-based serious gamesto identify attention levels [31]. In this paper, the Higuchi algorithm is usedto calculate FD features from EEG data. The idea of Higuchi algorithm is asfollows.

Let X (1) ,X (2) , . . . , X (N) be a finite set of time series samples. Then, thenewly constructed time series is

Xmt : X (m) ,X (m + t) , . . . , X

(m +

[N − m

t

]· t

). (1)

where m = 1, 2, . . . , t is the initial time and is the interval time [29]. t sets ofLm (t) are calculated by

Lm (t) =

⎧⎪⎨⎪⎩

⎛⎜⎝

[N−mt ]∑

i=1

|X (m + it) − X (m + (i − 1) · t)|

⎞⎟⎠ N − 1[

N−mt

] · t

⎫⎪⎬⎪⎭ /t. (2)

〈L (t)〉 denotes the average value of Lm (t), and one relationship exists

〈L (t)〉 ∝ t−dimH . (3)

Then the Fractal Dimension dimH can be obtained by logarithmic plottingbetween different t (ranging from 1 to tmax) and its associated 〈L(t)〉 [35]

dimH =ln 〈L(t)〉− ln t

. (4)

CogniMeter: EEG-Based Brain States Monitoring 113

Thus, the composed FD feature vector is

FVFD = [dimH ]. (5)

Statistical Feature. Statistical features are simple and widely used in EEG-based brain states recognition. For example, statistical features were usedin EEG-based emotion recognition algorithms [32,33]. Six statistical featuresincluding mean μX , standard deviation σX , mean of absolute values of the firstdifferences δX , mean of absolute values of the first differences of normalizedsignals δX , mean of absolute values of the second differences γX , and mean ofthe second differences of the normalized signals γX are extracted from EEG foremotion recognition. The equations are listed as follows:

μX =1N

N∑n=1

X(n), (6)

σX =

√√√√ 1N

N∑n=1

(X(n) − μX)2, (7)

δX =1

N − 1

N−1∑n=1

|X(n + 1) − X(n)| , (8)

δX =1

N − 1

N−1∑n=1

∣∣X(n + 1) − X(n)∣∣ =

δXσX

, (9)

γX =1

N − 2

N−2∑n=1

|X(n + 2) − X(n)| . (10)

The composed statistical feature is

FVstatistical = [μX , σX , δX , δX , γX , γX ]. (11)

3.3 Data Processing

In the current version of CogniMeter, we use subject-dependent algorithms. Thesubject-dependent algorithms consist of two parts: calibration algorithm andreal-time brain state recognition algorithm. The overall diagram of calibrationand real-time emotion, mental workload and stress recognition algorithms isshown in Fig. 2. The emotion, mental workload and stress calibration algorithmsfollow the same pipeline but with different features. Different stimuli are usedto induce brain states as well. First, the EEG data are labeled with classes suchas levels of valence, mental workload or stress for emotion, workload or stressrecognition correspondingly. Then, the EEG data are filtered, the correspond-ing features are extracted and the classifier is trained. After that, for real-time

114 X. Hou et al.

recognition of valence, mental workload or stress levels, raw EEG data is filteredand the corresponding features are extracted using a 4 s sliding window with3 s of overlap. Then, the data is input into the classifier with the pre-trainedmodel acquired during calibration. After that, the classifier assigns the recog-nized valence, mental workload or stress level to each 4 second of EEG data.

Fig. 2. The overall diagram of calibration and real-time brain states recognitionalgorithms.

Emotion Recognition. In this paper, the valence recognition algorithm pro-posed in work [39] is integrated into the CogniMeter system. Emotions can beevoked by audio or visual stimuli such as IADS [34] and IAPS [35]. The algo-rithm calibration procedure is described in Sect. 4.1. The algorithm is tested on10 subjects from DEAP database which is a publicly available affective database.Fractal dimension is used as the feature and thresholds are employed to classifythe brain states for up to four levels of valence, ranging from very unpleasant tovery pleasant. 12 channels (AF3, F3, F7, T7, P3, P7, AF4, F4, F8, T8, P4, P8)are used and the weighted average voting strategy is applied to make the finaldecision about the current valence level. As shown in Table 1, the best accuracyis 79.31 % and the mean accuracy is 49.4 % for four levels valence recognition.In the calibration phase, two EEG datasets labeled with very pleasant and veryunpleasant emotion are selected based on the questionnaire. Then, raw EEGdata from the selected 12 channels are filtered with a 2–42 Hz bandpass filter,and features are extracted by subtracting the FD values of 6 right-hemisphericchannels from the 6 left-hemispheric channels:

ΔFD = FDleft − FDright,∀left ∈ LH,∀right ∈ RH (12)

where a sliding window of size 512 and step size 1 (move forward 1 sample pointat each step) is used to calculate the FD values.

CogniMeter: EEG-Based Brain States Monitoring 115

In the next step, the simple moving average (SMA) function with non-overlapping window of size of 128 is applied to ΔFD and the obtainedaverage ΔFD is used as features. To decide the threshold, the maximal and min-imal average ΔFD values are obtained and the range between max ΔFD andmin ΔFD is equally divided. As 12 channels are used in the proposed algorithm,36 channel pairs are obtained (ΔFDAF3−AF4,ΔFDAF3−F4, . . . ,ΔFDFP1−FP2,ΔFDP7−P8) which leads to 36 sets of thresholds. In the real-time recognitionphase, the average ΔFD is calculated from each channel pair and compared withthe corresponding set of thresholds. The final valence level is determined by theweighted average vote strategy based on 36 valence results from each channel pairas follows. Define C(x) as the function that counts the votes for each valence level,where argument x ∈ {1, 2, 3, 4} is the valence level, 1 denotes the most unpleasant,2 denotes unpleasant, 3 denotes pleasant, and 4 denotes the most pleasant. C(x)gives the vote counts of valence level x and

∑4x=1 C(x) = 36. The final valence

level L is determined as:

L =∑4

x=1 xC(x)∑4x=1 C(x)

(13)

where rounded L value can be used.

Table 1. Classification accuracy of valence level recognition using weighted averagevote method

Subject ID Weighted average vote

1 47.41 %

5 48.28 %

7 41.38 %

10 74.14 %

13 46.55 %

14 39.66 %

16 79.31 %

19 47.42 %

20 29.75 %

22 40.09 %

Average 49.40 %

Mental Workload Recognition. Mental workload recognition algorithm pro-posed in [38] is integrated into the CogniMeter system. Since the mental work-load is closely related to multitasking, SIMKAP simultaneous capacity test isused to induce 4 levels of mental workload to calibrate the algorithm. The algo-rithm calibration procedure is described in Sect. 4.1. The algorithm was testedon the EEG database of 12 subjects with different feature combinations andclassifiers used. In Table 2, it is shown, that for all feature combinations, the

116 X. Hou et al.

average accuracy of SVM classifier is 9.56 % higher than k-NN classifier basedon mental workload EEG data. By combining statistical and FD features of 14channels and using SVM classifier, the best accuracy is 90.39 % for 2 levels men-tal workload recognition and 80.09 % for 4 levels mental workload recognition.Thus, in CogniMeter System, for mental workload recognition algorithm, we usestatistical and FD features and SVM classifier. Parameters used for the SVMclassifier are: polynomial kernel with penalty parameter C = 1, degree d = 5,gamma g = 1, coefficient r = 1. The feature vector defined for mental workloadrecognition is described as follows

FV w = [FV1w, FV w

2 , . . . , FV w14], (14)

FViw = [FV w

i(FD), FV wi(statstical)], (15)

where numbers 1 to 14 represent EEG channels AF3, F7, F3, FC5, T7, P7, O1,O2, P8, T8, FC6, F4, F8, AF4 according to the 10–20 international system.

Table 2. Average classification accuracy of different feature combinations for 2 and 4levels mental workload recognition using SVM and k-NN classifiers

Classifier Power Stat FD Power+ Stat Power+ FD Stat. + FD

2 Class 84.16 % 89.79 % 82.92 % 88.05 % 85.79 % 90.39 %

SVM (6.39) (5.22) (6.48) (4.97) (5.80) (4.63)

2 Class 79.32 % 84.67 % 80.27 % 81.69 % 80.04 % 84.97 %

k-NN (5.24) (6.45) (6.08) (4.95) (4.70) (6.39)

4 Class 69.49 % 78.56 % 65.26 % 75.90 % 71.78 % 80.09 %

SVM (8.03) (9.95) (10.06) (8.18) (8.14) (8.59)

4 Class 60.14 % 69.65 % 60.63 % 64.15 % 61.47 % 70.28 %

k-NN (7.60) (10.17) (9.05) (8.42) (6.48) (9.97)

Stress Recognition. Stress recognition algorithm proposed in [37] is integratedin the CogniMeter System. For calibration, the Stroop Color Word Test [36] isadapted to elicit different levels of stress. The calibration procedure is describedin Sect. 4.1. The algorithm was tested on the EEG database of 9 subjects wheredifferent levels of stress were induced using a Stroop color-word test.

The comparison results shown in Table 3 also shows that the feature com-bination and classifier setting of stress recognition is very similar to mental work-load recognition. By combining fractal dimension and statistical features withSVM as the classifier, four levels of stress can be recognized with an averageaccuracy of 67.06 %, three levels of stress can be recognized with an accuracy of75.22 %, and two levels of stress can be recognized with an accuracy of 85.71 %.For SVM classifier, the polynomial kernel is chosen with penalty parameter

CogniMeter: EEG-Based Brain States Monitoring 117

C = 10, degree d = 3, gamma g = 1, and coefficient r = 1. By using EEGsignal from all 14 channels, the feature vector defined for stress recognition is asfollows

FV s = [FV1s, FV s

2 , . . . , FV s14], (16)

FVis = [FV s

i(FD), FV si(statstical)]. (17)

Table 3. Average classification accuracy of different feature combinations for 2, 3,and 4 levels stress recognition using SVM and k-NN classifiers

Classifier Power FD + Stat Power+ FD + Stat

2 Class 73.18 % 85.71 % 80.96 %

SVM (10.20) (9.39) (8.86)

2 Class 66.36 % 76.72 % 69.93 %

k-NN (8.96) (10.67) (9.44)

3 Class 58.39 % 75.22 % 69.82 %

SVM (11.32) (12.53) (11.88)

3 Class 50.98 % 63.24 % 54.44 %

k-NN (10.99) (13.26) (10.63)

4 Class 49.66 % 67.06 % 60.71 %

SVM (12.20) (13.20) (11.50)

4 Class 41.35 % 54.31 % 44.97 %

k-NN (9.64) (10.74) (11.42)

4 Visual Monitoring

Based on the emotion, workload and stress recognition algorithms described, wepropose a real-time EEG-based brain states visual monitoring system namedCogniMeter which is implemented to visualize the recognized brain states likeemotion, workload, and stress levels. The overall pipeline of CogniMeter is illus-trated in Fig. 3. It includes two parts: recognition and visualization.

In the recognition part, the raw EEG data stream is obtained from the EEGdevice as input. Then, the EEG data is passed through the bandpass filter (2–42 Hz). After that, different features are extracted from the filtered EEG data.For emotion recognition, the FD feature is used. For workload and stress recog-nition, statistical and FD features extracted from EEG data are used. Theseextracted features are compared with thresholds (emotion recognition) or fedinto the pre-trained SVM classifier models (stress and workload recognition).Finally, emotion, workload and stress are recognized in real time.

In the visualization part, a Node.js server is created to receive the recognizedemotion, workload and stress states. These recognized brain states are sent byeach recognition program through TCP sockets. Then, a web browser connectsto the server and renders three meters showing emotion, stress and workload inreal time.

118 X. Hou et al.

Fig. 3. The overall diagram of CogniMeter system.

The two advantages of this framework are the independence of recognitionprograms and flexibility for use in various applications. When a new algorithmis proposed or a new EEG device is used, they can be easily integrated to thevisual monitoring system with a TCP connection. If the monitoring system isdeployed in different applications like driver’s model study, air traffic controlor maritime human factor study, the visualization tools can be easily changedaccording to simulation scenarios. The users can monitor a subject’s brain stateseven from their tablet PC or smart phones.

CogniMeter: EEG-Based Brain States Monitoring 119

4.1 Calibration Interface

As the proposed real-time EEG-based brain state recognition algorithms are sub-ject dependent, calibration is required before real-time recognition. Each algo-rithm has its own calibration interface and calibration procedure.

The emotion calibration interface is shown in Fig. 4. The stimuli to evoke cer-tain emotions can be audio, visual, or both. In our calibration, sound clips fromIADS database [34] targeted at evoking different valence levels are chosen andplayed one after another to the subjects. EEG data is recorded at the same timeas when the subjects are listening to the sound clips. After each sound clip whichlasts 60 s, the subject completes a prompted questionnaire (shown in Fig. 4a) toevaluate and describe his/her current emotion and feelings in words. After therecording is complete, the thresholds are trained based on the recorded EEGdata and given emotional labels and are saved for future use. Figure 4b, showsthe training of positive (happy) and negative (sad) emotions. When the “Goto Emotion Recognition” button is clicked, the program starts real-time emo-tion recognition based on the trained thresholds and sends recognized emotionalstates to the server.

The interface for mental workload calibration is shown in Fig. 5. SIMKAPsimultaneous capacity test is used to invoke different workload levels. In lowmental workload condition, subjects are required to complete item matching fordifferent types of items including numbers, letters and shapes. For high men-tal workload condition, subjects are required perform multitasking as shownin Fig. 5a. It combines item matching and auditory questions such as tele-phone book search, schedule checking and answering arithmetic questions at therequired time. After completing each workload level which lasts 60 s, the subjectfills a prompted questionnaire to evaluate his/her mental workload level on thescale from 1 to 9 and also attempts to describe their feelings in words, as shownin Fig. 5b. Figure 5c shows labels of the recorded EEG data trained as three levelsof workload: no task condition (Relaxed), item matching (Engrossed), and mul-titasking (Nervous). By clicking the “Start Recognition” button, the recognizedmental workload states are sent to the server in real time.

The interface for stress calibration is shown in Fig. 6. Stroop color-word test,often used as a reliable psychological stressor, is applied to induce different levelsof stress. The test has been shown to be one of the most effective methods forresearching human psychophysiological reactions in a stressful environment. Inthe Stroop color-word test, the subjects are given a list of color words in matchingor non-matching colors as shown in Fig. 6a. In the congruent task, the word’s fontcolor matches with the word’s meaning, e.g. the word “yellow” in a “yellow” font,and subjects are tasked to identify the word’s font color. For the incongruenttask, the word’s font color and the word’s meaning are different, e.g. the word“yellow” but in a “blue” font, and subjects are required to correctly identifythe word’s font color. Due to the mismatch in font color and word meaning,the incongruent session is more stressful than congruent section. For the morestressful task, the subject can be required to make the response within a limitedtime (1.5 s). In such a situation, even higher stress levels can be elicited. After

120 X. Hou et al.

Fig. 4. Screenshots of the emotion calibration interface: (a) questionnaire to label EEGdata; (b) labels of the recorded EEG data.

completing each Stroop color-word test, the subject is required to fill a promptedquestionnaire to evaluate his/her stress level on a scale of 1 to 9 and describetheir feeling in words, as shown in Fig. 6b. Figure 6c shows the labels of therecorded EEG data as five stress levels: no task condition (Relaxed), congruenttask (Normal), incongruent task (Engrossed), congruent task with time limit(Stressful), and incongruent task with time limit (Anxious). By clicking “StartRecognition”, the recognized stress level is sent to the server in real-time.

CogniMeter: EEG-Based Brain States Monitoring 121

Fig. 5. Screenshots of the mental workload calibration interface.

Fig. 6. Screenshots of the stress calibration interface.

122 X. Hou et al.

4.2 Visual Meters and Monitoring Report

In the CogniMeter system, the recognized emotion, workload and stress levelsare visualized as meters shown in Fig. 7. These visualization meters are devel-oped based on JustGage [41] using JavaScript. These three meters display thecurrent emotion, workload or stress level with changing positional angular barsand gradient colors (from green to red) in real time. A green colored bar on theworkload/stress meter indicates a low workload/stress level, while a red coloredbar represents a high workload/stress level. For the emotion meter, if the sub-ject’s state is positive, the color of the bar will be green. Otherwise, it will bered. Besides color representation, there is a word in the center of each meter todescribe current workload, emotion, and stress states. The words are updated atthe bottom of the meter and correspond to the meter colors. For workload/stress,“High” corresponds to red color, “Low” corresponds to green color and “Mid”corresponds to yellow color. For emotion, “Pos” corresponds to positive emotions(green) while “Neg” corresponds to negative emotions (red).

Fig. 7. Screenshot of the CogniMeter for real-time brain stats monitoring system.Meters listed from left to are current level of workload, recognized emotion and currentlevel of stress. (Color figure online)

When the brain states monitoring is complete, a report appears on the screen.It is developed using HTML5 JavaScript charting library CanvasJS [42] and isgenerated to summarize the distribution of emotion, workload and stress statesmonitored. For example, Fig. 8 shows the distribution of mental workload inthree levels (Low, Medium, and High), distribution of two emotions (positive andnegative), and distribution of stress in five levels (Low, Medium low, Medium,Medium High, and High). The report can help researchers perform better analy-sis of the changes in a subject’s brain state during experiments.

CogniMeter: EEG-Based Brain States Monitoring 123

Fig. 8. EEG-based brain states monitoring report of workload, emotion and stress forthree minutes monitoring.

5 Conclusion

In this paper, a novel CogniMerter system is proposed and implemented. Itallows monitoring of users real-time emotion, mental workload and stress usinga head mounted EEG device. Visual meters and report generation provide thepossibility to directly observe the relationship between the users emotions, men-tal workload, stress, and his/her task performance. This can help researchers topropose new hypotheses and refine their experimental procedures. We extracteddifferent feature combinations and used SVM classifier for real-time emotion,

124 X. Hou et al.

mental workload and stress recognition. To monitor brain states in real-time,dynamic meters are proposed and implemented. The CogniMeter system is usedas a novel user study method in different simulation scenarios such as air trafficcontrol and maritime training.

Acknowledgments. The work is supported by Fraunhofer IDM@NTU, which isfunded by the National Research Foundation (NRF) and managed through the multi-agency Interactive & Digital Media Programme Office (IDMPO).

References

1. Lim, C.-K.A., Chia, W.C.: Analysis of single-electrode EEG rhythms using MAT-LAB to elicit correlation with cognitive stress. Int. J. Comput. Theor. Eng. 7,149–155 (2015)

2. Hosseini, S.A., Khalilzadeh, M.A., Naghibi-Sistani, M.B., Niazmand, V.: Higherorder spectra analysis of EEG signals in emotional stress states. In: 2010 Sec-ond International Conference on Information Technology and Computer Science(ITCS), pp. 60–63 (2010)

3. Yuan-Pin, L., Chi-Hong, W., Tien-Lin, W., Shyh-Kang, J., Jyh-Horng, C.: EEG-based emotion recognition in music listening: a comparison of schemes for mul-ticlass support vector machine. In: IEEE International Conference on Acoustics,Speech and Signal Processing, pp. 489–492 (2009)

4. Delorme, A., Makeig, S.: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Meth-ods 134, 9–21 (2004)

5. Tadel, F., Baillet, S., Mosher, J.C., Pantazis, D., Leahy, R.M.: Brainstorm: a user-friendly application for MEG/EEG analysis. Comput. Intell. Neurosci. 2011, 13(2011)

6. Aguera, P.-E., Jerbi, K., Caclin, A., Bertrand, O.: ELAN: a software package foranalysis and visualization of MEG, EEG, and LFP signals. Comput. Intell. Neu-rosci. 2011, 5 (2011)

7. Sourina, O., Wang, Q., Liu, Y., Nguyen, M.: Fractal-based brain state recognitionfrom EEG in human computer interaction. Biomed. Eng. Syst. Technol. 273, 258–272 (2013)

8. Muller, K.-R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G.,Blankertz, B.: Machine learning for real-time single-trial EEG-analysis: from brain-computer interfacing to mental state monitoring. J. Neurosci. Methods 167, 82–90(2008)

9. Weiland, M.Z., Roberts, D.M., Fine, M.S., Caywood, M.S.: Real time researchmethods: monitoring air traffic controller workload during simulation studiesusing electroencephalography (EEG). In: Proceedings of the Human Factors andErgonomics Society Annual Meeting, pp. 1615–1619 (2013)

10. Mehrabian, A.: Pleasure-arousal-dominance: a general framework for describingand measuring individual differences in temperament. Curr. Psychol. 14(4), 261–292 (1996)

11. Mauss, I.B., Robinson, M.D.: Measures of emotion: a review. Cogn. Emot. 23,209–237 (2009)

CogniMeter: EEG-Based Brain States Monitoring 125

12. Takahashi, K.: Remarks on emotion recognition from multi-modal bio-potentialsignals. In: IEEE International Conference on Industrial Technology (IEEE ICIT2004), vol. 3, pp. 1138–1143 (2004)

13. Schaaff, K.: EEG-based emotion recognition. Diplomarbeit am Institut fur Algo-rithmen und Kognitive Systeme, Universitat Karlsruhe (TH) (2008)

14. Pradhan, N., Dutt, D.N.: Use of running fractal dimension for the analysis of chang-ing patterns in electroencephalograms. Comput. Biol. Med. 23, 381–388 (1993)

15. Lutzenberger, W., Elbert, T., Birbaumer, N., Ray, W.J., Schupp, H.: The scalpdistribution of the fractal dimension of the EEG and its variation with mentaltasks. Brain Topogr. 5, 27–34 (1992)

16. Liu, Y., Sourina, O.: Real-time subject-dependent EEG-based emotion recognitionalgorithm. In: Gavrilova, M.L., Tan, C.J.K., Mao, X., Hong, L. (eds.) Transac-tions on Computational Science XXIII. LNCS, vol. 8490, pp. 199–223. Springer,Heidelberg (2014)

17. Hart, S., Staveland, L.: Development of a multi-dimensional workload rating scale:results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.)Human Mental Workload (1988)

18. Waard, D., Studiecentrum, V.: The measurement of drivers’ mental workload.Groningen University, Traffic Research Center (1996)

19. Sweller, J.: Cognitive load during problem solving: effects on learning. Cogn. Sci.12, 257–285 (1988)

20. Sammer, G., Blecker, C., Gebhardt, H., Bischoff, M., Stark, R., Morgen, K.,Vaitl, D.: Relationship between regional hemodynamic activity and simultane-ously recorded EEG-theta associated with mental arithmetic-induced workload.Hum. Brain Mapp. 28, 793–803 (2007)

21. Lei, S., Roetting, M.: Influence of task combination on EEG spectrum modulationfor driver workload estimation. Hum. Factors J. Hum. Factors Ergon. Soc. 53,168–179 (2011)

22. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., Babiloni, F.: Measuring neuro-physiological signals in aircraft pilots and car drivers for the assessment of mentalworkload, fatigue and drowsiness. Neurosci. Biobehav. Rev. 44, 58–75 (2014)

23. Khosrowabadi, R., Chai, Q., Kai Keng, A., Sau Wai, T., Heijnen, M.: A brain-computer interface for classifying EEG correlates of chronic mental stress. In: The2011 International Joint Conference on Neural Networks (IJCNN), pp. 757–762(2011)

24. Calibo, T.K., Blanco, J.A., Firebaugh, S.L.: Cognitive stress recognition. In: 2013IEEE International Instrumentation and Measurement Technology Conference(I2MTC), pp. 1471–1475 (2013)

25. Hassellund, S.S., Flaa, A., Sandvik, L., Kjeldsen, S.E., Rostrup, M.: Long-termstability of cardiovascular and catecholamine responses to stress tests: an 18-yearfollow-up study. Hypertension 55, 131–136 (2010)

26. Ssanghee, S., Yeongjun, G., Jungtae, L.: The relation between affective style ofstressor on EEG asymmetry and stress scale during multimodal task. In: ThirdInternational Conference on Convergence and Hybrid Information Technology,ICCIT 2008, pp. 461–466 (2008)

27. Jacobs, S.C., Friedman, R., Parker, J.D., Tofler, G.H., Jimenez, A.H., Muller, J.E.,Benson, H., Stone, P.H.: Use of skin conductance changes during mental stresstesting as an index of autonomic arousal in cardiovascular research. Am. Heart J.128, 1170–1177 (1994)

28. The Emotiv EEG device. http://www.emotiv.com/eeg/features.php

126 X. Hou et al.

29. Higuchi, T.: Approach to an irregular time series on the basis of the fractal theory.Phys. D Nonlinear Phenom. 31, 277–283 (1988)

30. Wang, Q., Sourina, O.: Real-time mental arithmetic task recognition from EEGsignals. IEEE Trans. Neural Syst. Rehabil. Eng. 21, 225–232 (2013)

31. Wang, Q., Sourina, O., Nguyen, M.: Fractal dimension based neurofeedback inserious games. Vis. Comput. 27, 299–309 (2011)

32. Liu, Y., Sourina, O.: EEG databases for emotion recognition. In: 2013 InternationalConference on Cyberworlds (CW2013), pp. 302–309 (2013)

33. Lan, Z., Sourina, O., Wang, L., Liu, Y.: Real-time EEG-based emotion monitoringusing stable features. Vis. Comput. 32, 347–358 (2016)

34. Bradley, M.M., Lang, P.J.: The International Affective Digitized Sounds (IADS-2): affective ratings of sounds and instruction manual. University of Florida,Gainesville, FL, Technical report B-3 (2007)

35. Lang, P., Bradley, M.M.: The International Affective Picture System (IAPS) inthe study of emotion and attention. In: Handbook of Emotion Elicitation andAssessment, vol. 29 (2007)

36. Tulen, J., Moleman, P., van Steenis, H., Boomsma, F.: Characterization of stressreactions to the Stroop color word test. Pharmacol. Biochem. Behav. 32, 9–15(1989)

37. Hou, X., Liu, Y., Sourina, O., Tan, E., Wang, L., Mueller-Wittig, W.: EEG basedstress monitoring. In: IEEE International Conference on in Systems, Man andCybernetics (SMC), pp. 3110–3115 (2015)

38. Lim, W.L., Sourina, O., Wang, L., Liu, Y.: EEG-based mental workload recog-nition related to multitasking. In: 10th International Conference on Information,Communications and Signal Processing (ICICS), (2015, in press)

39. Lan, Z., Liu, Y., Sourina, O., Wang, L.: Real-time EEG-based user’s valence mon-itoring. In: Internation Conference on Information, Communications and SignalProcessing (ICICS), pp. 1–5, 2–4 December 2015

40. Hou, X., Liu, Y., Sourina, O., Mueller-Wittig, W.: CogniMeter: EEG-based emo-tion, mental workload and stress visual monitoring. In: International Conferenceon Cyberworlds (CW), pp. 153–160 (2015)

41. JustGage. http://justgage.com42. CanvasJS. http://canvasjs.com/