Emotion based E-learning System using Physiological ...€¦ · Emotion based E-learning System...

Emotion based E-learning System using

Physiological Signals

Dr. Jerritta S, Dr. Arun SSchool of Engineering, Vels University, Chennai

CHENNAI - INDIA

Emotion based E-learning System using Physiological Signals Dr. Jerritta S, Dr. Arun S

Outline

Introduction

Existing Research works on Emotion Recognition

Research Methodology

Results

Emotion based e-learning system

GUI

Future Work

MATLAB tools

Publications


Introduction

Emotions

Mental and physiological state associated with feelings,

thoughts and behavior

Learnt in diverse areas like Psychology, Cognitive Science,

Philosophy and Computer Science, However there hasn’t

been an universally accepted definition or categorization of

emotional states

Highly subjective and an integral part of any communication,

learning, perception and decision making


Introduction (Cont.)

Need of Emotion Recognition System

Intelligent machine interfaces for smooth interaction

between machines and human

Human Computer Intelligent Interaction (HCII) for natural

interaction between human and machines

Improves mutual empathy

Smart classrooms, Computer based Training, Medical

applications


Emotion Recognition Methods


SELF REPORT methods

Questionnaires, Ratings and descriptions provided by the subject

Participant biased


Emotion Recognition Methods (Cont.)


OBSERVOR (BEHAVIOUR) methods

Facial, Vocal and gesture cues

Dependant on external circumstances and prone to social

masking


Emotion Recognition Methods (Cont.)


NEURO-PHYSIOLOGICAL signal processing methods

Physiological response of the Central Nervous System (CNS) and

Autonomous Nervous System (ANS)

Electroencephalogram (EEG), Electrocardiogram (ECG),

Electromyogram (EMG), Galvanic Skin Response (GSR), Skin

Temperature (ST), Skin Conductance (SC) etc.,

Complex, but provides the TRUE emotional state of the person.


Applications of Emotion recognition systems


Robots

Dialogue systems

Computer based learning

Smart Classroom

Therapists for ASD

Medical Doctors


E-learning systems


Feedback mechanisms in existing e-learning systems help to

study at the pace of the user.

They take into account only the ‘understanding on the

subject’ and not the state such as ‘fatigue’, ‘emotions’ etc.,


Emotion based E-learning systems


Increases the receptiveness and productivity of the user

Suggests appropriate action to be taken depending on the

emotional state of the learner


Existing research works on EmotionRecognition

REFERENCES BIOSIGNALSNO OF

SUBJECTS

NO OF

EMOTIONS

EMOTIONAL

STIMULICLASSIFICATION RATE (%)

(Picard, Vyzas et al. 2001) EMG, BVP, SC, RR 1 8Personalized

Imagery 81(User Dependent)

(Lisetti and Nasoz 2004) GSR, HR, ST 29 6 Movies

72(User Dependent)

75(User Dependent)

84(User Dependent)

(Lan and Ji-hua 2006) ECG, ST, SC, RR 60 3 Movies 85.3 (User Dependent)

(Maaoui and Pruski 2008) BVP, EMG, SC,

ST, RR 25 6 Visual (IAPS) 88(User Dependent)

(Jonghwa and Ande 2008) EMG, ECG, SC,

RR

3 (22 trials)

MIT

database

4 Music 95(User Dependent)

70(User Independent)

(Kim and André 2009) EMG, ECG, SC,

RR

3 (22 trials)

MIT

database

4 Music 95% (user dependant)

70% (user independent)

(Kim and André 2009) EMG, ECG, SC,

RR

3 (22 trials)

MIT

database

4 Music 91% (user dependant)


Existing research works on EmotionRecognition (Cont.)

REFERENCES BIOSIGNALSNO OF

SUBJECTS

NO OF

EMOTIONS

EMOTIONAL

STIMULICLASSIFICATION RATE (%)

(Chuan-Yu, Jun-Ying et

al. 2010)

ECG, RR, GSR,

BVP 11 3 Movies 90.6% and 90.2% (user independent)

(Maaoui and Pruski 2010) BVP, EMG, ST,

SC, RR 10 6 Visual (IAPS)

90(User Dependent) and

45(User independent)

(Gouizi, Reguig et al.

2011)4 6 Visual (IAPS) 85% (user dependent)

(Valenza, Lanata et al.

2012) EDA, ECG, RR 35 2 Visual (IAPS) 90% (user independant)

(Vanny, Park et al. 2013) BVP, ST, SC 4 4 Visual (IAPS)100% for fear and joy, 60% for disgust

and neutral (User dependent)

(Chang, Chang et al.

2013) ECG, GSR, BVP 11 4 Movies 89.2% (user independant)



Emotion classification rate varies from 45% to 100% for the different

research works.

They cannot be compared as they vary in,

Number of subjects

Type of elicitation

Type of physiological data

Number and placement of electrodes

Type of analysis (user dependency)

Though there is no standardization, some of the significant advances

are

IAPS, IADS database for emotion induction

Development of AuDB database (4 emotional states)

Statistical features for emotion recognition (Picard et al., 2001)



Challenges

Emotion database of physiological signals

Emotional states must be elicited internally

The sensors should be less intrusive and at the same time

capture the emotional changes

Methodology

Complex, non-linear and non stationary nature of

physiological signals

Subject dependence of emotions

Reliability


Research Methodology

Step 1

•Development of Emotional data base (ECG, EMG)

Step 2

•Preprocessing

Step 3•Feature Extraction (Linear and non-linear methods)

Step 4

•Fusion of emotional features derived from ECG and EMG

Step5

•Emotion Classification

Step 6

•Development of GUI


Results

78.47

64.62

82.54

0

10

20

30

40

50

60

70

80

90

ECG EMG ECG and EMG

Max

imu

m C

lass

ific

atio

nA

ccu

racy

(%

)Performance of Emotion Recogntition System


Emotion based E-Learning system

User can recheck the emotion again before proceeding further

Suggests the activity based on the detected emotional state

Detects emotional state from ECG and EMG signals

EMOTIONAL STATE SUGGESTION

Neutral Start Lessons

Happiness Start Lessons

Sadness Listen to Music

Fear Listen to Music

Surprise Calm Down

Disgust Play a game


GUI


Future Work

On-line system

Develop efficient algorithms to capture the emotional states

More Automation

Integrate with mobile apps, robots and other personalized devices


MATLAB TOOLS

Signal processing

Filter Design for pre-processing

Feature Extraction algorithms – Wavelets, Fourier Transform, Hilbert Huang

Transform, Empirical Mode Decomposition, Hurst exponent

Machine Learning

Pattern Recognition – KNN, Regression Tree, Bayesian Classifier

Confusion Matrix

SIMULINK

Development of GUI


Publications

International Journals

– Jerritta S, M Murugappan, Sazali Yacob, Khairunizam Wan ‘ECG based Emotion Recognition System using Empirical Mode Decomposition and Discrete Fourier Transform,’ Journal: “Expert Systems’ by Wiley Publishers (IF : 0.733)

– Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, "Emotion recognition from Facial EMG signals using Higher Order Statistics and Principal Component Analysis", Journal of Chinese Institute of Engineers (JCIE)(IF : 0.295) .

– Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, "Classification of emotional states from electrocardiogram signals: a non-linear approach based on Hurst" Biomedical engineering online, May 2013. (IF:1.61).

– Jerritta S, Murugappan M, Khairunizam Wan, Sazali Yaacob, “Frequency study of Facial electromyography signals with respect to emotion recognition”, Biomedical engineering/ Biomedzinische technik, Degruter, (IF: 1.157) Dec 2013.

– Arun S, Sundaraj Kenneth, Murugappan M (2012) “Hypovigilance detection using energy of Electrocardiogram signals”, Journal of Scientific & Industrial Research, 71(12), 794-799. (ISI Impact Factor 0.505).

– Arun Sahayadhas, Kenneth Sundaraj and Murugappan Murugappan (2012) “Detecting Driver Drowsiness Based on Sensors: A Review”, Sensors, 12, 16937-16953. (ISI Impact Factor 1.953).

– Arun Sahayadhas, Kenneth Sundaraj and Murugappan Murugappan, (2013) "Drowsiness detection during different times of day using multiple features", Australasian Physical & Engineering Sciences in Medicine 36(2), 243-250 (ISI Impact Factor 0.885).

– Arun Sahayadhas, Kenneth Sundaraj M Murugappan, “Electromyogram signal based hypovigilancedetection”, Biomedical Research 2014; 25(3), ISI Impact Factor 0.177).

– Arun Sahayadhas, Kenneth Sundaraj M Murugappan, Rajkumar Palaniappan, “A Physiological Measures-Based Method for Detecting Inattention in Drivers Using Machine Learning Approach”, Biocybernetics and Biomedical Engineering, Accepted for publication, ISI Impact Factor 0.157


Publications

International Conferences

• Jerritta S, M Murugappan, R Nagarajan, Kahirunizam Wan, “Physiological Signals Based Human Emotion Recognition: A Review”, IEEE Colloquium on Signal Processing and Applications 2011, 3-5 March 2011, Penang, Malaysia.

• Jerritta S, M Murugappan, R Nagarajan, Khairunizam Wan, “Investigation on different Emotion Elicitation Methods for Human Computer Interactaion”, International Conference on Robotics Automation System 2011, 23-24 May 2011, Terungannu, Malaysia.

• Jerritta S, M Murugappan, R Nagarajan, Khairunizam Wan, “Digital Filtering based Pre-processing of Electrocardiogram (ECG) Signals for Human Emotion Recognition”, Malaysian Technical Universities International Conference in Engineering, 2011.

• M Murugappan, NQI Baharuddin, S Jerritta, “DWT and MFCC based human emotional speech classification using LDA”, International Conference on Biomedical Engineering (ICoBE), 2012.

• Jerritta S, M Murugappan, Khairunizam Wan, Sazali Bin Yaacob, “Emotion Recognition from ECG using Hilbert Huang Transform”, in IEEE STUDENT 2012, Oct 6-9, 2012.

• Jerritta S, M Murugappan, Khairunizam Wan, Sazali Bin Yaacob, “Emotion detection from QRS complex of ECG signals using Hurst Exponent for different age groups”, Submitted to IEEE 3rd Workshop on Affective Brain-Computer Interfaces, Geneva, Swizerland, Sept 2,2013.

• Arun S., Murugappan, M., & Sundaraj, K. (2011). “Hypovigilance warning system: A review on driver alerting techniques”. In Control and System Graduate Research Colloquium (ICSGRC), 2011 IEEE (pp. 65-69).

• Arun, S., Sundaraj, K., & Murugappan, M. (2012). “Driver inattention detection: A review” . In Sustainable Utilization and Development in Engineering and Technology (STUDENT), 2012 IEEE. (pp. 1-6)

Emotion based E-learning System using Physiological ...€¦ · Emotion based E-learning System...

Documents

Transcript of Emotion based E-learning System using Physiological ...€¦ · Emotion based E-learning System...