Continuous Emotion Detection in Response to Music Videos · Continuous Emotion Detection in...

Continuous Emotion Detection in Response to Music Videos

Mohammad Soleymani1, Sander Koelstra2, Ioannis Patras2, Thierry Pun1

1Computer Science DepartmentUniversity of Geneva, Switzerland

2School of Computer Science and Electronic Engineering, Queen Mary University of London, UK

[email protected]

mailto:[email protected]

Outline

• Introduction

• Highlight detection

• Dataset collection

• Continuous affect detection

• The public dataset

• Summary

EmoSPACE 2011, Santa Barbara3/29/2011 2

Outline

• Introduction





• Summary


Application scenario

A music recommendation platform without direct input from our couch potato!

EmoSPACE 2011, Santa Barbara

EEG

Physiologicalmeasurements

Camera

Presentationscreen

RecommendationEngine

Couch potato

SkipButton

3/29/2011 4

Outline

• Introduction





• Summary


Preliminary study• Online affective annotation

from 120 music clips. The music clips were chosen to cover the whole spectrum of emotions with minimum variance.

• 14+ participants annotated the first set


• One min. emotional highlight from each video

• Content features were used in a linear regression emotion estimation on movie shots:

– arousal features:

• e.g. Audio energy, Motion component, Visual excitement

– Valence features:

• e.g. Color variance, key lighting(Soleymani et al, 2009)

Affect estimation

3/29/2011 EmoSPACE 2011, Santa Barbara 7

Shot affect estimation


• Music clips were segmented into one minute long segments with 55 seconds overlap.

• Affect highlight score computed:

• The segment with the highest score was picked as the highlight

2 2

i i ie v a

Highlight detection


0 5 10 15 20 250

1

2

3

4

5

6

7

8

X: 12

Y: 7.496

hig

hlig

ht s

core

Emotional highlight curve for one clip

segment number

Outline

• Introduction





• Summary


Examples


HALV HAHV

3/29/2011 11

LAHVLALV

Valence

Aro

usal

Apparatus


Recordings


Recorded Dataset

• 32 participants, 16 male and 16 female in two sites aged between 19 and 37 (mean age 26.9),

• 40 videos were shown and 4 questions were asked during self assessments– Arousal

– Valence

– Dominance

– Like/dislike rating


Signals and Stimuli


Stimuli selection


• 40 videos were selected covering V-A plane

3/29/2011 16

DominanceLike/dislike

Outline

• Introduction





• Summary


EEG Features

• Eye blinking artifact was reduced by EOG and ICA

• Alpha (8-12 Hz), beta (12-30Hz), gamma (30-100 Hz) and theta (4-7Hz) power spectral density (PSD) for each electrode

• Lateralization features

– Asymmetry of brain activities caused by emotions

– Subtracted PSD from 14 pairs of electrodesa


Peripheral physiological features

• ECG

– HR, HRV

• EMG and EOG(Zygomatocus major, Trapezius)

– Energy, skewness, standard deviation, kurtosis

• GSR

– Mean, standard deviation, number of peaks,

• BVP

– Mean, standard deviation

• Respiration amplitude

– Central frequency, respiration rate, statistical moments3/29/2011 EmoSPACE 2011, Santa Barbara 19

MCA features

• Visual

– Color variance, motion component, key lighting, etc

• Audio

– Energy, MFCC, pitch, etc

Soleymani, M., Kierkels, J. J. M., Chanel, G., & Pun, T. (2009). A Bayesian Framework for Video Affective Representation. Proceedings of the International Conference on Affective Computing and Intelligent interaction (ACII 2009), Amsterdam, Netherlands.


Emotion detection

• Ridge regression

• Participant dependent

• Leave one out cross validation

• Mean absolute error was computed

• П-random results uses the training set distribution to generate a random estimate

• Pairwise t-test for the significance of superiority over random estimation


Results


0 0.5 1 1.5 2 2.5 3

Arousal

Valence

Dominance

Like/dislike rating

Random П

EEG+MCA

MCA

Peripheral

EEG

Mean Absolute Error (MAE)Mean Absolute Error (MAE)

Outline

• Introduction





• Summary


DEAP* dataset

• Data was recorded in UniGe and UTwente– 32 participants, videos from YouTube

– EEG and peripheral physiological signals

– Face videos for 22 participants

– Continuous self assessments, arousal, valence, dominance, like/dislike ratings

S. Koelstra, C. Muhl, M. Soleymani, J-S. Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras. DEAP: A Database for Emotion Analysis using Physiological Signals IEEE Transactions on affective computing, Special Issue on Naturalistic Affect Resources, under review.

• Available (very soon) at http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

* Database for Emotion Analysis using Physiological signals (DEAP)


http://www.eecs.qmul.ac.uk/mmv/datasets/deap/

Outline

• Introduction





• Summary


Summary

• A data set of emotion music videos were selected

• One minute emotional highlight from each video extracted

• A continuous emotion detection using linear regression was proposed

• A public dataset is developed for researchers in the community


Questions?


Continuous Emotion Detection in Response to Music Videos · Continuous Emotion Detection in...

Documents

Transcript of Continuous Emotion Detection in Response to Music Videos · Continuous Emotion Detection in...