Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener.

Jaewoo Chung • Scotty Vercoe • MAS630/Affective Computing • Spring '05

The Affective Listener



“How remarkable would it be if one could experience and express the spectrum of emotions embodied in music originating from oneself, without the crutch of a composer’s intercession…Can the touch that lies behind music be tapped?”–Manfred Clynes



Goal:• Direct music using affective cues

Issues:• Devise mapping scheme of music parameters• Correlate affective signals with music parameters• Disambiguate data collected during music listening• Develop algorithm to navigate music map by affect



Presentation Outline

1. Existing Work2. Project Overview:

• Music Parameters

• Affective Signals3. Listening

Experiment4. Data analysis5. Algorithm6. Conclusions7. Demonstration



Existing ResearchBerlyne (1974):• liking is greatest for stimuli of moderate arousal• arousal and liking should be dependent; familiarity and complexity affect arousal levelRussell et al., (1981): Circumplex model of emotion• puts emotions in a circle around grid of pleasant-unpleasant and arousing-sleepyNorth and Hargreaves (1997):• correlated circumplex model to valence/arousal dimensions• replaced pleasant-unpleasant axis with like-dislike axis in circumplex• coordinates of liking-arousal was found to be a reliable predictor of emotional reactionSchubert (1996):• people often enjoy music that is unpleasantRitossa and Rickard (2004):• tested dimensions of pleasantness and liking in circumplex• one of eight emotions reliably predicted using arousal, familiarity & pleasantness• pleasantness was more useful predictor of emotions than liking



Existing ResearchMarrin and Picard (1998): Conductor’s Jacket• records a variety of affective signals during conductingHealey, Picard and Dabek (1998): Affective DJ• uses a mixture of physiological and subjective data• compares SCR from first 30 seconds of current song to last 30 seconds of previous song• algorithm selects songs to change from current affective state to desired stateKim and André (2004): Composing Affective Music• uses ECG, EMG, SCR and RESP data• composes algorithmically• SCR is useful indicator for unsettling-relaxing• EMG useful indicator for positive-negative



Music AffectMapping

Music ParameterMapping

user study

music box

Project Overview1) Small-scale listening

experiment:Change music parameters, and

observe physiological and self-report data.

2) Music generator:Develop real-time algorithm that

modifies music parameters based on affect.



Music AffectMapping


user study

music box

layering

complexity

instrument layering:frequency-domain densitycomplexity: time-domain density

Music Parameter Mapping



Music AffectMapping


user study

music box

arousal

valence

arousal: reaction level to musicvalence: subject’s like/dislike of music

Music Affect Mapping



higharousal

like

Music Affect Mapping

lowarous

al

dislike

engagingannoying

soothingboring



higharousal

like

The Challenge

lowarous

al

dislike

engagingannoying

soothingboring

Current State Goal State



The Challenge

engagingannoyingCurrent State Goal State

engagingboring

engagingsoothing

soothingannoying

soothingboring



Changing Music Affect

engagingannoyingengagingboring

engagingsoothing

soothingannoying

soothingboring

MusicParameters Probability

?f-1, t-1

f-1, t+1

f+1, t-1

f+1, t+1

Start State End State

• Given set changes in parameters, what are the trends for motion in

affective space?• How do those trends change depending on the initial affective state?



Listening Experiment

Preparation of Music Clips:

• Five pieces with constant tempo were composed, ranging from jazz, rock to electronic music.

• For each piece, looping audio segments were produced to reflect the layer/complexity map.

• Segments were arranged into a 4x4 matrix for each piece.• For each piece, four clips were assembled by traversing the matrix

as follows:1. Increasing complexity2. Increasing instrument layering3. Decrease complexity4. Decrease instrument layering

((•Listen•))




Preparation for Data Collection:

Physiological Data• Galvactivator set up to measure GSR• microphone connected to measure presence of foot-tapping

Self-report Data• dual 7-point scales were set up for subject’s affective response self-report• two sets were prepared for subjects to report their initial and final

reactions




Conducting the Experiment:

• 8 participants: 6 male, 2 female• 20 audio clips (25-45 seconds each) were played for each subject• experiment lasted approximately 25 minutes

Physiological Data• GSR was sampled every second throughout entire listening, • BPM and velocity of foot-tapping was sampled every second, and presence

of foot-tapping was sampled every 2 seconds

Self-report Data• subject self-reported affective response twice during each clip (initial &

final)



Data Analysis

90

92

94

96

98

100

102

104

1 24 47 70 93 116 139 162 185 208 231 254 277 300 323 346 369 392 415 438 461 484 507 530 553 576 599 622 645 668 691 714 737 760 783 806 829 852 875 898

46

48

50

52

54

56

58

60

1 39 77 115 153 191 229 267 305 343 381 419 457 495 533 571 609 647 685 723 761 799 837

Total Physiological Response - Subject 3

Total Physiological Response - Subject 4



Data Analysis

Subject #2: physiological data

MUSIC CLIP #9: Increase Complexity

layering

complexity

0

2

4

6

8

10

12

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

Subject #2: self-report data

like

annoying engaging

dislike

boring soothing



Data Analysis

Subject #6: physiological data

MUSIC CLIP #9: Increase Complexity

layering

complexity

Subject #6: self-report data

like

annoying engaging

dislike

boring soothing

0

2

4

6

8

10

12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

UNUSUAL EXAMPLE:



Data Analysis

-6

-4

-2

0

2

4

6

-6 -4 -2 0 2 4 6

-6

-4

-2

0

2

4

6

-6 -4 -2 0 2 4 6

-6

-4

-2

0

2

4

6

-6 -4 -2 0 2 4 6

-6

-4

-2

0

2

4

6

-6 -4 -2 0 2 4 6

engaging

soothingboring

annoying

presence of foot-tapping

GSR



Data Analysis

GSR fallingGSR rising

S1: engaging S2: soothing S3: boring S4: annoying



Data Analysis

s1 s2 Sub totalPositive valence GSR UP 20 11 31

GSR DOWN 46 17 630.65 0.35 10.73 0.27 1

s3 s4 Sub totalNegative valence GSR UP 9 10 19

GSR DOWN 8 18 260.47 0.53 10.31 0.69 1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 2

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1 2

GSR: up down

GSR: up down


S1

S2

S1

S2

S3

S4

S3

S4



Algorithm Overview

A pair of state transition models was constructed based on physiological and self-report data:

The first model tries to detect the listener’s current affective state.

The second model chooses the direction most likely to induce the goal state.



Probability Table for Detecting Affective State

s1 s2 Sub totalPositive valence GSR UP 20 11 31

GSR DOWN 46 17 630.65 0.35 10.73 0.27 1

s3 s4 Sub totalNegative valence GSR UP 9 10 19

GSR DOWN 8 18 260.47 0.53 10.31 0.69 1




Affective State Transition Model

ACTION: Increase Complexity

layering

complexity

S1: engagingS2: soothingS3: boringS4: annoying



ACTION: Increase Layering

layering

complexity





ACTION: Decrease Complexity

layering

complexity





ACTION: Decrease Layering

layering

complexity





Summary of State Transitions

0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4

Series1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4

Series1

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1 2 3 4

Series1

S1-engagingS4-annoying

S3-boring

S1 S2 S3 S40

0.1

0.2

0.3

0.4

0.5

1 2 3 4

Series1

S2-soothingS1 S2 S3 S4 S1 S2 S3 S4

S1 S2 S3 S4



Summary of State Transitions




System Diagram

Physiological data input

Control

Affective statedetection algorithm

Direction choosing algorithm

Change music parameter

Goal stateInitial state

Action

Music

Affect induced

Perception


The Affective ListenerConclusions

• changes in music parameters can be correlated to affective response to music• Markov chains are a useful tool for constructing a predictive listening system• specific observations about mapping layering/complexity to arousal/valence: • engaged listeners tend to stay engaged • annoyed listeners tend to stay annoyed • soothed listeners tend to stay soothed, but also easily bored or engaged • bored listeners tend to become interested by any change in parameters • annoyed listeners are more likely to be engaged if first induced to boredom


The Affective ListenerFuture Work

• improve accuracy of predictions by incorporating more user data• improve affective state predictions using additional affective signals• apply affect-parameter mapping to algorithmic composition• use machine learning to customize predictions to individual subject


The Affective ListenerReferences

Berlyne, D.E. (1974) Studies in New Experimental: Steps Towards an Objective Psychology of Aesthetic Appreciation. Halstead Press.

Clynes, M. (1977) Sentics. Anchor Press.Healey, J., Picard, R. and Dabek, F. (1998) ‘A new affect-perceiving interface and its

application to personalized music selection,’ Proc. of the Perceptual User Interfaces Workshop, 4-6.

Kim, S. and André, E. (2004) ‘Composing affective music with a generate and sense approach,’ Proc. of Flairs 2004. American Association for Artificial Intelligence.

Marrin, T. (2000) Inside the Conductor’s Jacket: Analysis, Interpretation and Musical Synthesis of Expressive Gesture. PhD thesis, MIT Media Lab, Cambridge, MA.

Meyer, L. (1956) Emotion and Meaning in Music. University of Chicago Press.North, A.C. and Hargreaves, D.J. (1997) ‘Liking, arousal potential and the emotions

expressed by music’, Pscyhomusicology 14:77-93.Ritossa, D. and Rickard, N. (2004) The relative utility of ‘pleasantness’ and ‘liking’

dimensions in predicting the emotions expressed by music. Psychology of Music. v.32(1):5-22

Russell, J.A. (1980) ‘A Circumplex Model of Affect’, Journal of Personality and Social Psychology 39: 1161-78.

Rutherford, J. and Wiggins, G.A. (2002) ‘An experiment in the automatic creation of music which has specific emotional content,’ Proc. for the 7th International Conference on music Perception and Cognition, Sydney, Australia.

Schubert, E. (1996) ‘Enjoyment of negative emotions in music: an associative network explanation’, Pscyhology of Music 24: 18-28.

Sloboda, J.A. and Juslin P.N. (2001) Music and Emotion: Theory and Research. Oxford University Press.

Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener.

Documents

Transcript of Jaewoo Chung Scotty Vercoe MAS630/Affective Computing Spring '05 The Affective Listener.