Human Social Interactionperspectives from neuroscience

Dr. Roger NewportRoom B47

Student Drop-in Time: Tuesdays 12-2

www.psychology.nottingham.ac.uk/staff/rwn

Understanding Emotion: auditory emotion recognition

1

This lecture:Recognising emotion from prosodyTheories of emotion processing Current Research

Understanding emotion from auditory cues: prosody

Prosody is the melody or musical nature of the spoken voice - conveyed by changes in e.g pitch, syllable duration, volume

We are able to differentiate many emotions from prosody alone e.g. anger, sadness, happiness

Universal and early skill

What are the neural bases for this ability?

Are they the same as for language? Are they the same as for differentiating emotion from visual cues?

2

We can tell the difference between a spontaneous and a mechanical smile

We think we can tell something about size and attractiveness from the sound of someone’s voice

Babies can differentiate between the sounds of voices before they can understand speech

We can understand cartoon characters even though they do not speak

Children can produce the melody or intonation of speech before they can produce two-word combinations

Prosody skills - innate and special abilities 3

Uses of prosody in the film industry

Trombone Toy whistle BadgerWalrusMad DogStarving Bear

4

Emotion from prosody - the same as from facial expression?

Emotion from facial expressions reminder from last week:

Clear evidence for involvement of amygdala in fear recognition and the insula / striatum in the recognition of disgust

Not so clear for other emotions

Does the same hold true for recognising emotions from the expression of the voice?

5

Emotion from prosody, the same as from facial expression?

The insula and disgust

Most emotions identified at 50-60% accuracy (chance = 10-20%)

Disgust is nearly impossible to recognise from prosody

Very very difficult to measure in experimental conditions

Therefore very little successful research on recognition of disgust from prosody

6

Evidence from human neuropsychology - the amygdala and fear

Adolphs et al, 2001Recognition of emotion from prosody in 15 unilateral left, 11 unilateral right amygdala and 50 BD controls.No differences between groups(But bilateral amydala damage is usually necessary to abolish facial fear recognition)

Adolphs and Tranel, 19992 complete bilateral amygdala (inc. SM), 15 BD and 14 NC controls. SM normal emotion recognition from prosody

Amygdala’s role not as critical for prosody as facial emotion

Which regions might be involved? Are prosody and facial skills dissociable?

7

Evidence from human neuropsychology for right hemisphere involvement - Patient KB and amusia

Music activates brain regions that are also associated with emotion processing

Trained musicians are better at identifying emotion from prosody and tonic imitations of prosody

Children who study keyboard vs drama/nothing are also better

KB: amusic following RH stroke unable to discriminate pitch or rhythm patterns in linguistic or musical stimuli. Also impaired on prosodic perception tasks (e.g., discriminating statements from questions)

8

Evidence from human neuropsychology - hemispheric asymmetry

Barrett, et al., (1999) - patient with large left hemisphere lesionNormal emotional prosody yet severe inability to process propositional speech.

Schmitt, et al., (1997)27 RH patients; 25 LH disproportionately impaired recognition of emotion from facial and prosodic cues in the RH group when judging multimodal video clips

Peper and Irle (1997) RH disproportionatelyimportant for processing of emotion from prosody

Pell (2005) RH (mixed damage!) impaired at emotion from prosody; LH impaired at interpreting prosodic code within language content

9

Adolphs et al., 2002:Lesion analysis66 brain damaged subjects

Brain Damaged Patient Groups Human imaging (TMS) 10

Van Rijn et al. 2005

1Hz TMS for 12 minutes

Somatosensory area associated with lips/tongue/jaw

Slowed RTs to ‘withdrawal’ emotions (fear/sadness), but not approach (e.g. happy)

Human imaging of voice selective cortex

Belin et al., 2000

11

BellsHuman non-vocal soundsNoise (amplitude modulated)Vocal soundsScrambled voices

Friederici and Alter (2004) prosodic data adapted from Plante et al. 2001

Using fMRI, Buchanan et al. showed that the detection of emotional prosody is associated with increased activation in the right hemisphere (inferior frontal lobe and right anterior auditory cortex),

Human imaging of voice selective cortex 12

Wildgruber et al., 2005 fMRI experiment

Designed to separate phonetic from affective prosodic components

Emotionally neutral spoken sentences such asDer Gast hat sich für Donnerstag ein zimmer reserviert orDie Anrufe werden automatisch beantwortet

Read with 5 different emotions (happiness, sadness, anger, fear and disgust)

Tested for recognition behaviourally prior to fMRI(all 90-95% accuracy except disgust (77%).

2 tasks: say the emotion; say the vowel after the first ‘a’.

13

fMRI results:Disgust (and fear) recognition dropped to near 50% accuracy

14

* NS

The trouble with fMRI: both tasks involve listening and automatic processing of linguistic, syntactic, phonological and prosodic information as well as motor responses

So get activation of auditory cortex, phonolgical store, supplementary motor areas etc.

15

Use subtraction method:

2 areas associated with emotion identification: Right STS and right inferior frontal cortex. rIFC involved in emotion comprehension of both facial and prosodic cues.

16

Attend to left or right ear. Make gender judgment

Sander et al., 2005

Prosody, anger and attention

17Imaging of specific emotions

The recognition of emotion from prosody is not analogous to the recognition of emotion from facial expression

Recognizing emotional prosody draws on multiple structures distributed between left and right hemispheres

The roles of these structures are not all equal but may be most apparent in processing specific auditory features that provide cues for recognizing the emotion

Despite the distributed nature of the processing, the right hemisphere appears most critical - in particular the right inferior frontal regions, working together with superior temporal region in the right hemisphere, the left frontal regions, and subcortical structures, all interconnected by white matter.

Summary 18

Break

So far we have looked at

Types of emotion expression (facial/prosodic)

Brain regions associated with various emotions/types of expression

So we know what the brain does with emotionally expressive stimuli and where might be important for doing this, but we have not looked at HOW the brain might process emotionally expressive stimuli

Broadly speaking there are 2 main theories of emotion processing:

1. Theory theory2. Simulation theory

20

Theory theory:

Children as young as 5 have extensive causal knowledge in the form of intuitive theories

Screening off task

21

visually obtained knowledge of the facial configuration of the target

semantic knowledge concerning facial configurations

general knowledge concerning a given emotion, i.e. its typical elicitors or behavioural effects

knowledge that facial configuration ‘C’ is paired with emotion label ‘E’

Theory Theory of emotion recognition

information-based account

employs naiive psychological ‘folk’ theory to infer the emotional states of others

22

Simulation theory

We interpret the emotions of others by covertly simulating their response matching the outcome to our knowledge of outcomes

How might this work for emotion recognition?

Person A sees person B pulling facial configuration ‘C’A covertly facsimiles ‘C’ (or what she thinks ‘C’ to be)A attributes resulting emotion label ‘E’ to person B

23

Paired emotion deficits on FaBER tasks

E.g. patients SM and NM (Fear)

SMBilateral amygdala damage, neighbouring areas sparedDid not express fear, could not recognise expression of fear in others

Knew what fear was supposed to beKnew what should cause itEven knew what response might beBut could not show it.Unable to learn the significance of unpleasant situationsDoes not show fear conditioning

Evidence for simulation theory account of emotion recognition

Paired deficits for fear

24

Paired deficits for disgust

Imaging studiesPhilips et al, 1997, 1998 fMRIObserving FaBER disgust activates right insulaInsula known to involved in experience of unpleasant tastes and smells

Wicker et al., 2003. fMRI watching video of facial expressions in response to pleasant or disgusting smell vs. experiencing pleasant or disgusting smellBoth disgust conditions preferentially activated left ant. insula and right ant. cingulate cortex.

Patient studiesNK (Calder et al., 2000) Insula and BG damage. Paired impairment on disgust measures (questionnaire and FaBER)

25

Dopamine system: a neural subsystem involved in the processing of aggression in social-agonistic encounters in a wide variety of species

Dopamine system plays an important role in mediating the experience of the emotion of anger.

Dopamine levels in rats and other species are elevated in social-agonistic encounters.

Administration of dopamine antagonist (e.g. sulpiride) selectively impairs responses to agonistic encounters.

Sulpiride administration leads to selective disruption of FaBER for anger, while sparing recognition of other emotions.

Following sulpiride administration, subjects were worse at recognizing angry faces, but no such FaBER impairments of other emotions.

Paired deficits for anger 26

Generate and Test

A deficit in the production of an emotion (or its facsimile) leads to an impairment in the recognition of that emotion

How is this done?Random? Too slowTheory? How?

How is this done?V to P matchingLearnt or innate?

27

Generate hypothesised emotion

Produce a facial expression

Test expression(does it match expression of

other?)

Classify own emotional state and attribute this to other

No

Yes

Activation of facial musculature precedes emotion

Reverse simulation

28

Visual representation of other’s facial expression

Activation of facial muscles that imitate other’s facial expression

Experience of emotion

Classify own state and attribute that to other

Such imitation is innate+ adults covertly mimic FaBER stimuli (measurable by EMG) (Dimberg et al.)

Engages cognitive processes in reverse.Emotional state and facial expression are bidirectionalSensation of emotion --> facial expressionFacial expression --> mild sensation of emotion

Reverse simulation

A direct link from visual input of other’s face to somatosensory representation of what it would feel like to pull that face. Bypasses muscle activation so gets round Mobius problem, but no details of how link might work. 29

Visual representation of other’s facial expression

Activation of facial muscles that imitate other’s facial expression

Experience of emotion

Classify own state and attribute that to other

As if

Observation of other’s facial expression

Automatic activation (mirroring) of neural systems associated with facial emotion

Shared emotion Labelling of emotion

Unmediated resonance model or shared manifold hypothesis

Direct activationRequires no mediating structures or processes

30

Wild et al., 2003

You are slower to make an incongruent facial movement than a congruent one

Contagious emotions

31

visually obtained knowledge of the facial configuration of the target

semantic knowledge concerning facial configurations

general knowledge concerning a given emotion, i.e. its typical elicitors or behavioral effects

knowledge that facial configuration ‘C’ is paired with emotion label ‘E’

Theory Theory of emotion recognition

Can you lesion this model?

32

Summary

Research into understanding emotions has revealed that deficits in face-based recognition are paired with deficits in the production of the same emotion.

Of theory and simulation approaches the simulation theory seems to offer the best explanation of the data.

The precise mechanisms by which simulation theory might work are still unclear, but reverse models(with as if loop) and the more recent unmediated resonance model can both account for neuroscientific data

But some people still don’t believe it

33

Audiovisual integration of emotional signals in voice and face:An event-related fMRI study

There has been plenty of research into visual emotion recognition

And some research into auditory emotion recognition

But (almost) no-one has studied audiovisual integration of (dynamic) emotional stimuli

Behavioural and Event-related fMRI taskViewed and heard faces and words either A only, V only or AVNeutral + 6 basic emotions (surprise!)Looked for areas of AV overlap that were not A or V (AV>A)∩(AV>V)

Current Research July 2007

Rationale

Methods

34

HappySadFearDisgustAnger?

Results

Imaging results:Bilateral posterior superior temporal gyrus activation and right thalamus (not shown)

Behavioural results:People were better when both A and V information available

This is what they predicted (honest).35

Amygdala damage impairs emotion recognition from music

We know the amygdala is implicated in the recognition of fear (faces)

Patients following temporal lobe removal (inc. amygdala) are impaired at scary music recognition.

Is the amygdala (specifically) necessary for scary music recognition?

Current Research September 2006

Rationale

Methods

Neuropsychology (Patient SM)Music discrimination and emotion recognition /rating tasks

Results

SM ok at discriminationPoor at negative emotion music recognition (sad and scary) 36

Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing. TICS

Current Research September 2006

Review article

37

Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing

Current Research September 2006

Review article

38

Beyond the right hemisphere: brain mechanisms mediating vocal emotional processing

Current Research September 2006

Review article

39

Behold the voice of wrath: Cross-modal modulation of visual attention by anger prosody. Brosch et al. 2008

Current Research 2008

We know the amygdala directs our attention to visual socially relevant stimuli

We also know that the amygdala responds to anger prosody

Does anger prosody direct our visual attention?

Rationale

Methods

Dichotic listening with cueing paradigm

Results

Yes it does. Visual targets detected faster when on same side as anger delivered 40

Emotional prosodic processing in auditory hallucinations

Current Research 2007

Schizophrenics are impaired at prosody recognition

Prosodic cues are important for speaker identity

Could prosodic deficit be responsible for misattribution of voices in auditory hallucinations

Rationale

Methods

Rate emotional (but semantically neutral) spoken sentences from Sad to Happy on Likert scaleGroups: normal, Schz with hallucinations, Schz without

ResultsOnly hallucinating patients were impaired compared to controls

41

Next lecture: revision/FAQ lecture on the 3rd of November.

What to do now.

Don’t panic (I’ll tell you when).

Read some articles and start planning and writing up an experiment

Submit revision questions using feedback page before revision/feedback lecture on November the 3rd