Social Signal Processing: UnderstandingNonverbal Communication in Social Interactions

Alessandro Vinciarelli1,2 and Fabio Valente21University of Glasgow - Sir A.Williams Bldg, G12 8QQ (UK)2IDIAP Research Institute - CP592 Martigny (Switzerland)

e-mail: vincia@dcs.gla.ac.uk, fvalente@idiap.ch

Outline

• Part I - What is SSP?

• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action

• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior

• Machine analysis of social behavior

• Part II - SSP in Action

• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action

• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action

• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action• Analyzing conversations

• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives

• The SSPNet• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives• The SSPNet

• Challenges ahead

Slide 2 of 25

Outline

• Part I - What is SSP?• Nonverbal behavior• Machine analysis of social behavior

• Part II - SSP in Action• Analyzing conversations• Roles, groups, stories and conflicts

• Part III - Future Perspectives• The SSPNet• Challenges ahead

Slide 2 of 25

Part I - What is SSP?

Slide 3 of 25

Social Signals and Social Behaviour

Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.

Slide 4 of 25

Social Signals and Social Behaviour

forwardposture

forwardposture

vocalbehaviour

mutualgaze

interpersonaldistance

NonverbalBehavioural

Cues

height

gesture

Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.

Slide 5 of 25

Social Signals and Social Behaviour

Social Signalforwardposture

forwardposture

vocalbehaviour

mutualgaze

interpersonaldistance

NonverbalBehavioural

Cues

height

gesture

Our attention focuses on words, but we are immersed in a richnon-verbal world influencing not only the meaning of words, butalso our perception of the social context.

Slide 6 of 25

Nonverbal Communication

Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.

Slide 7 of 25

Nonverbal Communication

clothes, attractivenesssomatotype, etc.

self touching

facial expression

prosody, pitch,

postural congruence, etc.

gaze behaviour, etc.

rythm, etc.

distamce, seating

Behaviouralcues

Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.

Slide 8 of 25

Nonverbal Communication

PhysicalAppearance

GesturesPostures

Face and EyesBehaviour

VocalBehaviour

SpaceEnvironment

clothes, attractivenesssomatotype, etc.

self touching

facial expression

prosody, pitch,

postural congruence, etc.

gaze behaviour, etc.

rythm, etc.

distamce, seating

CodesBehavioural

cues

Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.

Slide 9 of 25

Nonverbal Communication

PhysicalAppearance

GesturesPostures

Face and EyesBehaviour

VocalBehaviour

SpaceEnvironment

clothes, attractivenesssomatotype, etc.

self touching

facial expression

prosody, pitch,

postural congruence, etc.

gaze behaviour, etc.

rythm, etc.

distamce, seating

forming impressions

deceiving anddetecting deception

sending messages ofpower and persasion

managing interaction

expressing emotion

Behaviouralcues Functions

Codes

sending relationalmessages

Nonverbal communications is based on nonverbal behavioural cues,codes, and functions.

Slide 10 of 25

Social Signal Processing

DataCapture

PersonDetection

MultimodalBehavioural

Streams

CuesBehavioural

ExtractionSocial SignalsUnderstanding

ContextUnderstanding

BehaviouralCues

SocialBehaviours

Raw Data

Preprocessing

MultimodalBehavioural

Streams

Social Interaction Analysis

- A.Pentland, “Social Signal Processing”, IEEE Signal Processing Magazine,

24(4):108-111, 2007.

- A.Vinciarelli, M.Pantic, H.Bourlard, “Social Signal Processing: Survey of an

Emerging Domain”, Image and Vision Computing, 27(12):1743-1759, 2009.

Slide 11 of 25

Part II - SSP in Action

Slide 12 of 25

A Conversation Analysis Framework

[...] the most widely used analytic approach is based onan analogy with the workings of the market economy. Inthis market there is a scarce commodity called the floorwhich can be defined as the right to speak. Havingcontrol of this scarce commodity is called a turn. In anysituation where control is not fixed in advance, anyonecan attempt to get control. This is called turn-taking.

G.Yule,“Pragmatics”, Oxford University Press (1996)

Slide 13 of 25

Turn-Taking

t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1

The turn taking pattern can be represented with a sequenceS = {(s1,∆t1, ), . . . , (sT ,∆tT )} where si ∈ A = {a1, . . . , aG} is aperson label, and ∆ti the length of the i th turn.

• Among the most robustly detectable behavior evidences,

• but how far can we go by just modeling who talks when andhow much?

Slide 14 of 25

Role Recognition in Broadcast Material

• People play functional roles (Anchorman, Guest, etc.)

• Adjacent speakers are supposed to interact

• Around 85% of data time correctly labeled in terms of role

A.Vinciarelli, IEEE T-Multimedia, 9(6):1215-1226 (2007)

H.Salamin et al., IEEE T-Multimedia, 11(7):1373-1380, 2009

Slide 15 of 25

Role Recognition in Meetings

x1 = (1,1,1,1) x2 = (0,0,1,1) x3 = (1,1,1,0)

w1 w2 w3 w4

a1 2a a3

t1 t 2 t3 t4 t 5 t 6 t7

w1 w2 w3 w4

ts =a2 3

t

s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1

actors

events

• Social networks suitable for functional roles, lexical analysissuitbale for semantic ones

• Affiliation networks are suitable for small groups

• Around 75% of data time correctly labeled in terms of role

N.Garg et al., Proc. of ACM-Multimedia, pp. 693-696 (2008)

Slide 16 of 25

Roles and Prosody

t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1

R∗ = arg maxR∈R

p(R|X , ~α)

• Conditional Random Fields allow the combination ofturn-taking and prosody

• Entropy of main prosodic features

• Results up to 89%, but combination does not always lead tosignificant improvements

H.Salamin et al., Proc. of ACM-Multimedia, to appear (2010)

Slide 17 of 25

Story Segmentation

t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1

Story 1 Story 2 Story 3

The problem consists of finding the following story sequence H:

H = arg maxH∈H

p(X |H)p(H) (1)

• The purity is around 0.75

• Longer stories are better recognized

A.Vinciarelli et al., Proc. of ACM-Multimedia, pp. 261-264 (2007)

Slide 18 of 25

Conflict Analysis

t1 t 2 t3 t4 t 5 t 6 t7 ts =a2 3s1 s3 =a1 s4 =a 3 s5 =a 2 s6 =a 1 s7 =a 2=a1

Q = arg maxQ∈Q

πq1

N∏n=2

p(qn|qn−1) (2)

where qi is one of the two groups or the moderator.

• People tend to react to someone they disagree with

• Groups correctly reconstructued in 66% of the cases (randomgrouping has an average 6.5% performance).

A.Vinciarelli, IEEE Signal Processing Magazine, 26(5):133-136, 2009

Slide 19 of 25

Part III - Challenges Ahead

Slide 20 of 25

Open Issues and Challenges

• Getting psychology and engineering closer

• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches

• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary

• Mutual efforts of both disciplines

• Applying multimodal approaches

• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches

• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches

• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous

• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data

• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data• Artificial settings are sometimes too simple

• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications

• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications• Applications link research to reality

• Applications provide realistic benchmarks

Slide 21 of 25

Open Issues and Challenges

• Getting psychology and engineering closer• SSP is inherently multidisciplinary• Mutual efforts of both disciplines

• Applying multimodal approaches• Social signals are, by evolution, ambiguous• Multimodal approaches are more robust to ambiguity

• Working on real-world data• Artificial settings are sometimes too simple• Social interactions are ubiquitous in many kinds of data

• Identifying relevant applications• Applications link research to reality• Applications provide realistic benchmarks

Slide 21 of 25

The SSPNet

Universita’ di Roma Tre

Queen’s University BelfastUnversity of Edinburgh

Imperial College London

University of TwenteDelft University of Technology

DFKI

Idiap research instituteUniversity of Geneva

CNRS

Slide 22 of 25

Human-Human and Human-Machine Interaction

SSP in Human Machine Interaction

Behavior AnalysisBehavior Modeling

DFKI, CNRS, U. of Twente

Delft, U. of EdinburghIdiap, Imperial College, TUQueen�’s U. Belfast, U. of Roma

Tre, U. of Geneva

Behavior Synthesis

SSP in Human Human Interaction

2 Research Foci: Human-Human and Human-Computer Interaction3 Scientific Domains: Behavior Modeling, Analysis and Synthesis5 Years to go: From 2009 to 2014

Slide 23 of 25

SSPNet: The Portal

The most tangible aspect of the SSPNet will be the web portal:

http://www.sspnet.eu

• lowering the entry barrier of SSP, i.e. to reduce significantlythe effort required to start research in the domain,

• providing common benchmarks for rigorous performanceassessment and comparison between different approaches,

• disseminating literature, data and tools relevant to SSP.

Our ambition is to make of the portal THE reference for SSP.

Slide 24 of 25

Thank You!

Slide 25 of 25