Modelling and Analyzing Multimodal Dyadic Interactions
Using Social Networks
Sergio Escalera, Petia Radeva, Jordi Vitrià, Xavier Barò and Bogdan Raducanu
Outline
1. Introduction
2. Audio – Visual cues extraction and fusion
3. Social Network extraction and analysis
4. Experimental Results
5. Conclusions and future work
1. Introduction
- Social interactions play a very important role in people’s daily lives.
- Present trend: analysis of human behavior based on electronic communications: SMS, e-mails, chat
- New trend: analysis of human behavior based on nonverbal communication: social signals
- Quantification of social signals represents a powerful cue to characterize human behavior: facial expression, hand and body gestures, focus of attention, voice prosody, etc.
Social Network Analysis (SNA) has been developed as a tool to model social interactions in terms of a graph-based structure:
- ‘Nodes’ represent the ‘actors’: persons, communities, institutions, etc.
- ‘Links’ represent a specific type of interdepency: friendship, familiarity, business transactions, etc.
A common way to characterize the information ‘encoded’ in a SNA is to use several centrality measures.
Our contribution:
- In this work, we propose an integrated framework for extraction and analysis of a SNA from multimodal (A/V) dyadic interactions*
- The advantage is represented by the fact that it is based on a totally non-intrunsive technology
- First: we perform speech segmentation through an audio/visual fusion scheme- In the audio domain, speech is detected through clusterization of audio features
- In the visual domain, speech is detected through differential-based feature extraction from the segmented mouth region
- The fusion scheme is based on stacked sequential learning
*We used a set of videos belonging to the New York Times’ Blogging heads opinion blog. The videos depict two persons talking on different subject in front of a webcam
Block-diagram representation of our integrated framework
- Second: To quantify the dyadic interaction, we used the ‘Influence Model’, whose states encode previously integrated audio-visual data
- Third: The Social Network is extracted based on the estimated influences* and its properties are characterized based on several centrality measures.
* The use of term ‘influence’ is inspired by the previous work of Choudhury:
T. Choudhury, 2003. “Sensing and Modelling Human Networks”, Ph.D. Thesis, MIT Media Lab
2. Audio – Visual cues extraction and fusion
• Audio cue– Description
• 12 first MFCC coefficients• Signal energy• Temporal cepstral derivatives (Δ and Δ2 )
• Audio cue– Diarization process
• Segmentation– Coarse segmentation according Generalized Likelihood
ratio between consecutive windows
• Clustering– Agglomerative hierarchical clustering with a BIC stopping
scheme
• Segments boundaries are adjusted at the end
• Visual cue– Description:
• Face segmentation based on Viola-Jones detector• Mouth region segmentation• Vector of HOG descriptors for for the mouth region
• Visual cue– Classification:
• Non-Speech class modelling• One-class Dynamic Time warping based on the
following dynamic programming equation
• Fusion scheme– Stacked sequential learning (suitable for
problems characterized by long runs of identical labels)
• Fusion of audio-visual modalities• Determining temporal relations of both feature sets
for learning a two-stage classifier (based on Ada-Boost)
3. Social Network extraction and analysis
- Influence Model (IM), was a tool introduced for quantification of interacting processes using a coupled Hidden Markov Model (HMM)
- In the case of social interaction, the states of IM encode automatically extracted audio-visual features
Influence Model Architecture
parameters represent the ‘influences’
- The construction of the Social Network is based on ‘influences’ values
- A directed link between two nodes A and B (designated by A → B) implies that ‘A has influence over B’
- The SNA is based on several centrality measures:- degree centrality (indegree and outdegree) - Refers to the number of direct connections with other persons
- closeness centrality
- Refers to the facility between two persons to communicate
- betweeness centrality
- Refers to the relevance of a person to act as a ‘bridge’ between two sub-groups of the network
- eigenvector centrality
- Refers to the importance of a person in the network
4. Experimental results
- We collected a subset of videos from the New York Blogging Heads’ opinion blog
- We used 17 videos from 15 persons- Videos depict two persons having a conversation in front of
their webcam on different topics (politics, economy,…)- The conversations have an informal character and
sometimes frequent interruptions can occur
Snapshot from a video
- Audio features - The audio stream has been analyzed using sliding windows of 25 ms with
an overlapping factor of 50%.- Each window is characterized by 13 features (12 MFCC +E), complemented with Δ and Δ2 - The shortest length of a valid audio segment was set to 2.5 ms
- Video features- 32 oriented features (corresponding to the mouth region) have been extracted using the HOG descriptor
- the length of the DTW sequences has been set to 18 frames (which corresponds to 1.5 s)
- Fusion process- stacked sequential learning was used to fusion the audio-visual features - Adaboost was chosen as classifier
Visual and audio-visual speaker segmentation accuracy
The extracted social network showing participants’ label and influence directions
Centrality measures table
5. Conclusions and future work
- We presented an integrated framework for automatic extraction and analysis of a social network from im-plicit input (multimodal dyadic interactions), based on theintegration of audio/visual features.
- In the future, we are planning to extend the current work to study the problem of social interactions at larger scale and in different scenarios
- Starting from the premise that people's lives are more structured than it might seem a priori, we plan to study long-term interactions between persons, with the aim to discover underlying behavioral patterns present in our day-to-day existence
Top Related