Sam Spaulding - Emotion AI Developer Day 2016
Transcript of Sam Spaulding - Emotion AI Developer Day 2016
![Page 1: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/1.jpg)
Socially Assistive Robots, Educational Tutoring, and Affective Computing
Sam Spaulding MIT Media Lab
![Page 2: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/2.jpg)
Social Robots Inventing our Future while Learning about Ourselves
Robotic Engineering
Artificial Intelligence
Studies of Human Behavior
Human-Robot Interaction
Educational Companion
Aging-in-Place HelperAssembly-mate
![Page 3: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/3.jpg)
Socially Assistive Robots (SAR)
Socially Assistive Robots are designed to leverage their social and affective attributes to provide social support to people in order to sustain engagement, motivate, coach, monitor, educate, or facilitate communication & teamwork for improved outcomes.
![Page 4: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/4.jpg)
![Page 5: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/5.jpg)
Embodiment matters!
Robots produce higher learning
gains
Robots are more able to form long-term
bonds
Robots produce greater
compliance
Leyzberg et al. (2012)
Bainbridge et al. (2012)
Kidd & Breazeal (2008)
Compared to screen-based representations...
![Page 6: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/6.jpg)
Intelligent Tutoring Systems
Use domain-general inference and modeling algorithms
Have been extensively tested in real-world environments over long periods of time
![Page 7: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/7.jpg)
Intelligent Tutoring SystemsRobotic Tutors
The best of both...
Social PresenceOnboard Sensing
Agent-based Interaction
Adaptive PersonalizationData-driven Student Models
Physically embodied robot tutors that: sense and understand emotions
build models of students based on affective data act intelligently as a result of the model info
![Page 8: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/8.jpg)
Affect-aware Student Models for Robot Tutors
- Child and Robot interact through shared “Storymaker” game context
- Robot framed as peer, periodically asks child to demonstrate reading ability
(to be presented at AAMAS ’16)
![Page 9: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/9.jpg)
Knowledge State Estimation
- A key challenge for adaptive, computational tutors is - “how to personalize experience?”
- In order to provide personalized curriculum, the tutor must first determine students’ initial knowledge state
Task Difficulty
Student Ability
Boredom
Overwhelmed
Flow: Optimal Challenge
“Flow”
![Page 10: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/10.jpg)
Bayesian Knowledge Tracing (BKT)
Bayesian Knowledge Tracing (BKT)
P(Lit)
Correctt
P(Lit+1) P(Li
t+2)... ...
Correct t+1 Correct t+2
Background Model Domain Evaluation Contributions
Sparse channel for knowledge, but widely studied
Each ‘traced’ skill modeled by an HMM
![Page 11: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/11.jpg)
Affective Bayesian Knowledge Tracing
Affective data drawn from 5s before question asked to 5s after question answered
![Page 12: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/12.jpg)
Affect-aware Student Models for Robot Tutors
- 25 children came and played with Dragonbot
- 13 children did same interaction with Tablet only
- Experiment conducted in Summer 2014, affective analysis completed 6mo. later
(to be presented at AAMAS ’16)
![Page 13: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/13.jpg)
Are Children More Emotionally Expressive When Interacting with A Robot?
... ... ... ...} {Avg Smile: 18Avg BrowFurrow: 7Avg BrowRaise: 62Avg LipDepress: 4Avg Valence: -24Avg Engage: 67
} {Median Filter } {Mean
Metric Value
Session FootageRaw Affdex MeasurementsMedian-smoothed Affdex Data
Subject Bag
Average Metric Value over Interaction
![Page 14: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/14.jpg)
Are Children More Emotionally Expressive When Interacting with A Robot?
*
**
* p < .05
n=25 “Robot” conditionn=13 “Tablet condition
![Page 15: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/15.jpg)
Training and Evaluating Skill Models
BKT and Aff-BKT models trained for 4 Skills via Expectation Maximization
Model classes evaluated via log-likelihood comparison, with Leave-one-out cross-validation
![Page 16: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/16.jpg)
Does Including Affective Data in Training Yield Better Models?
... }Affective + Right/Wrong Training Data
Subject Bag
Expectation Maximization {
P(Lit)
CorrecttSmilet
P(Lit+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...
P(Lit)
CorrecttSmilet
P(Lit+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...}Training Data Trained Model Trained Model
Subset
{Model Subset
P(Lit)
CorrecttSmilet
P(Lit+1)
...
Engagedt Correctt+1Smilet+1 Engagedt+1
...
P(Lit)
Correctt
P(Lit+1)
...
Correctt+1
...
Held-out Right/Wrong Test Data, D'
Likelihood of Model, Given Test Data D’
}D’ P(D’| θaff)
θaff = maxθ P(Daff|θ)θaff, a subset of θaff, containing only
BKT model parameters
θaff
^
^
^
Daff
![Page 17: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/17.jpg)
Does Including Affective Data in Training Yield Better Models?
[ [ [ [
Exact-Correct
BKT Aff-BKT BKT Aff-BKT BKT Aff-BKT BKT Aff-BKT
First-Letter Length Last-Letter
![Page 18: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/18.jpg)
Does Including Affective Data in Training Yield Better Models?
p < 1.0 x 10-4
*
*
***
***
**p < 1.0 x 10-5
p < 1.0 x 10-6
*
***
![Page 19: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/19.jpg)
Are children more emotionally expressive when interacting with robots?
Can we leverage emotional expression data to create better student models?
Two main questions:
![Page 20: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/20.jpg)
Are children more emotionally expressive when interacting with robots?
Can we leverage emotional expression data to create better student models?
Two main questions:
![Page 21: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/21.jpg)
Intelligent Tutoring SystemsRobotic Tutors
The best of both...
Social PresenceOnboard Sensing
Agent-based Interaction
Adaptive PersonalizationData-driven Student Models
Physically embodied robot tutors that: sense and understand emotions
build models of students based on affective data act intelligently as a result of the model info
![Page 22: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/22.jpg)
Meet Tega!
![Page 23: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/23.jpg)
Tega: a “real-world ready” social robot!
Student models allow us to personalize curricular content. How do we personalize affective support?
Affective Personalization
![Page 24: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/24.jpg)
Affective Personalization of a Social Robot Tutor for SSL
(presented at AAAI ’16)
•What’s it all about?•Children learning Spanish as a second language with a robot companion
•An Integrated System•Tega Robot •Custom Educational Game• Affdex affective sensor •All synchronized and coordinated through a ROS-based cognitive architecture
•The Study•Long-term, in-the-wild, fully autonomous interaction•Personalization of affective response
![Page 25: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/25.jpg)
Integrated System: Software
Game:- Unity-based sprite game- 8 sessions of content + review- Fully autonomous play- Virtual “instructor” character, robot as peer
Affdex phone:- Real-time detection of facial
expressions- Valence / Engagement used as
reward to RL algorithm
![Page 26: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/26.jpg)
Educational Context
Robot again framed as peer, with ‘bilingual’ Toucan as teacher
As the student plays through the game, the robot provides affective support through verbal + nonverbal actions
![Page 27: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/27.jpg)
Reinforcement Learning on Affective Data
SARSA Algorithm
Reward = .4(Engagement) +.6( (Valence+100) )2
State Space = 3 x 2 x 2 x 2 = 24 states total
Neg./Neut./Pos. Valence
Hi/LoEngagement
On/OffTask
Right/Wrong Last Question
Action Space = 3 x 2 + No-action = 7 action classes total
ɛ-greedy algorithm, with ɛ decreasing across sessions
![Page 28: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/28.jpg)
Sample Actions
![Page 29: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/29.jpg)
Timeline• 8 Weeks of In-Class deployment• 1 Pre-test Session• 6 Study Sessions (part review, part new content)• 1 Post-test Session
![Page 30: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/30.jpg)
Affective Personalization
***
***
*
***
![Page 31: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/31.jpg)
Affective Personalization
*
Appropriate affective responses are critical to avoiding “novelty effect”
![Page 32: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/32.jpg)
Did they actually learn?
*
![Page 33: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/33.jpg)
Contributions
• Novelty of current study:• Long-term interaction (8 sessions)• Fully autonomous social robot (Tega)• In-the-wild experiment (inside a classroom)• Affective personalization (Affdex)• Age of participants (3-6 yrs)
Social robot personalized its affective response, thus increasing children’s valence during long-term
interaction.
![Page 34: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/34.jpg)
Looking ahead…
Old challenges are becoming tractable: sensing, deployment, robust systems.
New challenges are conceptual and computational - i.e. how to fully integrate emotions into an agent’s cognition
![Page 35: Sam Spaulding - Emotion AI Developer Day 2016](https://reader030.fdocuments.net/reader030/viewer/2022020213/587021851a28ab7f428b80c5/html5/thumbnails/35.jpg)
Collaborators and Supporters
This research was supported by the National Science Foundation(NSF) under Grant CCF-1138986 and Graduate ResearchFellowship Grant No 1122374.
Luke Plummer JinJoo Lee
Goren Gordon Jacqueline KoryProf. Cynthia Breazeal