CPSX: Measurement of Collaborative Problem Solving Skills

saad.khan@act.org

Alejandro Andrade, Pravin Chopade, David Edwards,

Scott Pu and Bryan Maddox

2

Competencies to Succeed in the 21st Century

Challenge of Evidence Identification

3

AI based innovations in Evidence Identification

4

Body Posture:

Relaxed

Facial Expression:

Smiling, Eye Contact

Voice:

Calm

Overall

State:

Engaged

Multimodal Analytics (NLP+CV+ML)

AI based innovations in Evidence Identification

5

High-level inferences

Mid-level representations

Low-level features

SVM HMM

HOG MFCCEuler Angles

Multimodal data

Video 3D Audio

Dat

a Th

eory

Bayesian Networks, JHCRFs

Speech, Prosody, Facial Expressions, Gestures

Turn-taking, Engagement, Affect

Collaborative Problem Solving

Knowledge Assimilation

Positive Communication

Resource Sharing

Deep Hierarchical Inference Models

Khan (2017). In Innovative Assessment of Collaboration. Springer.

Measuring Collaborative Problem Solving Skills

Using Games for Education?

Collaborative Game: Crisis in Space (CIS)

Operator Engineer

Player Interfaces

Gameplay SegmentsG

ame

Lau

nch Select Role

Create Match

Complete Tutorial

Mis

sio

n 1 Wires

Symbols

Intermission

Mis

sio

n 2 Wires

Symbols

Asteroid

Intermission

Mis

sio

n 3

Symbols

Asteroid

Maze

Intermission

Mis

sio

n 4 Wires

Asteroid

Maze

Solar Panel

Intermission

Mis

sio

n 5

Symbols

Asteroid

Maze

Satellites

Done

The game is divided into Five missions – a total of 16 tasks must be completed

Players alternate roles between missions

Operator Engineer

4 Wires

CIS Task: Circuit Board

13

• Involves two inter-related components:

• Cognitive: Traces the phenomenon of macro-cognition wherein team members transform data into information and then into knowledge that enables taking action and monitor problem solving.

• Socio-Emotional: Collaboration requires building and maintaining relationships among team members within a healthy and open emotional environment.

Collaboration in the CIS Game

14

CPSX Skills Map

Cognitive Socio-Emotional

CPSX Skills

Leadership Cooperation Relationships ResilienceIndividual

Knowledge Building

Team Knowledge

Building

Decision Making

and Execution

Monitoring

Experimental Design and Data Sources

CPS Human-Human (X) interaction equipment setup and telemetry data collection.

Players During Gameplay Session

17

Machine Learning based Evidence Identification and Inference

18

Multimodal Data Fusion for Annotation and Feature Extraction

https://www.youtube.com/watch?v=QmuLeBKnvF0&list=PLw2QJrKWPUyJe0pS4g91DSEwjiReX03ut

UserCamera

Collaborator Skype Video

GameFrame

Gaze poin

ts

Annotation of ‘high-level’ behaviors

GRP13_E_P16

Tracking Player Affective State

Okay i think i'm live and i see you in the corner of here so you know i think i'm good yeahstart recording the action start by clicking on this way have tio okay i r the wrenches in the erstarting from left bottom it's three to the from the left and then what okay i got a right corner one in one day okay go three down and three left to left ok now the fires and the upper right corner but it's one down okay go to write three up one right right that's right no thiss one doubt for me it's all the way to the left oh ok down too oh i can't do that okay where are you ?

Okay i think i'm live and i see you in the corner of here so you know i think i'm good yeah start recording the action start by clicking on this way have tio okay i r the wrenches in the er starting from left bottom it's three to the from the left and then what okay i got a right corner one in one day okay go three down and three left to left ok now the fires and the upper right corner but it's one down okay go to write three up one right right that's right no thiss one doubt for me it's all the way to the left oh ok down too oh i can't do that okay where are you ?

21

Automate Speech Recognition for Player Transcripts

22

NLP Based Discourse Analysis

• Questions (“What did you say?”)

• Describe task to partner (“I see symbols”; “There is a tool and a fire”)

• Monitoring time and strikes (“We only have five seconds left”)

• Closing Loop (“Roger that”)

• Sentiment Analysis• Both from Keywords (+/-) and Waveform

(pitch, tonality, etc.)

Transformer Networks, e.g. BERT

Dyad Utterances

𝑔(𝑠) 𝑔(𝑡) cos 𝜃

USE

23

CPSX Skills Map

Initiative Responsiveness AttentivenessDisplay of Emotions

EncouragingRemains Positive

Individual Information Gathering

Individual Information

Synthesis

Information Exchange

Shared Knowledge

Suggest Course of

Action

Implement Action

Evaluate Solution Quality

CPSX Skills Map

Individual Information Gathering

Individual Information

Synthesis

Information Exchange

Shared Knowledge

Suggest Course of

Action

Implement Action

Evaluate Solution Quality

• Name the Circuit task (O)• State number, color and

sequence of wires (O)

• Locate the Circuit task (E)• Ask question (E/O)

• Map number, color and sequence to rule (E)

• Share information from partner’s viewpoint (E/O)

• Cut wire (O)

• State wire to be cut (E)

• Assess solution (E/O)

• Monitor time and/or strikes (E/O)

Inference Model – Accumulate Evidence of CPS Skills

Pilot Studies: CPSX Game Data Collection

Type of

Data

Attributes of Data

Survey

Data

• Demographics: Gender, Game-play experience

• CPSX Survey

• Tessera SEL

In-game

Data• Chat logs + Audio = Transcript

• Video (User camera & Screen Recording)

• Game Logs/Telemetry

• Eye Tracking & Areas of Interest

CPSX Pilot 2: Iowa City area schools

Post Gameplay Survey

High Very HighModerateLowVery Low

Post Gameplay Survey

Feedback & Reporting interface

KCRG Coverage

https://www.kcrg.com/content/news/ACT-asks-students-to-solve-crisis-in-space-video-game-as-part-of-collaboration-study-509924761.html

Contact: Saad KhanDirector AI and Machine Learning, ACTNext

(saad.khan@act.org)