Some Psychometric and Design Implications of Game-Based Learning Analytics

David GibsonCurtin University

Jody Clarke-MiduraHarvard & MIT

Abstract

• The rise of digital game and simulation-based learning applications has led to new approaches in educational measurement that take account of patterns in time, high resolution paths of action, and clusters of virtual performance artifacts.

• The new approaches, which depart from traditional statistical analyses, include data mining, machine learning, and symbolic regression.

The Premise

In an interactive digital-game, traces of a learner’s progress, problem-solving attempts, self-expressions and social communications can entail highly detailed and time-sensitive computer-based documentation of the context, actions, processes and products.

Example Virtual Performance Assessment

• Contexts: Farm, Playground, Science Lab• Actions: Talking, Testing, Walking to…• Processes & Products: Test Results, Explanations

Clarke-Midura & Gibson, 2013

Interaction Traces = Evidence

There is a need for new frameworks, concepts and methods for measuring what someone knows and can do based on game interactions and artifacts created during serious play

Why? Because ubiquitous, unobtrusive, interactive big data (fast, varied, voluminous) is created by people working in digital media performance spaces

Example of Ecological rationality & Empirical probability

• Shifts from prediction to claim indicate that the simulation might be educative

Clarke-Midura & Gibson, 2013

Sensors

• Wireless EEG– Facial muscles, emotional

clusters, raw EEG• Wireless Galvanic Skin

Conductance – Arousal level

• Eye Tracker– Gaze-point, duration, mouse-

clicks• Haptics– Button presses, head tilt

Anatomy of the System

Helen Chavez & Javier Gomez, ASU

Data Dashboard at ASUHelen Chavez and Javier Gomez, ASU

Biometric Sensor Nets

• What patterns do we find?

• How do they change over time?

• How do they relate to baseline and experimental activities?

Challenge: New Psychometrics

• What are some of the measurement and analysis considerations needed to address the challenges of finding patterns and making inferences based on data from digital learning experiences?

Network Graphs

Digraphs illustrate structural relationships in the causative factors during a time slice or event frame.

Network Analysis

Adjacency tablesCentrality

  AF3 F7 F3 FC5 T7 P7 O1 O2 P8 T8 FC6 F4 F8 AF4 GX GY

AF3      

F7    

F3      

FC5      

T7    

P7      

O1      

O2  

P8        

T8      

FC6        

F4      

F8      

AF4        

GX      

GY    

Symbolic Regression

Automated search for algorithms

Clarke-Midura & Gibson, 2013

Complex nonlinear relationships can be discovered

New Space for Performance

• Unfold in time • Cover a multivariate space of possible actions• Assets contain both intangible (e.g. value,

meaning, sensory qualities, and emotions) and tangible components (e.g. media, materials, time and space)

NOTE: Asset utilization during performance provides evidence of what a user knows and can do

Example of Cluster Analysis

One group used far fewer resources labeled as ‘salient strategies’

Example of a rule-based graph

Students who had this pattern of resources were most likely to show evidence of forming a hypothesis

Clarke-Midura & Gibson, 2013

Performance Space Features

• Unconstrained complex multidimensional stimuli and responses

• Dynamic adaptation of items to user, which entails interactivity and dependency

• Nonlinear behaviors with both temporal and spatial components

NOTE: Higher order and creative thinking is supported in such a space

The Game-Based Psychometric Landscape

• A “do over” for performance assessment• New ways of performing = new methods of

data capture, analysis and display• Complex tasks and artifacts containing– higher order thinking (e.g. decision sequences)– physical performances demonstrating skills– emotional responses

Thinking States

Rise inuncertainty and interest

Agreement & concentration drop

During thinking

What Games & Sims Teach

• Understanding big ideas - systems knowledge• Dealing with time and scale• Practice in decision-making• Active problem-solving• Concepts, strategies, & tactics• Understanding processes beyond experience• Practice makes improvement

(Aldrich, 2005)

Conclusion

Methods based in data-mining, machine learning, model-building and complexity theory form a theoretical foundation for dealing with the challenges of time sensitivity, spatial relationships, multiple layers of aggregations at different scales, and the dynamics of complex behavior spaces.