In-vivo Experimentation
22
©2012 Carnegie Learning, Inc. In-vivo Experimentation Steve Ritter Founder and Chief Scientist Carnegie Learning
description
In-vivo Experimentation. Steve Ritter Founder and Chief Scientist Carnegie Learning. An attempt to find meaning in three acts. Design : Geometry Contiguity (Vincent Aleven , Kirsten Butcher) Modeling : Adjusting learning curve parameters (Cen, Koedinger , Junker) - PowerPoint PPT Presentation
Transcript of In-vivo Experimentation
©2012 Carnegie Learning, Inc.
In-vivo Experimentation
Steve RitterFounder and Chief Scientist
Carnegie Learning
©2012 Carnegie Learning, Inc.
An attempt to find meaning in three acts
• Design: Geometry Contiguity (Vincent Aleven, Kirsten Butcher)
• Modeling: Adjusting learning curve parameters (Cen, Koedinger, Junker)
• Personalization: Word problem content (Candace Walkington)
©2012 Carnegie Learning, Inc.
DESIGN
©2012 Carnegie Learning, Inc.
Geometry angles
©2012 Carnegie Learning, Inc.
ContiguityEarly Version Commercial Version
(Carnegie Learning)
Research Version
(Carnegie Mellon)
Butcher, K., & Aleven, V. (2008). Diagram interaction during intelligent tutoring in geometry: Support for knowledge retention and deep transfer. In C. Schunn (Ed.) Proceedings of the Annual Meeting of the Cognitive Science Society, CogSci 2008. New York, NY: Lawrence Earlbaum.
Hausmann, R.G.M. & Vuong, A. (2012) Testing the Split Attention Effect on Learning in a Natural Educational Setting Using an Intelligent Tutoring System for Geometry. In N. Miyake, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th Annual Conference of the Cognitive Science Society. (pp. 438-443). Austin, TX: Cognitive Science Society.
©2012 Carnegie Learning, Inc.
Early Tutor
©2012 Carnegie Learning, Inc.
Revised (commercial) tutor
©2012 Carnegie Learning, Inc.
Geometry Contiguity• Design and field experimentation– Butcher and Aleven (2008)• Diagram interaction led to better transfer
and retention• Analysis of impact– Hausmann and Vuong (2012)• Unit-level effects mixed• Advantage for harder skills
©2012 Carnegie Learning, Inc.
Geometry Angles
©2012 Carnegie Learning, Inc.
Lessons• Change is constant• Transition from research to
production always requires adaptation
©2012 Carnegie Learning, Inc.
MODELING
©2012 Carnegie Learning, Inc.
Skillometer
©2012 Carnegie Learning, Inc.
Expression Writing
©2012 Carnegie Learning, Inc.
What gets learned?
©2012 Carnegie Learning, Inc.
Bayesian Knowledge Tracing
Cognitive tutor traces these skills differently
©2012 Carnegie Learning, Inc.16
Learning Curve Parameter Fitting• Field study looking at learning area of geometric figures
– One group used adjusted learning parameters based on previous year’s data• Optimized group took 12% less time to reach same performance
• Significant learning gain in both groups• No difference in learning gain between groups (p = 0.772 )
0
20
40
60
80
100
120
Square
Parallelogram
Triangle
Trapezoid
Polygon
Circle
Optimized
Control
©2012 Carnegie Learning, Inc.
Lessons• Learning efficiency is a great
outcome• Small, systemic changes can have
big impact• Optimizing skills requires appropriate
skill model– Koedinger, McLaughlin and Stamper
(2012) - LFA
©2012 Carnegie Learning, Inc.
PERSONALIZATION
©2012 Carnegie Learning, Inc.
Word problem customization
©2012 Carnegie Learning, Inc.
Personalization field study• Students who got problems related
to their interests made fewer errors• Also affected subsequent unit• Interaction with readability
©2012 Carnegie Learning, Inc.
Lessons• Content matters– Challenge for knowledge component
modeling• Are we personalizing preferences,
reading level or both?
©2012 Carnegie Learning, Inc.
Summary• It’s not about whether A is better
than B– It’s about why A is better than B
