Sae Schatz

MESH Solutions, LLC – a DSCI Company

Situated Tutors Tutorial

Schedule

• Part 1: Background

• Part 2: Theory

• Part 3: Technical Details

• Part 4: Use Case

• Part 5: Recommendations

Part 1: Background

ITS Effectiveness

Instructional Outcomes: Knowledge, Performance, etc.

Num

ber

of L

earn

ers

ClassroomLearning

(Human) Tutor

Top 2%

Average

Performance

Num

ber o

f Stu

dent

s

Bloom, B. S. (1984).

The Challenge

EfficientFor training declarative/procedural skills

Computer-Based

Training

Didactic Lecture to a

Group

EffectiveFor higher-order cognitive skills

Apprentice Learning

One-on-One (Human) Tutoring

Currently, higher-order skills training is effective or efficient

IntelligentTutors

Instructional Simulations

Situated Tutors

Static Computer-Based Learning

Same for everyone

Intelligent Tutors

Different for different students

ITS Effectiveness

Instructional Outcomes: Knowledge, Performance, etc.

Num

ber

of L

earn

ers

ClassroomLearning

(Human) Tutor

Top 2%

Average

SHERLOCK (1988)

Ecolab (1999)

Andes (2005)

ITS Effectiveness & EfficiencyTypical effectiveness gains of ITSs:

— 0.48–0.61σ (Dede, 2008)— 1.0σ (Lane, 2006)— LISP tutor = 48% improvement on posttest (Anderson, 1990)*

Typical efficiency gains of ITSs:— One third of the time vs. classroom (Lajoie & Lesgold, 1992)

— 4σ efficiency gain over traditional CBT (Romero et al., 2006)

— Air Force electronics tutor for 20hr = 48 months of OJT (Lesgold et al., 1990)

— LISP tutor = 30% less time vs. classroom (Anderson, 1990) *same study as above

Intelligent Tutors ProsAdaptive

Manpower Efficiency

Embedded Pedagogy/Andragogy

Intelligent Tutors Cons Lacks Intrinsic Feedback

Usually Declarative/Procedural

Usually More Defined Domains

Usually Single-user

Pros and Cons

IntelligentTutors

Instructional Simulations

Situated Tutors

Simulation-Based TrainingInstructional simulations include those simulations that employ a

systematic instructional methodology (scenario-based training, for our purposes), as well as accurately represent the problem-solving domain (Salas, Bowers, & Rhodenizer, 1998; Oser et al., 1997). — Average performance gains of SBT vs. classroom:

• 72% fewer errors in practice with SBT (Haque & Srinivasan, 2006) meta-analysis

— Average efficiency gains of SBT:• 84% less time vs. traditional (Haque & Srinivasan, 2006) meta-analysis

But, in practice, SBT often falls short…

Low EfficiencyHeavy instructor workloadInstructors must be SMEs,

instructional designers, and technologistsDeployed systems may have no instructional staff

(e.g., McCarthy, 2008; Loftin et al.,2004; Smith-Jentsch et al., 1998).

Low EffectivenessIf instructors cannot meet all requirements or cope with the workload, suboptimal training may result involvedThis may result in negative training(e.g., Loftin et al., 2004; NRC, 1985; Houck & Thomas, 1991; Andradóttir et al., 1997; Air Force, 1991; Wray et al., 2004)

Simulation ProsGood Transfer of Training

Often Supports Team Training

Supports Complex Contexts

Simulation Cons One-Size-Fits(?) All

Relies on Instructor for Pedagogy

Relies on Instructor for Sequencing

Heavy Instructor Workload

No Good Instructor = Poor Training

Pros and Cons

IntelligentTutors

Instructional Simulations

Situated Tutors

Situated Tutors

Situated tutors are a special class of Intelligent Tutoring Systems that combine the features of an intelligent tutor with the scenario-based situated learning environment of instructional simulations.

INTELLIGENTTUTOR

+SIMULATION-

BASED LEARNING

=SITUATED

TUTOR

INTELLIGENTTUTOR

SIMULATION-BASED LEARNING

+ =SITUATED

TUTOR

Adaptation

Includes instructional support

Careful operationalization of domain

Extrinsic feedback

Automation

Supports higher-order cognitive skills

Situated learning context

Intrinsic feedback

Team training

Facilitate less determinate domains

Situated Tutors

Part 2: Theory

Situated Tutors

Situated tutors are computer-based instructional technologies that, at a minimum, include a simulated learning or training environment of Interactive Multimedia Instruction (IMI) Level 3 or above and instruct with intelligent adaptation. Further, these features are, at least, loosely federated with each other.

Simulation Depth

IMI LEVEL 1Page turner: does not include any simulation-like features

Example: A basic website, like the Red Cross’s Preparing for Events

IMI LEVEL 2Medium simulation: supports limited interactivity, such as asking and scoring a response to a question

Example: Interactive courseware or website scripting, like this quiz from Discovery.com

IMI LEVEL 3High simulation: Surface simulation with 2-3 levels of complex branching

Example: A highly interactive simulation, such as Dafur is Dying, a robust serious game made in Flash

IMI LEVEL 4Full simulation: rich interactivity and branching, extensive high-fidelity surface simulation capabilities

Example: A video game, such as America’s Army

Does the system offer sufficient psychological fidelity and freedom of action to support the training of higher-order cognitive skills?

Sophistication of Adaptation

Does the system support tailored pre-task adaptation (e.g., instructional sequencing)and during-task adaptation (e.g., personalized hinting and feedback).

ADAPTATION AS PREFERENCELearner choice: Allows learner to control nature of interactions—generally diminishes outcomes

Example: Self-sought instruction, like this flash game

ROLE ADAPTATION

Categorical: Broad learner-selected categories, such as by MOS, often distinguishes content presented

Example: Many training websites, such as the GPRIME medical trainer

MACRO ADAPTATION

Tailored Pre-training: Individual learner KSAs and traits affect pre-task adaptation

Example: Often found in CBT systems; supports sequencing and ATI, e.g., some LMSs

MICRO ADAPTATION

Tailored During-training: Tailored intervention is triggered based on during-task actions

Example: Found in conventional ITSs; supports immediate feedback; e.g., PAT intelligent tutor

ACTIVE ADAPTATION

Overall: Combination of effective macro- and micro-adaptations

Example: Facilitates immediate feedback and long-range sequencing; e.g., Rosetta Stone

Situated Tutors: Tasks, Conditions, and Standards

Degree of Component Integration

Does the system support simultaneous functioning and robustdata interchange between the ITS and SBT components?

NO FEDERATIONSeparated: No data are passed between the simulation and ITS components. Generally, an ITS lesson is delivered and then the student is told to use the simulation

Example: Microsoft Flight Simulator training web site

LOOSELY FEDERATEDSide-by-Side: The ITS and simulation exchange only outcome data. The two systems are often physically separated

Example: Many military situated tutors follow this model; protocols such as IPA, DTECS, and SITA facilitate this integration; e.g., FBCB2/Tactical Decision-Making ITS

TIGHTLY FEDERATEDFull Integration: The ITS and simulation components can exchange data constantly; ITS features often “overlay” the simulation

Example: The most sophisticated military systems, such as PORTS TAO ITS; I/SIS protocol can be used (but is rarely applied)

Find & Classify Situated Tutors

Detailed literature review of 86 situated tutors

Definitions from: Schatz, S., Oakes, C., Folsom-

Kovarik, J. T., & Dolletski-Lazar, R. (2012). ITS +

SBT: A Review of Operational Situated Tutors.

Military Psychology.

Through 1990 1991-1995 1996-2000 2001-2005 2006-Present0

5

10

15

20

25

30

Situated Tutor Development Timeline

Year of Introduction

New

Sit

uate

d Tu

tors

Int

rodu

ced

USAF

US Army

US Navy/USMC

US Joint/Coalition

Education

Medical

Energy

Transportation (Incl. Space)

Law Enforcement

Manufacturing

Other (Incl. Basic Research)

0 5 10 15 20 25

Situated Tutors by Domain

Through 1990 1991-1995 1996-2000 2001-2005 2006-Present0

1

2

3

4

5

6

7

8

9

10

Military Acquisition TimelineUSAF US Army USN US joint military non-US military

Year of Introduction

New

Sit

uate

d Tu

tors

Int

rodu

ced

Situated Tutors Effectiveness

ELECT BiLAT (aka VCAT)(USC Institute for Creative Technologies)

ExpertCop(Furtado & Vasconcelos, 2006)

87%Improved with

simulation + ITS

87% of police officers explained additional crimes with both the ITS and sim. vs. simulation alone (Furtado & Vasconcelos)

59%Better than the

video only

Ablative test: 56% in the video-only condition, 67% in the no-coach condition, and 89% of in the coach condition were successful (Lane et al., 2008).

Situated Tutors Efficiency

TAO ITS(Stottler Henke)

Over 2000%More Efficient

Per Class

Previously, one instructor needed for two students, for a class of 42; now one instructor manages whole class (Stottler & Panichas 2006)

IATS(Madni, 2010)

98%Cost Saving Versus

F2F Approach

AFRL’s IATS reduced costs from $1172 a seat/year to $28 a seat/year for shipboard maintenance training (Madni, 2010).

Part 3: Technical Details

Domain Model

LearnerModel

PedagogicalModel

Domain Content

LearnerData

InstructionalMethods

Traditional Intelligent Tutor

Domain Model

LearnerModel

PedagogicalModel

Domain Content

LearnerData

InstructionalMethods

SituatedTutor

Game Engine

Historic Inputs:• Prior Knowledge• General Aptitude• Constitutional Attributes• Affective Attributes• Learner Preferences

Micro-Adaptation:• Give Hints/Coaching• Change Teaching Approach• Change Challenge Level• Adjust Scenario Story• Give Intrinsic Feedback

Macro-Adaptation:• Content Selection • Preset Hints/Coaching• Preset Teaching Approach• Scaffold Challenge Level• Preset Scenario Variables

Immediate Inputs:• Current Performance • System Use/Abuse• Affective State

(e.g., boredom, confusion, delight, flow, and frustration)

Overlay models ignore details of how students learn and instead track what students have learned in a simple way,

similar to a checklist. Slow to develop and moderately effective for macro-adaptation.

ITS Learner Model Varieties

Overlay Models Classifier Models

Constraint-Based Models

Example Tracing Models

(or “Pseudotutors”)

Perturbation Models

(or “Buggy Models”)

Production Rule Models

Bayesian Networks

Dynamic Bayesian Networks

Finite-State Automata

Decision Trees

Neural Networks

Behavior Transition Networks

… and Others …

Case Libraries

Model Tracing Systems

ACT-R Models or “Cognitive Tutors”

More Detailed

LessDetailed

Perturbation models or buggy models, try to describe all the incorrect knowledge the learner may have. Can

require extensive investment and have mixed results.

Bayesian networks and other classifiers are efficient but have lower detail than some other model types.

Monitor the immediate problem state. As long as a learner never reaches a state that the model identifies as

wrong, he or she may perform any action. Highly effective.

Authors define incorrect responses for single questions, and they are less concerned with the cognitive theories,

hence example-tracing systems were called pseudo-intelligent tutors or pseudotutors.

Uses model tracing algorithm (i.e., rules drawn from a general model of human cognition), hence called

“cognitive tutors.” Slow to develop, but have high returns.

Lowest reported development to learning

time ratio

Highest reported effect on learning

Macro- and micro-adaptation

Overlay models 24:1 1.02, compared to on-the-job training Macro

Bayesian networks and other classifiers 30:1 0.7, compared to work

without feedback Both

Constraint-based models 220:1 1.3, compared to work

without feedback Micro

Example tracing 18:1 0.75, compared to paper homework Micro

Perturbation and buggy models 133:1 Not significant Both

Production rules and model tracing 200:1 1.2, compared to

classroom learning Micro

Folsom-Kovarik, J. T. & Schatz, S. (2011).

Part 4: Real Example

37

Process— Review of literature— Interviews with SMEs, stakeholders— Concept designs for team — Learning objectives Dynamic tailoring requirements — GOTS/COTS Trade-off analysis— Hardware/software feasibility/cost analysis— Iterative requirements authoring— Iterative development— Iterative testing

Baselining

PercepTS Virtual Ville Concept Virtual Ville

Concept Creation

Multiple OPs (or Combat Outposts)

TOC

AAR & Vicarious

Learning Room

OT / Control Room

Virtual Ville

Spatial Layout

Scenario Content

Database

Trainee Records

Database

Trainee Module

Assessment Module

(Micro/Macro)

Speech Recognition

Dynamic Tailoring Module

AAR Module

INSTRUCTIONAL AND EXPERT KNOWLEDGE DATABASES

SYNTHETIC TRAINING ENVIRONMENT

INPUT/OUTPUT

Observer Trainer

Terminal

Domain Module

Additional Simulation

Plug-Ins

Optical System

Visualization

RadioInterface

Controllers

Visualization System

Virtual Environment

Database

Simulation Environment

(Torque)

Dynamic TailoringDatabase

Patterns of Life Database

Radio Interface

Device

Optical Interface Devices

AUTHORING AND MANAGEMENT

Metrics Authoring

ToolkitScenario and Lesson Toolkit

Dynamic Tailoring Toolkit

TrainingContent

Authoring Interface

SAF Behavior

Authoring Interface

Positional TrackingSystem

NPC Controller

(SAF Behaviors)

Virtual Team

Speech Generation

Concept Creation: Conceptual Architecture

Detailed Learning Objectives

Requirements

Trade Off Analysis

Part 5: Recommendations

#1: Reporting Situated Tutor Development

• Report systems’ • (a) interactivity (including IMI Levels)• (b) forms of adaptation • (c) integration of features

#2: Report Situated Tutor Evaluation Results

• Empirically assess situated tutors’ effectiveness and efficiency• Use ablative conditions, not just versus classroom• Remark on real-world impacts (e.g., reduced cost per seat,

increased readiness reports)

#3: Expand Intrinsic Adaptation

• Investigate novel situated tutor methods• Carefully assess adaptations impacts• Document categorical types of intrinsic adaptation, their best

uses, and potential pitfalls to avoid• Consider macro-adaptive approaches, such as dynamic

scenario generation, too

#4: Expand Higher-Order Instruction

• Emphasize sophisticated cognitive, affective, and psychosocial competencies• Examine instructional strategies—specifically for situated

tutors—that engender higher-order skills

#5: Embrace “Instructional Fidelity”

• Need to focus on the development of expert mental models• Need to move beyond just operationally-situated practice• Need to design instructional experiences for developing expertise

– Thus, scenarios may not always be “realistic”– But they will include the necessary cues to provide appropriate instruction

Experts don’t just know more, they know differently…

#5: Embrace “Instructional Fidelity” (Cont)

• Instruction built-in to scenarios• Apply a rich blend of instructional strategies to simulations

– Need to use scenarios in novel ways (beyond situated practice)– Need to embed appropriate pedagogical strategies within systems– Enables more effective training

• Embed educational experiences within the scenario– Need to reconceptualize “Scenarios” “Instructional Scenarios”– Need to focus on the development of expert mental models– Enables more efficient, situated training

Instructional scenarios are synthetic experiences designed to move a person from one level of understanding to the next…

Sae Schatz

sschatz@mesh.dsci.com

Thank you!