Automated Assessment for Adaptive Learning of Complex Tasks
Alan Koenig
Markus Iseli
Allen Munro
CCT
Characteristics of Cognitively Complex Tasks
Often require multiple, non-trivial steps to complete
Performance can be highly variable
Often include interdependent task features
Often well suited to representation in games & simulations
– Multiple scenarios
– Low cost
Examples:
Tactical Planning Simulation Damage Control Simulation
– Low risk
– But… assessment can be difficult
Anti-Submarine Warfare
Anti-submarine warfare (ASW) is a branch of naval warfare that uses surface warships, aircraft, or other submarines to find, track and deter, damage, or destroy enemy submarines.
– Carried out by teams of people within and across ships
– Requires synthesis of large quantities of information in short periods of time
– Heavy emphasis on situation awareness and decision making skills (especially among TAO’s)
Assessment Methodology Overview
ASW Ontology Construction
ExpertsRules /Doctrine
Procedures /Processes
Facts /Observations
DOMAIN
Ontology Excerpt
Bayesian Network Development
OntologyNodes
Directionalityof Links
Observables /SME Input
Strength of Relationships
Characterizing the nature of relationships is essential
Observable Actions
Latent Variables
SonarPlanning
PassiveSonar Planning
ActiveSonar
Planning
UseShipPassiveSonar DropPassiveSonobuoys
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
ASW Skilled
Torpedo CounterMeasures
MEUProtectionSub Hunting
Plan for Opposition
Intell on Opposition
Opposition Estimation
ShowMissionInfo ShowBriefing
EstLLSubSpeed
EstLLTorpedoRange
0.5
SonarPlanning
TDA AvoidManeuvers
PassiveSonar Use
ActiveSonar Skill
AlterCourseTo AvoidFast
AlterCourseTo AvoidSlow
UseShipPassiveSonarDropPassiveSonobuoy
s
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
TDA Avoidance
DontEverEnterTDALimLines
Estimation
EstDatumSubSpeed EstDatumTorpedoRange
Datum Estimation
TorpedoDeception
TorpedoEscape
NixieDropped MICDropped
TorpedoEvasion
TorpedoCounter
Bayesian Network for TDA Avoidance
ASW Skilled
Torpedo CounterMeasures
MEUProtectionSub Hunting
Plan for Opposition
Intell on Opposition
Opposition Estimation
0.5
TDA AvoidManeuvers
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
TDA Avoidance
LimLines Estimation
Datum Estimation
TorpedoDeception
TorpedoEscape
Bayesian Network for TDA Avoidance
ShowMissionInfo ShowBriefing
EstLLSubSpeed
EstLLTorpedoRange
SonarPlanning
PassiveSonar Use
ActiveSonar Skill
AlterCourseTo AvoidFast
AlterCourseTo AvoidSlow
UseShipPassiveSonarDropPassiveSonobuoy
s
DontEverEnterTDA
EstDatumSubSpeed EstDatumTorpedoRange
NixieDropped MICDropped
TorpedoEvasion
TorpedoCounter
Conditions for TDA Avoidance to be considered by Bayesian network
Observable Actions
Latent Variables
The Advantage of Bayesian Networks
SonarPlanning
PassiveSonar Planning
ActiveSonar
Planning
UseShipPassiveSonar DropPassiveSonobuoys
DropActiveSonobuoys UseDippingSonar UseShipActiveSonar
Probability of mastery of the latent variables is inferred from the scored observable actions
P(ASP): Probability of mastery of concept Active Sonar Planning
P(UDS | ASP): Conditional probability. Probability of mastery of concept UseDippingSonar, given information about mastery of concept Active Sonar Planning
Example: Determining probabilities
P(UDS):Probability of mastery of concept UseDippingSonar
P(ASP) x P(UDS | ASP)
P(UDS)P(ASP | UDS) =
Real-Time Formative Assessment
•Provide performance feedback
•Provide practice / resources
•Add / change tasks
•Add / change affordances
FORMATIVE ASSESSMENT
Simulation(Sandbox)
ADAPTIVE OPTIONS
Observable Actions & Events
CAA(with Bayesian Network)
Check the Briefing—Good!
1st PROBLEM: Chinese Kilos Threat Problem
00:00:00 .k.Announce. ShowBriefing 0.95
The fact that we’re up against Kilos implies a silent sub speed of 6 Kt.and a torpedo range of 12,000 yds.
Failed to Correct Estimates—Bad!
1st PROBLEM: Chinese Kilos Threat Problem
07:26:32 .k.Announce. EstSubSpeed 0.207:26:32 .k.Announce. EstTorpedoRange 0.2
The estimated sub speed was left at 3 Kt. So the limit lines were not accurate, relative to the briefing.
Scored at end, because these values were never changed.
Failed to Correct Estimates—Bad!
1st PROBLEM: Chinese Kilos Threat Problem
07:26:32 .k.Announce. DontEverEnterTDA 0.1
Uh-oh! The MEU entered the TDZ.
17
Problem 1
18
Problem 1
19
Problem 1
20
Problem 1
21
Problem 1
Check the Briefing—Good!
2nd PROBLEM: Threat West of Gibraltar
This time the mission brief gives specific estimates of hostile sub speed and torpedo range.
00:00:00 .k.Announce. ShowBriefing 0.95
Datum Estimates2nd PROBLEM: Threat West of Gibraltar
The datum estimates forsub speed and torpedorange don’t match the briefing.
Datum Estimates Corrected—Good!
2nd PROBLEM: Threat West of Gibraltar
Set to the correct values.
01:14:03 .k.Announce. EstSubSpeed 0.9501:14:03 .k.Announce. EstTorpedoRange 0.95 Scored at end.
Check Limit Lines2nd PROBLEM: Threat West of Gibraltar
Perhaps not a wide enough berth…
Plot a New Course2nd PROBLEM: Threat West of Gibraltar
A safer course?
Check Limit Lines Again2nd PROBLEM: Threat West of Gibraltar
Looks safer
Successful Avoidance2nd PROBLEM: Threat West of Gibraltar
Success!
01:14:03 .k.Announce. DontEverEnterTDA 0.95
29
Problem 2
30
Problem 2
31
Problem 2
32
Problem 2
33
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Problem 1Problem 2
1 2 3 4 50
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Problem 1Problem 2
Plan for Opposition
TDA Avoidance
Sub speed: Wrong estimate
Entered TDA
Prob
abili
ty o
f Mas
tery
Prob
abili
ty o
f Mas
tery
time
time
ASW Skilled
Torpedo CounterM.
MEUProtectionSub Hunting
Plan for Opposition
Intelligence on Opposition
Opposition Estimation
GetMissionInfo GetBriefing EstSubSpeed EstTorpedoRange
SonarPlanning
TDA AvoidManeuvers
SenseQuietly
SenseLoudly
ManeuverFastManeuverSlow
ShipsPassivSonar PassiveSonobuoys ActiveSonobuoys DippingSonar ShipsActivSonar
TDA Avoidance
B20: score 0.95D20: score 0.6F20: score 0.1
B16: score 0.95D16: score 0.7F16: score 0.05 B5: score 0.9
D5: score 0.7B6: score 0.95D6: score 0.7
B11: score 0.95D11: score 0.8F11: score 0.2
B12: score 0.95D12: score 0.8F12: score 0.2
B36: score 0.9E36: score 0.3F36: score 0.2
B40: score 0.95C40: score 0.4E40: score 0.3
C44: score 0.9D44: score 0.6F44: score 0.05
B28: score 0.95D28: score 0.6
B32: score 0.95E32: score 0.3
DontEnterTDAIfPos
B24: score 0.95C24: score 0.9F24: score 0.1
TDA avoidanceBayesian network sub-net
Initial Evaluation Feedback
CAA Debugging OutputInstructional Feedback High Level Assessments
First Successful Problem
Into the Next Problem
Adaptive Problem Selection
A new problem is advised.
Loading the Problem
The advised problem is selected
Initial Actions in the New Problem
Preparing to Find the Sub
Launching a helicopter
Detecting the Sub
Sonobuoys detect sub
Recording of Performances & Assessments
Simulation(Sandbox)
Log Files(Actions and Events)
These performance recordscan be replayed.
Report Generation(CAA Monitor)
Assessment History(Per Student)
Report
CAA
Questions / Comments?
CCT
Top Related