Machine Learning Approaches to Cognitive Parameter Acquisition

Terran LaneUniversity of New

Mexicoterran@cs.unm.edu

Chris Forsythe, Patrick Xavier

Sandia National Labs{jcforsy,pgxavie}@sandia.gov

Sandia’s Cognitive Modeling Framework

Computational models of human decision-makers

Models attention, perceptual cues, situational awareness, decision making

Based on oscillatory models of activation Spreading activation networks and feedback

loops between functional elements Applications -- data analysis, security,

tutoring… Bottleneck: models hand-built/tuned

Expensive and slow!

The Big PictureW

orl

d

Cue0 0

Cue1 1

CueN N

Situation0

Situation1

SituationM

Act

ion

s/D

eci

sion

s

01

10

N1

NM

Automated Model Acquisition High predictive accuracy

87% correct prediction of operator’s interpretation of scenario (incl. relevance)

91% correct in recognizing situation only Insights into operator decision-making

process Models are task & user specific

Only 26% overlap between users Large effort in building and tuning models

Project goal: (semi-)automate acquisition of parameters, network topologies, etc.

Prediction accuracy secondary concern

Roles for Machine Learning

Parameter acquisition Interconnection weights Activation levels Oscillator frequencies

Network topologies Inter-cue spreading activation network Cue <-> situation relations Feedbacks

Cues and situation identification

Parameter AcquisitionW

orl

d

Cue0 0

Cue1 1

CueN N

Situation0

Situation1

SituationM

Act

ion

s/D

eci

sion

s

01

10

N1

NM

Parameter Acquisition: Issues

Superficially supervised learning Observe features/cues and operator actions;

induce params (find s.t. f:CA) Similar to ANN backprop, EM, etc. Many effective, well understood techniques

Problem: not just high-likelihood params Actually want params used by human operator Much harder – observable stimuli don’t

directly reflect operator’s internal state Cognitive plausibility constraint

Parameter Acquisition: Approaches

Additional instrumentation Measure characteristics of operator Biometrics – eye tracking, MEG, etc. Expensive, not widespread Maybe not informative to params anyway

Utility elicitation techniques Software queries user about why

decisions were made / state of attention Picks questions to maximally improve

model Emulates expert knowledge engineer

Network Topology InductionW

orl

d

Cue0 0

Cue1 1

CueN N

Situation0

Situation1

SituationM

Act

ion

s/D

eci

sion

s

01

10

N1

NM

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

L=137

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

L=137 L=238

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

L=137 L=238 L=493

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

L=137 L=238 L=493 L=318

Topology Induction: Issues

Find structure of interconnections between variables (I.e., cues, situations)

Much harder than parameter acquisition Formally, maximum likelihood/MAP search

through all possible networks

L=137 L=238 L=493 L=318

Topology Induction: Approaches

Principles of structure search well understood

Gradient ascent, annealing, genetic search, constrained search, etc.

Difficult in practice Computationally intractable Resulting models very sensitive to data Spurious likelihood spikes low

confidence models Compounded by cognitive plausibility

constraint Can get leverage from cognitive plausibility,

though

Cue and Situation IdentificationW

orl

d

Cue0 0

Cue1 1

CueN N

Situation0

Situation1

SituationM

Act

ion

s/D

eci

sion

s

01

10

N1

NM

Cue and Situation Identification: Issues

Discern cues and whole environmental situations employed by user

Related to constructive feature induction, nonlinear projection identification, relational learning, etc.

Search across all possible nodes/relations

N=2 N=3

Cue and Situations: Approaches

Cutting-edge ML problem Direct elicitation is probably most

promising approach Formulating search space/uncertainty

reduction not straightforward Even user interface is difficult (naming

synthetic nodes/relations)

Conclusions

Decrease time/effort/cost to construct and tune cognitive model

Constrained to correspond to human’s internal model Both bane and boon to automated

model construction Insights into operator’s mental

state/decision-making process Requires/drives novel ML algorithms Future work: all of it…

Questions?