Using Expectations to Drive Cognitive Behavior

Unmesh KurupChristian Lebiere, Tony Stentz, Martial Hebert

Carnegie Mellon University

Cognitive Decision Cycle

t+1

Calculate Mismatch

World

High-level Cognition

Retrieve Response WorldAction Action

Action

PredictionPrediction

Prediction

t-1 t

Cognition

• Cognition is driven by Expectations/Predictions.

Pedestrian Tracking & Behavior Classification

Goals:• Investigate use of expectations• Integrate with perception• Run both offline & real-time

Integrated System

Partial Matching & Blending

Chunk2 isa location-

chunkid person2 nextx 1010nexty 500

Chunk3isa location-

chunkid person3 nextx 187nexty 313

Chunk4 isa location-

chunkid person1 nextx 299nexty 100

+retrieval>isa location-chunkid person1nextx 300

Chunk1 isa location-

chunkid person1 nextx 255nexty 100

Chunk4 isa location-

chunkid person1 nextx 299nexty 100

Declarative Memory

Partial Matches

Chunk5isa location-

chunkid person1 nextx 293.91nexty 100

Blended result

Chunk1 isa location-

chunkid person2 nextx 255nexty 100

Using Expectations: TrackingChunk-type visual-location

id X Y Dx Dy Nextx Nexty

Foreach Object o: +blending>

isa visual-locationid o

compare to (x,y)s from perceptionpick thresholded closest match, calculate newdx, newdy, newx, newy+imaginal>

isa visual-locationid o…

Features

straight1 straight2 detour left straight3 veerFeatures:

Behavior Features

Normal – Straight straight1, straight2, straight3

Normal – Left straight1, straight2,left

Peek straight1, detour, left, no-chk-pt

Behavior Features

Detour straight1, detour, straight3, chk-pt

Veer straight1, straight2,left, veer, chk-pt

Walkback straight1, straight2, left, straight2, straight1, chk-pt

Using Expectations: Detecting Features from Data

Straight & Left

Deviation from expected location indicates a point of interest

Foreach location+blending>

isa visual-locationx =x y =y

compare to (x,y)s from perception

if path deviates more than threshold, mismatch!

+imaginal>isa visual-locationid o…

Cluster points into regions

Detected Features

Data

• Combined Arms Collective Training Facility(CACTF) at Fort Indiantown Gap, PA.

• 4 examples. 3/1 split.• Multiple behavior set– 10 behaviors.

Behaviors

Straight & Left

Peek

Detour

Veer

Walkback

Results

Hand-coded Model(Single Behavior Set)

Hand-coded Model(Multiple Behavior Set)

Made 99.3% Made 46.5%Correct 99.15% Correct 30.2%

Incorrect 0.15% Incorrect 16.3%

Learning Model(Single Behavior Set)

Learning Model(Multiple Behavior Set)

Made 86.1% Made 82.4%Correct 68% Correct 43.8%

Incorrect 18.1% Incorrect 38.6%

Future Work – Semantic Labels

Future Work – Using Semantic Labels

Behavior Features (Spatial) Features (Semantic)Normal – Straight straight1, straight2,

straight3Sidewalk, Pavement

Normal – Left straight1, straight2,left

Sidewalk

Peek straight1, detour, left, no-chk-pt

Pavement, Sidewalk

Detour straight1, detour, straight3, chk-pt

Pavement

Veer straight1, straight2,left, veer, chk-pt

Sidewalk, Pavement

Walkback straight1, straight2, left, straight2, straight1, chk-

pt

Sidewalk

Future Work

• Generic model of monitoring using expectations

• Learn expectations• Monitor for deviations from expectations– Signal failure– Provide for recovery

Collaborators

Max Bajracharya, JPLBob Dean, GDRS

Brad Stuart, GDRSFMS lab, CMU