Synthetic TeammateProject

March 2009

Jerry Ball

Air Force Research Laboratory

2

Synthetic Teammate Project

• Project Goal: Develop a Synthetic Teammate capable of functioning as the Air Vehicle Operator (AVO) or pilot in a 3-person simulation of a Unmanned Air Vehicle (UAV) performing reconnaissance missions

– Cognitively Plausible

• Using ACT-R

– Functional

• Large-scale

– Empirically Validated

• Not valid if it’s not functional!

few research teams

attempting to do these

at once!

3

• Guiding principle: Don’t use any computational techniques which are obviously cognitively implausible

• Key Assumption: Adhering to well-established cognitive constraints may actually facilitate development by pushing development in directions that are more likely to be successful

– Short-term costs associated with adherence to cognitive constraints may ultimately yield long-term benefits

– Don’t know what you’re giving up when you adopt cognitively implausible techniques

Synthetic Teammate Project

4

Synthetic Teammate Project

• Collaborative project between the Air Force Research Laboratory (AFRL) and Cognitive Engineering Research Institute (CERI)

– Applied research funds from AFRL/RHA

– Basic research funds from AFOSR

– Basic research funds from ONR

• Using the Cognitive Engineering Research on Team Tasks (CERTT) Synthetic Task Environment (STE)

– Developed with funds from AFOSR

5

CERTT Synthetic Task Environment

AVO (flies UAV)PLO (takes pics) DEMPC (plans route)

Team Goal: Fly UAV

Reconnaissance

Missions

6

UAV Reconnaissance Missions

• AVO, DEMPC and PLO collaborate to complete a 40 minute reconnaissance mission

• AVO must fly UAV past a sequence of waypoints which are determined by the DEMPC and communicated to the AVO as a flight plan

• Waypoints may have altitude and airspeed restrictions and have an effective radius for fly by

– Route based restrictions, waypoint type and effective radius must be communicated from DEMPC to AVO

– Photo restrictions must be communicated from PLO to AVO

• PLO must take pictures of target waypoints within the effective radius, but does not take pictures of entry and exit waypoints

7

Importance of Communication

• Communication is critical to the success of reconnaissance missions

• PLO and DEMPC must communicate restrictions to AVO

• DEMPC must communicate flight plan to AVO

• When the unexpected happens—e.g. unplanned waypoint added to mission, photo missed—teammates must develop workarounds and communicate adjustments

8

AVO Workstation

Instruments Warnings

Text

Chat

DEMPC to AVO: LVN is our first waypoint

AVO to INTEL: Copy

INTEL to all: OK team, mission 1, good luck.

Are there any restrictions for LVN?

9

Synthetic Teammate Integration

AVO

Synthetic AVO

Teammate

10

Synthetic Teammate Integration Standalone Mode

• Using an agent development framework to provide “light-weight” implementations of the DEMPC and PLO for development purposes

–Low-cognitive fidelity, scripted agents

–Eliminate need to have humans acting as DEMPC and PLO during development

11

Text Chat

OutputLanguage

Comprehension

Language

Generation

Dialog

Manager

Task Behavior Model

Motor

ActionsSituation ModelVisual

Input

Text Chat

Input

System Overview

12

Language

Comprehension

Language

Generation

Dialog

Manager

Task Behavior Model

Situation Model

System Overview

Text Chat

Output

Motor

ActionsVisual

Input

Text Chat

Input

13

Language Comprehension

• Theory of Language Processing (Ball 2007…1991)

– Activation, selection and integration of constructions corresponding to the linguistic input

– Nearly deterministic, serial processing mechanism (integration) operating over a parallel, probabilistic (constraint-based) substrate (activation & selection)

• Theory of Linguistic Representation (Ball 2007)

– Focus on encoding of referential and relational meaning

• Implemented in a Computational Cognitive Model

– Using the ACT-R Cognitive Architecture

• Adheres to well-established Cognitive Constraints

14

Cognitive Constraints

• Incremental processing – word by word

• Interactive processing – lexical, syntactic, semantic, pragmatic and task environment information used simultaneously to guide processing

– Highly context sensitive – but limited to preceding context (no access to subsequent context)

– Word recognition and part-of-speech determination integrated with higher-level syntactic, semantic and discourse processing (single pass)

• Robust processing

– Must handle ungrammatical input, incorrectly spelled words and non-sentential input

– Minimize number of “hard constraints” (e.g. whole word matching) which can lead to failure when they aren’t satisfied

15

Cognitive Constraints Processing Mechanisms

• Serial, nearly deterministic (controlled) processing operating over a parallel, probabilistic (automatic) substrate

– Parallel, probabilistic substrate interactively integrates all contextual information leading to selection of the best choice given the available local context at each incremental choice point

• Soft constraints or biases

– Once a choice is made the processor proceeds serially and deterministically forward in real-time

– When a locally preferred choice turns out to be dispreferred in wider context, context sensitive context accommodation mechanism kicks in

16

• The following example is from the Language Processing Model

– “no airspeed or altitude restrictions”

Language Processing in the Model

17

no

“no” object specifier object referring expression

= nominal construction

18

no airspeed

“airspeed” object head

Tree structures created from output of model

automatically with a tool for dynamic visualization

of ACT-R declarative memory (Heiberg, Harris & Ball 2007)

integration

19

no airspeed or altitude

“airspeed or altitude” object head

Accommodation

of conjunction via

function overriding

override

20

no airspeed or altitude restrictions

“airspeed or altitude” modifier“restrictions” object head

Appearance of parallel processing!

airspeed or altitude = head vs.

airspeed or altitude = mod

Accommodation

of new head via

function shift

shift

21

Computational Constraints

• Processor needs to operate in near real-time to be functional

• Large-scale systems that can’t handle non-determinism efficiently (e.g. Context-Free Grammars) typically collapse under their own weight

• Deterministic processing is computationally efficient

• Probabilistic and Parallel processing—often combined with a limited “spot light”—are alternative mechanisms for dealing with non-determinism

• Parallel processing can be computationally explosive on serial hardware

– Forced to use some “hard constraints”—e.g. first letter match—in word recognition subcomponent

22

Computational Constraints

• No limited domain assumption to simplify model

– CERTT text chat shows broad range of grammatical constructions and thousands of lexical items

• Relational database integrated with ACT-R to support scaling up model to a full mental lexicon

– Plan to integrate sizeable subset ( > 15,000 lexical items) of most common words in WordNet lexicon ( > 100,000 lexical items)

• Can’t ignore lexical ambiguity!

– Study underway to compare performance of model when Declarative Memory (DM) is stored in an external DB vs. internal Lisp process

• Internal Lisp process is faster for small DM, but can only handle 30% of WordNet before running out of memory!

23

Start with a Domain General Language Processing System

• Contains 2000 most common words in English and 2500 words in total

• Handles a broad range of construction types

– Declarative, Imperative, Yes-No Question, Wh-Question

– Intransitive, Transitive & Ditransitive Verbs, Verbs with Clausal Complements, Predicate Nominals, Predicate Adjectives and Predicate Prepositions

– Specifier, Head, Complement, Pre- and Post-Head Modifier

– Conjunctions of numerous functional categories

– Relative Clauses, Wh-Clauses, Infinitive, -ing, -en & Bare Verb Clauses

– Long-distance dependencies

– Passive constructions

24

Start with a Domain General Language Processing System

• Representations are in the spirit of the “Simpler Syntax” of Culicover & Jackendoff (2005) except that there are no purely syntactic representations

Semantic Features

Trace bound

to subject

Functional

Categories

Referring Expression

Predicates

He is eager to please.

25

Extend to Handle Scripted Comm

• AVO: DEMPC, please let me know the first waypoint!

• DEMPC: The first waypoint is LVN. It’s an entry point. There are no airspeed or altitude restrictions. The effective radius is 2.5 miles.

• AVO: PLO, I’m heading towards LVN.

• DEMPC: We’re within the effective radius so go to the second waypoint.

• AVO: Are there any altitude or airspeed restrictions for the second waypoint?

• DEMPC: The second waypoint is H-AREA. It’s a target. The airspeed restriction is between 50 and 200 knots. There is no altitude restriction. The effective radius is 5 miles.

• PLO: AVO, please keep the altitude over 3000 feet for the photo!

• PLO: I have a good photo of H-AREA.

26

Scripted Comm

• Full sentences

• Correct spelling

• Explicit discourse acts

• Still lots of variability

– Declarative sentences

– Imperative sentences

– Questions

– Conjunctions

27

Extend to Handle Text Chat for a 40 Minute Mission – without editing!

• PLO to AVO: avo-don't ever proceed from a target if i haven't taken the picture

• AVO to PLO: ok -- keep me in the loop!

• INTEL to all: ok team, mission 2

• PLO to AVO: effective radiu

• PLO to AVO: avo i need to be below 3000

• AVO to PLO: copy, will 2000 do?

• DEMPC to AVO: LVN is our 1st entry point with a radius of 2.5

• AVO to PLO: speed?

• AVO to DEMPC, PLO: 1 mile out/ 30 seconds

• PLO to AVO: i don't have a speed for lvn so go faster

• AVO to DEMPC, PLO: speed 340

• PLO to AVO: avo i'll need to be above 3000 for h area

• AVO to PLO: above 3000 copy -- can we proceed to h-area yet?

28

Extend to Handle Text Chat for a 40 Minute Mission – without editing!

• PLO to AVO: lets get out of effective zone

• DEMPC to AVO: Speed=50-200, Altitude=500-2000

• AVO to DEMPC, PLO: wait -- my flight plan changed -- are we going to Z1?

• PLO to AVO: can yougo faster yet or is it stll 200

• DEMPC to AVO: no speed or alt. restrictions

• PLO to AVO: avo i need to be above 3000 for s ste- go there when you think it would be most effective

• PLO to AVO: avo 3000

• DEMPC to AVO: YES to S-StE=Target

• PLO to AVO: `avo get back within 5 miles of s ste

• PLO to AVO: aavo dont slow down

29

Handle Communication with Unscripted Human DEMPC and PLO

• Language varies significantly from team to team

– Can’t predict vocabulary requirements in advance

• Teams adapt particular ways of communicating which can’t be predicted in advance

– Text becomes more cryptic as mission continues

• Discourse acts are often implicit

30

Word Recognition Subcomponent

• Word recognition subcomponent largely compatible with the E-Z Reader model of reading (cf. Reichle, Warren & McConnell 2009) with extensions to support higher-level language processing

• Perceptual window used for low-level processing of linguistic input

– Model can “see” space delimited “word” in focus of attention

– Model can “see” up to first 3 letters of word in right periphery following space

• Retrieved word is verified against actual input

– Consistent with Activation-Verification model of Word Recognition (Paap et al. 1982)

31

Word Recognition

• Word recognition is an interaction between low-level perceptual and higher-level cognitive processing

• Perceptually identified letters, trigrams and space delimited “words” spread activation to words (and multi-word units) in DM

• Most-highly activated word or multi-word unit consistent with retrieval template is retrieved

– Need not be a space delimited “word”

32

Generating Linguistic Representations

• Incremental, interactive generation of linguistic representations which encode referential and relational meaning

Referring Expressions

Relations

He is eager to please.

33

Mapping into the Situation Model

• Referring expressions in the linguistic representation get mapped to objects and situations in the situation model

• Indefinite object referring expression typically introduces a new object into the situation model

• Definite object referring expression typically identifies and existing object either in the situation model or salient in the context

• Situation referring expressions typically introduce a new relation into the situation

34

Language

Comprehension

Language

Generation

Dialog

Manager

Situation Model

System Overview

Text Chat

Output

Motor

ActionsVisual

Input

Text Chat

Input

Task Behavior Model

35

Centrality of Situation Model

DomainKnowledge

Task Behavior

World KnowledgeSituation

Model

Language OutputLanguage Input

Language Knowledge

Task Input

36

Situation Model

• Situation Model (Zwann & Radvansky, 1998)

– Spatial-Imaginal (and Temporal) representation of the objects and situations described by linguistic expressions and encoded directly from the environment

• Non-propositional (at least in part)

• Non-textual

• No available computational implementations

– Provides grounding for linguistic representations

– Integrates task environment and linguistic information

37

Abstract Concepts vs. Perceptually Grounded Language

“pilot”“pilot” “pilot”

PILOT

Real World Mental Box Mental BoxReal World

perception Language

of Thought

The Prevailing View An Emerging View

gro

un

din

g

perceptionImplicit(Abstract)

Explicit(Perceptual)

38

Abstract Concepts vs. Perceptually Grounded Language

“pilot”“pilot” “pilot”

PILOT

Real World Mental Box Mental BoxReal World

perception Language

of Thought

The Prevailing View An Emerging View

gro

un

din

g

perceptionImplicit(Abstract)

Explicit(Perceptual)

39

Situation Model

• Propositional Content

– Planning to use Hobbs’ theory of “ontological promiscuity” and his well-developed logical notation (translated into ACT-R chunks) to represent propositional content

• The logical notation should be as close to English as possible

• The logical notation should be syntactically simple to support inferencing

40

Situation Model

• Spatial Content

– Planning to use Scott Douglass’ spatial module extension to ACT-R which implements a matrix-like representation of spatial information

• Discourse Content

– Working on identification and representation of Discourse Acts which are often only implied in linguistic input

• “I need to be above 3000 feet for the photo”

–This is a request to increase the altitude of the UAV (human is not actually in UAV)

41

Empirical Validation

• Experiment conducted with human subjects in conditions using 1) spoken language and 2) text chat to provide data for model development

– AVO station moved into separate room so DEMPC and PLO don’t see AVO

– Text chat condition showed team performance effect similar to spoken language condition

• Goal is to conduct an experiment with Synthetic AVO Teammate interacting with human DEMPC and PLO

42

Questions?