AQUAINT TestbedActivity

John Burger, Marc Light, Scott Mardis, Alex Morgan

The MITRE Corporation

© 2002, The MITRE Corporation

Models of Technology Transfer

Research Transfer

Iterative feasibility experiments

End of program handoff

Parallel Transfer

MITRE’s role in AQUAINT

ResearchersDeployment

User Eval& Analysis

User

User Testbed

EA

EA

EA

EA

EA

What types of things we might learn

Performance constraints- System response needs to be shorter than X, if a search-

engine-like interaction is to be used.Features needed

- Questions tend to be answered by only one source and thus source selection could be a win.

- Questions involving temporal reasoning are frequent.

- Answers tend to occur only once in the sources.User interface feedback

- Users prefer knowing answer source and pedigree.System internal

- Question sequences tend to utilize a small pool of documents.

- Tables tend to contain answers to "how many" questions.

Testbed DimensionsGoal: give analysts useful QA capabilities

Features & Functionality what question types what answer types

User interface speed of technology access technology tool suite vs. system

Data feeds vs. static text vs. speech vs. IM structured vs unstruct multi vs. monolingual

ResearchDimensions

OperationalDimensions

Users, Domains,Scenarios

Plumbing server platform existing clients network protocol

MeasuringUsefulness instrumentation surveys

Picking the Right Point(s) in the Space

One can start with any one dimension and constrain the others

However, we recommend

- Start with users and domain

- The data, functionality, and interface dimensions are then greatly constrained

- The scenarios constrain the measuring usefulness dimension

Start with “easy” point Multiple points will be needed

Example: Nuclear Non-Proliferation

Features & Functionality question types: ? answer types: ?

User interface ?

Data unstructured English

language reports on non-proliferation, country reports

PPNN database mostly static, some live

Users IC analysts

Domainsnuclear non-proliferation

Scenarioscreating country profiles

Plumbing classified @ IC-TESTNET unclass @ mitre.org

MeasuringUsefulness instrument server Research

DimensionsOperationalDimensions

Example: Chemical Transportation Emergency Center

Features & Functionality list questions factual questions temporal facts believability of data is NOT an issue

User interface ?

Data feeds and static text, telephony chem Hazard DB English Research

Dimensions

Users, Domains, Scenarios CTEC

Plumbing no issues with classified data

MeasuringUsefulness instrumentation surveys

OperationalDimensions

Other User Group Possibilities

AQUAINT researchers- news

Air Force Information Warfare Center- computer networks & security

Armed Forces Medical Intelligence Center- biological threats

Focus group from multiple intel agencies- unified by a single topic of interest

We will reach a decision on an initial user group by Fall, 2002

Purpose of this demoGain some momentum

- Experience for MITRE- Experience for AQUAINT contractors

Jump start discussion- What do we really want?

The testbed is not intended for comparing systems- Not a different kind of TREC

Integration is desirable, but not a requirement- Fielding individual system may make sense

User modality is wide open- Browser, chat, newsgroup, email, ...

Demo Architecture

MITRE

Web server

LCC QAserver

MITREQA

server

CMU QAserver

question1

answers4

question2

answers3

3

3

HTTP TCP

QA server connections use ad hoc protocol over TCP

HTTP would simplify network securityissues

Next steps:

- add more sites

- collate answers: merge, cluster, rank

- preview to small number of users

Testbed Stages for Developers

QA servers located at development sites- promoting stability- provides access to open source collections- on analysts’ desktop

QA servers located at mitre.org- rehearsal for classified environment- packaging, simplifying distribution- performance improvements?

QA servers located on IC-TestNet- operational data- limited access (classified, non-internet, separate

physical lab)

Classified network- access to users, data, scenarios may be restricted

Available at multiple agencies- potentially accessible by many analyst groups- access is not from the analysts’ primary

environment- user labs in the planning and construction stage

No electronic connections to IC-TestNet- occasional updates possible, but static data easier - no remote monitoring or maintenance !

Requirements- 5-10 page proposal- may need some hardware

IC-TestNet

Deploy here only after some successes in unclassified environments ! !

Architecture

MITRE

QAPortal

Datacapture

QAserver

QAserver

QAserver

One way to do it.

documents1

question3

question4

answer5

answer6

documents2

(Every n hours/days)

QAserver

Architecture

MITRE

QAPortal

IREngine

QAserver

Another way.

question2

question3

answer6

documents5

query 4answer

7

documents1

QAserver

IR or KBserver

Component Integration

Collaborate with AQUAINT’s full system contractors- share service or share executable- keep us in the loop so that we can compile and

promote best practices, and reduce duplicate effort

Integrate with QANDA- download from

http://www.openchannelfoundation.org- wrap module- MITRE will assist

Testbed Schedule

Preliminary user group(s) identified Aug, 2002

Identify tasks/scenarios Aug, 2002

Propagate constraints Sept, 2002

Non-classified MITRE-hosted QA portal- installation Oct, 2002- pilot study Dec, 2002

Classified (IC-TestNet)- proposal Dec, 2003- hardware installation Mar, 2003- software installation May, 2003- pilot study Sept, 2003