-FMCReport No. R-6267

Control for Intelligent Tutoring Systems:

A Comparison of Blackboard Architectures and

Discourse Management Networkscv)

'U--

0William R. Murray

ISeptember 1988

DTIC& - --ECTEll

This report was prepared under the NavyManpower, Personnel and Training R&DProgram of the Office of the Chief ofNaval Research Under Contract NumberN00014-86-C-0487.

Reproduction in whole or part is per-mitted for any purpose of the UnitedCentral Engineering Laboratories States Government. Approved for public

Santa Clara California release, distibution unlimited.

81 .0t'.

Control for Intelligent Tutoring Systems:

A Comparison of Blackboard Architectures and

Discourse Management Networks

William L Murray

September 1988

Thi report wue prepared under the Navy Manpower, Personnel and Traiing R&D Program of - -

the Office of the Chief of Naval Research Under Contract Number N00014-86-C-0487.

Reproduction in whole or part is per mitted for any purpose of the United States Government.Approved for public relesse, distribution unlimited.

JJCRITY CLASSIFICATION OF TWOS5 PArE

I REPORT DOCUMENTATION PAGEI& REPORT SICURITY CLASSIUICAt"O 16 RISTW1Iv MARKWGSUNCLASSIFIED

2a SECURITY CLASSIFICATION AUTH4ORITY I DITEOU1111AVAASMITY Of REPORT

2b O0L11C 9 &ICTION'O~WNGRADING SCHEDULE APPROVED FOR PUBLIC RELEASE,

4 PERFORMING ORGANIZAION REPORT NUMIER(S) S 11o01101101111 OR1GANIZATION REPORT NUMIIERES)

R-6267

64 NAME OF PERFORMING ORGANIZATION I bOFIESMIO. 7a, fA OF MONTORING ORGANIATION COGIIECENTRAL ENGINEERING LABORATOR IE (i pjcw) OFFICE OF NAVAL RESEARCH - PROGRAE

6C ADDRESS (Qtp. St01t. ad ZIP Code) ?b. AOMSS of. St. &-W J ICWFMC CORPORATION OFFICE OF NAVAL RESEARCH (Code 1142PT)1205 COLEMAN AVENUE 800 NORTH QUINCY STREETSANTA CLARA, CA 95050 ARLINGTON, VA 22217-5000

11a. NAME Of OUNOINGSPONSORING lb OFFICE SYMSO I , QUM OI w$TRtAjft IINYICATIOW NUM111RORGANIZATION (if apowable

222 N00014-86-C-04878c. A00411[SS (City. Stato. and ZIP Co*.) 10 SCUMI OP IWICING NUM0OIRS

800 NORTH QUINCY STREET PFGRP PROACT TASK WWORK UNITARLINGTON, VA 22217-5000 ELEMEUTno. NO. NO0. CSS11101NO1

________________________________________ 622,33N RM33M20

I I TITLE (Include SuCwrfY 0Claua0Oat1een)CONTROL FOR INTELLIGENT TUTORING SYSTEMS: A COM'PARISON OF BLACKBOARD ARCHiTECTURES ANDDISCOURSE MAN4AGEMENT NETWORKS

12 PERSONAL AUTH4OR($)WILLIAM R. MURRAY

13a TYPE OF REPORT 13b TINA( COVERED 14 DATE OP REPORT (Yost. ASONK Day) S PAGE COUNTTECHNICAL IFROM TO 1988, SEPTEMBER, 16 2

'~ UPPEMETAY NTATON Supported by the Office of the Chief of Naval Research Manpower,Personnel, and Training R&D Program.

ICOSATI COOES I I SUSGECT TERM lrwt &I awf mew amnE~ - wa b bf Nec neftFIED GOUPI UBGROP 'INTELLIGENT TUTORING SYSTEMS.INSTRUCTIONAL PLANNING,

DYNAMIC PLANNING, BLACKBOARD'ARCHITECTURE, DISCOURSEI I ~MANAGEMENT NETWORK, Av~bVi~ J

.9 AISTKACT (Conbwm an 'evOM If*0 .matyed i*WAF~ 6V bl00d 11110111

An intelligent tutoring system must have some control mechanism for deciding what instructionalaction to perfonm next. Traditional CAI systems have inflexible control mechanisms that procedurallyencode tutorial strategies and prevent them from being readily extended to handle new domains ortutorial strategies. Two alternative architectures proposed recently to address these problems arediscourse management networks and blackboards. This paper compares these two architectures andargues that an intelligent tutoring system controlled by a blackboard architecture can deliversignificantly more effective and flexible instruction than one controlled by a discourse managementnetwork. - r 1

20 DISTRIGUTIOF4 IAVAILABIUITY Of AISTRACT 2t. A*BSTKT SICM"I CLASSIFI11CATMOOU4CLASSIFIIOUNUMITEO In SAME AS RPT DO0T USERS UNCLASSIFIED

Us. PtAM JlQFROM I y~j lWA b 1111E110E11111ONWO AM 2h.Ili OPPI STMIO

L~t~.3U~M1Lfl±rr1 (2021) 696-&'-iiq ONR IL4C00 FORM 1473, &4 MAR 53 APR edition maye be wsd W saftod. SEUT PAS P"M OF nTHS pFu

All 01the *dition are 46=11.

ECUftery CLAMW¢ICATIG Of T19 PAst

The key components of each architecture are compared and related to the functionality that results.The key components of the blackboard architecture are a global database, independent knowledgesources, and agenda control. In contrast, discourse management networks are much simpler, resembling

finite state machines or augmented transition networks. Discourse management networks arecharacterized by explicit tutorial states and a fixed control mechanism that handles transitions betweenstates. They are well-suited for supporting opportunistic tutoring strategies, but they must be coupledwith other control mechanisms to support lesson planning, discourse that requires dialog planning, andmixed-initiative interaction where student actions can invalidate assumptions underlying the currentdiscourse plan. Blackboard architectures provide more flexibility in handling discourse situations andfor the integration of lesson planning and discourse planning. Furthermore, blackboard architecturesmore easily support multiple tutorial strategies and the develoi. ment of customized, globally coherentcurriculum plans and lesson plans. Both architectures explicitly represent tutorial strategies apart fromdomain knowledge to simplify the construction of tutors for new domains. However, the blackboardarchitecture has the additional advantage that it facilitates the separation of knowledge about planningfrom both domain knowledge and tutorial strategies.

These claims are supported by considering an architectural formalization of each architecture andthe way in which each architecture's key components facilitate or hamper the dynamic planning ofinstruction. The relationship between this ability to plan instruction dynamically and to generateeffective and flexible instruction is also supported, justifying the utility of this kind of comparison. Toground these discussions, two examples of each architecture are provided: GUIDON and MENO-

TUTOR exemplify discourse planners implemented as discourse management networks while IDE-INTERPRETER and the BLACKBOARD INSTRUCTIONAL PLANNER exemplify plannersimplemented in the blackboard architecture. Application of each architecture to control an intelligenttutoring system to teach LISP is also considered to further illustrate differences in capability that resultfrom choice of architecture.

Accession For

NTIS GRA&IDTIC TABUnannounced 0Justification ,,

Bycopy

Distribution/ 4

Availability Codes

Ava and or

Dist Speci1a1

.........

Control for Intelligent Tutoring Systems: A Comparison ofBlackboard Architectures and Discourse Management Networks

William R. Murray

Artificial Intelligence CenterFMC Corporation

1205 Coleman Avenue, Box 580Santa Clara, CA. 95052

Abstract

An intelligent tutoring system must have some control mechanism for deciding what instructional

action to perform next. Traditional CAI systems have inflexible control mechanisms that procedurallyencode tutorial strategies and prevent them from being readily extended to handle new domains ortutorial strategies. Two alternative architectures proposed recently to address these problems are

discourse management networks and blackboards. This paper compares these two architectures andargues that an intelligent tutoring system controlled by a blackboard architecture can deliversignificantly more effective and flexible instruction than one controlled by a discourse management

network.

The key components of each architecture are compared and related to the functionality that results.

The key components of the blackboard architecture are a global database, independent knowledge

sources, and agenda control. In contrast, discourse management networks are much simpler, resemblingfinite state machines or augmented transition networks. Discourse management networks are

characterized by explicit tutorial states and a fixed control mechanism that handles transitions betweenstates. They are well-suited for supporting opportunistic tutoring strategies, but they must be coupledwith other control mechanisms to support lesson planning, discourse that requires dialog planning, andmixed-initiative interaction where student actions can invalidate assumptions underlying the current

discourse plan. Blackboard architectures provide more flexibility in handling discourse situations andfor the integration of lesson planning and discourse planning. Furthermore, blackboard architecturesmore easily support multiple tutorial strategies and the development of customized, globally coherent

curriculum plans and lesson plans. Both architectures explicitly represent tutorial strategies apart from

domain knowledge to simplify the construction of tutors for new domains. However, the blackboardarchitecture has the additional advantage that it facilitates the separation of knowledge about planningfrom both domain knowledge and tutorial strategies.

These claims are supported by considering an architectural formalization of each architecture and

the way in which each architecture's key components facilitate or hamper the dynamic planning ofinstruction. The relationship between this ability to plan instruction dynamically and to generate --

effective and flexible instruction is also supported, justifying the utility of this kind of comparison. To

ground these discussions, two examples of each architecture are provided: GUIDON and MENO-

A

TUTOR exemplify discourse planners implemented as discourse management networks while IDE-INTERPRETER and the BLACKBOARD INSTRUCTIONAL PLANNER exemplify plannersimplemented in the blackboard architecture. Application of each architecture to control an intelligenttutoring system to teach LISP is also considered to further illustrate differences in capability that resultfrom choice of architecture.

FIL

mA

Table of Contents1. Introduction I

I.I. Importance of Dynamic Instructional Planning in Intelligent Tutoring Systems I1.2. Types of Instructional Planning 3

2. Discourse Management Networks 42.1. Architectural Formalization 42.2. Two Exemplars 52.3. Support for Dynamic Instructional Planning 8

3. The Blackboard Architecture 93.1. Architectural Formalization 103.2. Two Exemplars 103.3. Support for Dynamic Instructional Planning 12

4. Scenario: Controlling a Programming Language Tutor 144.1. Scenario I: Dynamic Instructional Planning in the Blackboard Architecture 144.2. Scenario 2: Opportunistic Tutoring with a Discourse Management Network 15

5. Summary 17

List of FiguresFigure 2-1: The discourse management network architecture 5Figure 2-2: MENO-TUTOR's Discourse Management Network 6Figure 2-3: Student program in MENO-TUTOR dialog 7Figure 2-4: GUIDON's Discourse Management Network 8Figure 3-1: Blackboard Interpretation Cycle 10Figure 3-2: Domain and Control Blackboards of the BLACKBOARD II

INSTRUCTIONAL PLANNERFigure 3-3: Categories of instructional actions in the BLACKBOARD 11

INSTRUCTIONAL PLANNER

b-

I

1. IntroductionThe research discussed in this paper addresses the problem of control for intelligent tutoring

systems. At any point during instruction a tutoring system must select from many possible instructionalactions. For example, topics can be introduced, reviewed, motivated, summarized, explained in depth,related to earlier topics, etc. Similarly, student knowledge can be assessed by true/false or multiplechoice tests, direct questioning, self-assessment, and many other means. The problem of control is toselect the most effective action, given the current lesson objectives, student model, tutorial strategy,subject matter, and resources available.

An important decision in designing an intelligent tutoring system is the choice of architecture forthis control mechanism. Traditional CAI systems have inflexible control mechanisms that procedurallyencode tutorial strategies, and which prevent them from being readily extended to handle new domainsor tutorial strategies. Two alternatives proposed recently to address these problems are discoursemanagement networks and blackboard architectures. This paper compares the advantages anddisadvantages of using these two architectures for control in intelligent tutoring systems. Based on thiscomparison, we argue that blackboards address important limitations of discourse management networksthat limit their ability to deliver customized, globally coherent, planned instruction, and to support moreflexible mixed-initiative instruction.

The framework for comparison is the ability of each architecture to support the dynamic planningof instruction. Dynamic instructional planning is a planner-based approach to control for intelligenttutoring systems in which appropriate plan generation' and revision occur during the tutorial session andin response to the changing tutorial situation. A dynamic instructional planner reasons about alternativelesson objectives, and alternative instructional plans to realize the tutorial strategies it selects. We arguethat such planning is necessary to build effective, robust, and flexible tutoring systems and thatblackboard architectures provide better support for this planning than discourse management networks.Before comparing the two architectures, we first justify the relevance and importance of dynamicinstructional planning to intelligent tutoring systems.

1.1. Importance of Dynamic Instructional Planning in Intelligent Tutoring SystemsWhy should tutoring systems incorporate dynamic instructional planning? Two points must be

supported here: first, that a tutoring system should have a plan; and second, that it should be able toplan. It should have a plan to properly manage its time and generate globally coherent instruction.Proper time management prevents spending too much time on relatively unimportant topics or packingtoo many topics or exercises into one lesson. Global coherence means that topics are logicallyconnected to support instructional objectives and that topics are sequenced and presented in a mannersensitive to the student's perceived knowledge, the tutor's instructional objectives, the student's interest

IFor the purposes of this paper we allow plan selection to be considered a trivial form of plan generation, otherwisediscourse management networks would not support dynamic instructional plaming merely by their inability to assemble aplan during instruction.

and motivation, and the time available for lessons. Transitions between topics are smooth and the tutoris consistent; that is, it does not contradict previous or future instruction. One sign of incoherentinstruction is recurring student confusion about the tutor and its intentions, resulting in distractions fromthe subject matter being taught.

Another advantage of having a plan is the ability to introduce material in a layered fashion, firstproviding an overview of the material and lesson plan, and then introducing successive layers of detailthat build on and refine earlier material. A plan also allows the tutor to recognize opportunities tomotivate and lead into future instruction in response to unexpected student questions, or to deferanswering the questions since they will be addressed later in the lesson. The tutor can use a plan toselect a sequence of problem-solving cases that cover a set of topics, and to focus its conversation on thecentral topics for each case, deferring or abbreviating discussion of other topics best addressed insubsequent cases. Finally, a plan assists in explaining and motivating current instruction based on itsrelationship to future instruction.

One alternative to dynamic instructional planning is for the tutor simply to interpret a highlydetailed plan provided by a human curriculum author; this approach is used in traditional computer-assisted instruction (CAI). Although theoretically the plan could provide for any eventuality that mightarise, practically the combinatorics of different student responses and tutorial states require that studentinitiative be curtailed, fine-grained student modeling avoided, and a significant amount of time spentpreparing lesson plans. One goal of intelligent tutoring systems is to explicitly represent the tutorialstrategies that ate encoded procedurally in such a system, allowing for more rapid development of tutorsfor new instructional domains, and experimentation with alternate strategies. 2 By providing intelligenttutoring systems with a planning capability, their reliance on human-generated plans is reduced whiletheir ability to generate the high-quality expository instruction of well-crafted CAI systems is increased.

A tutor that can plan is also better able to handle a mixed-initiative dialog than a tutor that cannotplan. Replanning can be used to handle topic transitions brought about by student questions and torevise lesson plans to omit or cover topics to satisfy student requests. Again, the combinatorics of thedifferent possible student requests, questions, and tutorial states argue for representing the knowledgefor handling student initiative in planning knowledge rather than as different plan contigencies in a pre-stored plan. This planning knowledge allows the tutor to plan topic transitions, to replan in response tothe amount of time remaining in the lesson, and to decide how to return to the current topic or whetherto abandon it altogether.

Planning knowledge is used not only to customize the lesson plan in response to student requestsduring the lesson, but also before the lesson begins. A tutor that can plan can generate an initial lesson

21ntelligent tutoring systems also differ from CAI systems by their emphasis on sophisticated student modeling, thetailoring of imtruction to the student model, and the ability to solve problems, model expertise, and azuwer questiom aboutthe domain they are teaching.

3

plan customized to the student's interests, assessed capabilities, and background. The lesson plan can berevised as the tutor's student model changes or in response to student requests. Instead of the tutorassuming a fixed set of instructional objectives for all students, the student can request instruction onparticular areas of a subject. As before, it is impractical to anticipate the number of different studentinterests, objectives, and backgrounds in teaching complex subjects. It is more economical to representlesson planning knowledge than to provide lesson plans for all the different possible combinations. Therepresentation is more economical in the sense that the same knowledge is represented explicitly in aconcise form and not replicated implicitly in the branches of a single highly conditionalized lesson plan,or as a very large number of lesson plans unique to different situations.

These arguments for the tutor being able to plan rest on a subtle point that is worth reemphasizing.Simple plan-fotlowing tutors (CAI) can always replicate the behavior of a dynamic instructional plannerfor any particular situation. However, the dynamic instructional planner is better able to handle thecombinatorial explosion of different tutorial situations when mixed-initiative instruction is allowed, andwhere lesson plans are tailored to different student models and to varying amounts of time for eachlesson. The CAI system limits mixed-initiative dialog and lesson customization, and omits or simplifiesthe student model to reduce the number of different tutorial situations. The dynamic instructionalplanner imposes fewer limitations since it can apply its planning knowledge to the different tutorialsituations that arise.

The economic representation of planning knowledge in a dynamic instructional planner alsocontributes to its greater ease of use than that of a CAI system. The CA system procedurally encodesplanning knowledge and implicit assumptions about the tutorial state at each branch of its pre-storedinstructional plan. The planning knowledge in a dynamic instructional planner makes fewerassumptions about the tutorial situation and can thus be more readily applied in the construction of newtutoring systems.

1.2. Types of Instructional PlanningWe may distinguish three levels of instructional planning:

* Curriculum planning - planning an extended sequence of lessons for a subject.

* Lesson planning - determining the subject matter to present in a single lesson, and the orderof presentation.

* Discourse planning - planning communicative actions between the tutor and student withina lesson.

These levels cannot in practice be so cleanly separated and frequently discourse planning and lessonplanning are intertwined, as are lesson planning and curriculum planning. Typically, a humaninstructor's lesson plan will include not only a sequence of topics but also some common, discourseprocedures such as collecting homework or having students solve and explain problems on the board.This three-level distinction is useful here since discourse management networks support primarily

4

discourse planning3 while the blackboard architecture supports all three kinds.

The remainder of this paper compares discourse management networks and blackboards withrespect to their support for dynamic instructional planning. Section 2 formalizes the discoursemanagement network architecture, provides two examples, and then discusses this architecture's supportfor dynamic instructional planning. Similarly, Section 3 formalizes the blackboard architecture,provides two examples, and then discusses support for dynamic instructional planning. Next, a specificapplication, control of a LISP tutor, is considered in Section 4 to illustrate the advantages of theblackboard architecture. The last section summarizes these arguments.

2. Discourse Management NetworksDiscourse management is the selection by the tutor of the next discourse action or intended

sequence of actions. In this architecture, the tutor does not reason about the results of discourse actionsor future tutorial situations; the only planning performed is plan selection or skeletal planning.However, planning is not a prerequisite for flexible behavior [10] and tutors built with this architectureallow sophisticated control of tutorial dialogs.

A discourse management network (DMN) is a kind of procedural network similar to an augmentedtransition network. Nodes represent tutorial procedures or actions corresponding to tutorial states. Arcsrepresent state transitions. The tutor is in one state at any time. The state determines what actions areperformed. After performing the actions or procedures, control follows one of the arcs leaving from thestate. The choice of arc, if there is more than one, depends on the predicates on the arcs.

2.1. Architectural FormalizationFigure 2-1 represents the key features of the discourse management network architecture. Circles

represent tutorial states named SO... S n and Sdefault. Arrows represent possible transitions to other states,not all of which are shown. Assume the tutor is originally in state So. Then A0 represents the actionsthat will be performed in state So. The heavy line represents a default transition to the state Sdefault.Other lines from So represent other possible transitions that can be taken. Each predicate PI...Pn isconsidered in turn and the first true predicate causes control to transfer to that state. If no predicateP... Pn is true then the default transition is taken. The discourse management network has no memoryother than the current state and the registers RI...Rm. These registers can be set to scalar or symbolicvalues by any of the actions and accessed in any of the predicates. Actions can only set registers orperform communicative actions (e.g., give a test). Actions cannot change control in any way other thanthat described above. For example, an action cannot change the current state or the arcs emanating from

a state.

3Plaming in the sense of choosing a course of action, not in the sense of masoning about the results of actions andprojecting future discourse situations.

SOS

so [Pii AC s it w

S2

Figure 2-l: The discourse management network architecture

As formalized here, discourse management networks are a kind of finite state machine withregisters used to reduce the number of states. Usually discourse management networks are coupled withother control mechanisms (e.g., an agenda) and an external memory to provide a topic selection -mechanism. Two examples of discourse management networks are discussed below, along with

extensions they make to the abstract model above.

2.2. Two ExemplarsMENO-TUTOR (12] uses a discourse management network to control a Socratic question and

answer dialog with a student that alternately probes the student's knowledge and refines his knowledgeby teaching new information or correcting misconceptions. Figure 2-2 [13] shows the state transitiondiagram of MENO-TUTOR. Numbered states indicate a possible sequence of states that might occur ina dialog. It differs from the abstract model above in its use of hierarchical abstraction: tutorial states atthe bottom of the diagram are refinements of the tutorial states above them. In this diagram onlypossible transitions from more abstract to less abstract states are shown; default transitions are notshown. The actions of a particular state result in the generation of a single utterance by a surface naturallanguage generator.

MENO-TUTOR also differs from the idealized model in its use of meta-rules to represent statetransitions other than to default states. Rather than explicitly represent these arcs and their predicates,the meta-rules operate as demons that can override default transitions. This movement to a new statefrom a specified prior state happens only when a meta-rule's trigger condition is true. Since the meta-rule can be replaced by arcs to the new state with predicates using the meta-rule's trigger condition, themeta-rules provide only a more compact representation but no added functionality compared to theabstract model above.

MENO-TUTOR was reverse-engineered to produce Socratic question and answer dialogs similar tothose obtained from human protocols. One such dialog that MENO-TUTOR could produce, if coupledto a surface language generator, is shown below (these examples are from [13]). The Pascal programbeing referred to is shown in Figure 2-3.

Tutor. Do you know that GRADES, in line 9, is a control variable for the WHILE loop?

_!1

6jeOAOCO-C *r'* 1t T 0t. ̂ .re eL tc. STA,, 9 t o

T - I Ii-lA r Tt e

Figure 2-2: MENO-TLTOR's Discourse Management Network

Student: Yes

Tutor: Does that mean that GRADES should be

(a) Initiated before the loop?(b) Added into SUM within the loop?(c) Modified within the loop?(d) Both (a) and (c)

Student: C

Tutor You seem to understand about the control variable of the WHILE loop. Now let's talk aboutsome other command statements.

What do you think READ (GRADES) in line 8 does with the value of GRADES?(a) Reads all values of GRADES at one time.(b) Reads in a single value of GRADES.(c) Assigns several new input values to GRADES.

Student: A

Tutor Then do you think that GRADES := GRADES + I updates the next of those values?

Student: Yes

Tutor: There is your misconception. Adding one to a variable is a common way to step through arange of values. But it cannot apply here because your READ instruction in line 8 retrieves only asingle value. You should replace line 13 with a second call to READ (GRADES). -

7-I

1 PROGRAM LESSON1 (INPUT, OUTPUT);2 VAR3 SUM, GRADES, STUDENTS: INTEGER;4 IMDLN: REAL;5 BNGIN6 SUM:0;7 STUDENTS::0;8 READ (GRADS);9 WHILE GRADES>0 DO10 BEGIN11 SUM:=8UM + GRADES;12 STUDENTS : STUDENTS+1;13 GRADES:=GRADES+l;

should be READ(GRADES); -I14 ZiND;

15 MDIAN: -SUM/STUDENTS;16 WRITELN17 ('TE DMDIAN GRADE IS', MEDIAN:8:3)18 END -

Figure 2-3: Student program in MENO-TUTOR dialog

GUIDON [21 extends the utility of the discourse management network formalism by introducing

quite sophisticated tutorial actions. Instead of simple actions, such as suggesting a new topic, a packet

of tutorial rules (t-niles) can be invoked. Such a collection of t-rules is called a discourse procedure, and

t-rules themselves can call other discourse procedures. The result is that most of the selection of

di course actions occurs as a result of the operation of the t-rules within a state, and the discourse

management structure is more of a convenient means of organizing discourse procedures, rather than the

primary control resulting in the selection of discourse actions.

The domain of discourse for which GUIDON was developed is a MYCIN consultation. A medical

student using GUIDON attempts to learn or refine his understanding of the knowledge represented byMYCIN's diagnostic rules. Discourse topics include MYCIN rules, goals, clinical data, and student

hypotheses. Discourse procedures are organized around these topics as shown in Figure 2-4 ( [31, page

59). Selection of the next discourse procedure is also performed within each state, thus the predicates on

the arcs are implicit in the t-rmles that determine the next discourse procedure to select.

The point here is not that GUIDON's enhancements alter the DMN architecture or that GUIDON

does not handle discourse in a sophisticated way (since it does), but that GUIDON's power comes from

its use of these discourse procedures and t-rules and not from the underlying discourse network

architecture. The discourse management network architecture - the states, arcs, and predicates - is not

where most of the decision-making is performed regarding what discourse action to take, rather this is

performed by the numerous t-rules in each state.

RULE GOAL

Q \ndeeper$I~r GOAL DT0oc

recent- success eO blockrecent - failuresro o t - a lu e no-n ted-to -ask

0 0

all-rules-fail 0

completed

present-valu s summary HYPOTHESIS Q

/0 present-values

advice

known-by- definition

presnt- values RULE

end-of -rul) related-rules

Figure 2-4: GUIDON's Discourse Management Network

2.3. Support ror Dynamic Instructional PlanningThe main feature that discourse management networks lack to support dynamic instructional

planning is an explicit plan representation that can be generated and interpreted during instruction. Thelack of such a repnsentation hampers:

a Incremental planning - since there is no explicit representation of partially specifiedsubplans or operators, or representation of constraints on those parts of the plan that havenot been elaborated to the level of primitives.

* Plan repair or improvement by revising existing plans - since there is no plan data structurethat can be modified and then subsequently interpreted.

* Gracefully suspending, resuming, or abandoning plans - since suspension and resumptionof instructional plans requires more than placing a plan on a stack and then popping it offlater. Instead a coherent line of discourse must be maintained in suspending or abandoninga plan and moving to an entirely new or resumed plan. Topic transitions may need to beplanned or an explanation generated for why the current line of instruction is now changing.

4

1. 9

Sophisticated dialog management in GUIDON and MENO-TUTOR is still maintained, essentiallyby dynamically selecting from pre-stored discourse plans.4 This dynamic selection allows reactivediscourse that responds to student questions and requests, but still does not provide the flexibility andglobal coherence that dynamic instructional planning could provide. Two examples from GUIDON willillustrate limitations that can be addressed by dynamic instructional planning. First, note thatGUIDON's discussions can be verbose and lacking global direction since GUIDON does no planningbefore the dialog begins. Instead, GUIDON considers only one level of MYCIN's AND-OR goal tree atany time when controlling its dialog. Clancey [3] discusses two deficiencies that became apparent whenGUIDON was applied to the nonmedical domain of SACON (structural analysis):

First, better overviews are required. A SACON dialogue is tedious because GUIDON methodicallyguides the dialogue in a depth-first way. only motivating the discussion one step at a time. To convey aglobal sense of purpose, the program must reason about the relation between subgoals (e.g., distinguishbetween definitions and hierarchical subtype) and summarize subtrees.

Second, the constraint on time for the session prevents treating each "context" (e.g., culture, loading,organism) with equal emphasis; some contexts must be omitted or mentioned in passing. WhileGUIDON has a number of methods for reducing the number of rules that are mentioned and hasmethods for mentioning them economically, it never looks down into the solution, beyond one level ofthe tree, to decide what topics to focus on. ( [3), pages 230 - 231)

GUIDON is also limited in its ability to tolerate student interruption since it lacks dynamic instructionalplanning. Although students can interrupt, students cannot receive new case data during the interruptionsince the current (suspended) discourse procedure or tutorial rule may need to be abandoned or modified

S to account for the student's new knowledge. GUIDON cannot do this and so attempts to avoidsituations where this problem might arise:

Exchange of initiative in the tutorial involves complex interactions with the knowledge models anddiscourse procedures, given that student interruptions can disrupt a plan at any time. We minimizethese interactions in GUIDON by taking the initiative for just short sequences; long lectures orexplanations are more prone to student interruptions that overturn the teacher's plans. When it makes adifference, as during hypothesis evaluation, the program preserves the current dialogue context, andhence the logic behind its initiative, by not giving new case data during a student interruption. In amore flexible tutor, modeling and tutoring would occur in tandem. Specifically, in presentations of anyconsiderable length, like hypothesis evaluation, it is desirable for the tutor to periodically stop, listen,and reevaluate its present course. ( (3], pages 89 - 90)

It is precisely this ability to replan dynamically that would allow dynamic instructional planning toprovide greater flexibility in handling student interrupts than that provided by GUIDON.

3. The Blackboard ArchitectureLThis section considers the blackboard architecture: its architectural formalization, two examples of

planners implemented in that architecture, and then its support for dynamic instructional planning.

*These pre-stored discourse plans are .,presene in GUIDON as discourse proedures. In MENO-TUMOR they atsequences of states connected by default transitions.

pI

I0I10

3.1. Architectural FormalizationThe distinguishing features of a blackboard architecture are:1. Hierarchically structured global database - Solution elements to the problem being solved

are posted here. The hierarchical structure of the blackboard facilitates problemabstraction and the efficient triggering of knowledge sources.

2. Independent knowledge sources - These are rules triggered by changes to the blackboardwhose actions contribute to an evolving solution by adding or modifying solution elementson the blackboard. Knowledge sources communicate only by adding or changing thecontents of the blackboard.

3. Agenda control - An agenda of possible actions to perform is maintained. Activationrecords created by the triggering of knowledge sources are added to the agenda. Ascheduler selects the next action to execute from the agenda.

Figure 3-1 depicts the standard blackboard interpretation cycle. Knowledge sources are triggered bychanges to the blackboard. This causes new activation records to be added to the agenda, indicatingpossible actions corresponding to the knowledge sources. The scheduler selects the next action toperform. Its execution results in additional changes to the blackboard, possibly triggering newknowledge sources.

Execution Activation Recovrd chdl Ato

Figure 3-1: Blackboard Interpretation Cycle

3.2. Two ExemplarsThe BLACKBOARD INSTRUCTIONAL PLANNER [61 (abbreviated here as BB-IP) best

illustrates the direct application of a blackboard architecture to construct a dynamic instructionalplanner. Conceptually, the problem being solved - a lesson plan5 being incrementally constructed - isrepresented on a domain blackboard. A second blackboard, called the control blackboard, representsdecisions about how to construct and modify this plan. The lesson plan itself represents decisions aboutwhat instructional actions to take now and in the future. These blackboards are shown in Figure 3-2.There are three levels in the domain blackboard. The highest level represents a partial ordering oflesson objectives, concepts, or skills to teach. The second level represents tutorial discourse plans.Each plan consists of one or more steps that are in turn represented on the third level. Each step

VPan of the lesson plan is a current discoure plan.

influences the scheduler to select instructional actions from one particular category of instructionalactions. For example, an instructional plan could first bias the scheduler to select instructional actions to

motivate a topic, then actions to provide an overview of the topic, then actions to present the topic indetail, and finally actions to assess the student's understanding of the actions. There are several types of

actions to present or assess a topic, including presentation of graphics or specific kinds of tests such as atrue-false test. The categories of instructional actions are shown in Figure 3-3. Plan steps can direct the

scheduler to favor actions from any one of these categories. The next step in a plan is selected once thecurrent step's goal is satisfied. The goal might be that all such actions on the agenda have been

executed or that the tutor's belief that the student is likely to know a lesson objective has exceeded some

threshold. The control blackboard sets up default preferences to favor particular kinds of tutorialdiscourse plans (e.g., those that always assess a student's knowledge of a topic before discussing it).

These initial preferences are altered during the tutorial session by the action of knowledge sources thatmonitor time remaining, the efficiency of instructional plans, student questions, and student requests.

Lee~n Knowedg SOwee

Objectives Events

Figure 3-2: Domain and Control Blackboards of the BLACKBOARD INSTRUCTIONAL PLANNERKnowldge n

,F..1u ToJ sA~L..1sCOsul JI .A .1 tEi~41 JC TIIM..A.Ms. .t.a~ To"-STMI1GY -" -eELBNSTS40 =T- "

ITIE

11AM(11 noin

LACT C

Figure 3.3: Categories of instructional actions in the BLACKBOARD INSTRUCTIONAL PLANNER

IDE-INTERPRETER [81 is a second example of an instructional planner implemented in a +

blackboard architecture. The planner is not built over a general-purpose blackboard system, as BB-IP is

LA

12

built over BB 1, but the planner's architecture contains the three key components of the blackboardarchitecture. There is an explicit representation of the lesson plan at various levels of abstraction using atree data structure (first key component). Changes to the tree trigger rules that correspond to theknowledge sources of the blackboard architecture (second key component). The actions of theseknowledge sources are to further refine parts of the lesson plan. Each rule is encapsulated in a record asa possible task to be performed on an agenda (third key component - agenda control). A scheduler,consisting of a collection of heuristics, selects the next goal expansion task. Terminal nodes in the treecorrespond to instructional units that are directly executable self-contained instructional procedures.Typically these are associated with the presentation or assessment of a particular topic. Thus thisplanner emphasizes lesson planning over discourse planning since most specific discourse actions areprocedurally encoded in the instructional units and not reasoned about by IDE-INTERPRETER. BB-IPdiffers in its equal emphasis on both lesson and discourse planning, as illustrated by its reasoning aboutmore fine-grained discourse actions along with actions that refine or sequence lesson objectives.

3.3. Support for Dynamic Instructional PlanningEach of the three key architectural features of the blackboard architecture supports dynamic

instructional planning. The hierarchically structured global database supports an explicit hierarchicallystructured plan representation. This in turn supports the following functionality that facilitates dynamicinstructional planning:

" Planning at multiple levels of abstraction - this plan abstraction reduces the search space foran effective instructional plan and facilitates the separation of those aspects of theinstructional plan related to curriculum planning, lesson planning, and discourse planning.

*Incremental planning - the explicit plan representation allows the representation of those

parts of the plan that have not yet been refined and any constraints on partially refinedplans. This allows the planner to avoid premature commitment while retaining constraintsthat ensure plan consistency.

" Plan repair and plan improvement - the current plan need not be discarded entirely if somepart of it fails, instead it can be patched to retain the remainder of the plan. If the plan is notfailing but an unexpected opportunity to improve it arises - perhaps a student questionleading into planned instruction - then the plan can also be revised accordingly. Withoutthe ability to modify only parts of the plan either the entire plan must be discarded andanother selected, or the curriculum designer must attempt to anticipate all possible planfailures or opportunities for improvement - and again the combinatorics argue against thepracticality of this approach.

* Plan justifications - justifications for plan steps can be recorded and used to explain to thestudent, or to an instructional designer debugging a tutorial strategy, the reasons for thecurrent instructional actions. These plan step justifications can also be used in replanning.Although plan justifications can be used for explanation in systems that only select but donot assemble, patch, or revise plans, such as GUIDON, those systems cannot use thejustifications to replan.

The second key component of the blackboard architecture is the use of independent knowledgesources. This architectural feature supports dynamic planning by separating diverse kinds of knowledge

13 I

while allowing their cooperation during the problem solving process, in this case the construction of aninstructional plan. The following kinds of knowledge are easily separated with this scheme:

e Possible instructional actions - new instructional actions, such as giving a new kind of testor explanation, can be added (or actions can be deleted) without requiring modification ofexisting tutorial strategies.

e Tutorial strategies - new tutorial strategies can be added without modification of therepresentation of instructional actions.

* Planning strategies - different approaches to planning can be added without affecting thelibrary of tutorial strategies or instructional actions.

Although systems based on discourse management networks separate out the fust two kinds ofknowledge, and thus improve on CAI where both kinds of knowledge are procedurally encoded, they donot separate out the third kind of knowledge. The second kind of knowledge can in turn be separatedinto different methods for curriculum planning, lesson planning (topic/skill selection and sequencing),strategic discourse planning (for preplanned presentation and assessement of topics), and reactivediscourse planning (fo" handling student questions and requests).

Another disadvantage of DMNs is the lack of independence of knowledge sources when used inthat architecture. With the extensions that Clancey has made to the basic DMN architecture, the t-rulesand discourse procedures act as knowledge sources that select discourse actions, select future DMNstates, and save values in registers. Such knowledge sources are not independent since they mustcommunicate through registers, and there must be protocols for which knowledge sources set whichregisters and which read them. For example, some of GUIDON's t-rules set registers indicating the

value of discussing particular MYCIN domain rules. Other t-rules responsible for presenting the rules

must access these values. Clancey notes that:

Interactions ;f this kind are rare in GUIDON, but they illustrate the general problem of control whenthe decision about what to do is separated from its execution. The program must maintain a record ofpossible and/or intended actions, so that after all possibilities have been weighed, the program canreturn to the best plan and follow it. ... For the moment, we must hand-craft interactions between t-rulepackets. ( [31, page 80)

The blackboard architecture facilitates solutions to these problems: the agenda maintains a record ofpossible actions, the global database records an explicit plan representation of intended actions, andknowledge sources are independent and communicate entirely through the blackboard.

The third key component of the blackboard architecture, agenda control, supports dynamicinstructional planning by allowing the best instructional action (either an interactive action involving thestudent or further planning by the tutor) to be selected at any time with respect to possibly conflictingmultiple goals. Agenda control supports global planning by allowing multiple goals to be consideredsimultaneously. It also allows actions to be favored that achieve more than one goal at a time. Althoughsimilar results can be obtained in theory by procedurally encoding heuristics that take into account allcurrent goals and available actions, in practice this flexibility is difficult to obtain for complex domains.This difficulty arises from the unmanageable number of different possible sets of goals and actions thatmust be anticipated.

14

4. Scenario: Controlling a Programming Language TutorTo illustrate the additional support that the blackboard architecture provides for dynamic

instructional planning relative to discourse management networks, we will consider a hypothetical

scenerio. This scenario provides examples of how dynamic instructional planning in a blackboard

architecture provides more flexible and effective instruction than a system based on discourse

management networks, that cannot so plan. We consider the operation of the tutor first when controlled

by a dynamic instructional planner implemented in a blackboard architecture and then by a discourse

management network. To consider a familiar domain, the scenario assumes that the tutor is teaching

LISP in a self-paced course. The student is called S and the tutor T below.

4.1. Scenario 1: Dynamic Instructional Planning in the Blackboard Architecture

T greets S and explains that the first order of business is to draw up a curriculum plan for the term

based on the student's interests and the requirements set by the computer science department. T

provides S with a brief questionnaire about his background and interests. S knows Pascal and is

interested in learning LISP for use in an Al course he is taking concurrently. He is planning on doing a

term project in that class in LISP, and is less interested in the theoretical underpinnings of LISP (e.g.,the lambda calculus) than in using LISP for Al programming applications. T draws up a lesson plan

given the inferred cognitive stereotype of the student (novice with only one Pascal course), the number

of lessons expected in the course (20 lessons, each lasting about an hour to an hour and a half), and the

student's interests. A different curriculum plan is drawn up for each student. Another student in a

programming languages course might receive a much smaller number of lessons providing only a brief

introduction to LISP and concentrating on those unique features that distinguish it from other languages.

A typical lesson between T and S might appear as follows. T briefly reviews material covered in

the last lesson and then explains today's lesson plan, which covers mapping functions and iteration. T

proceeds to explain the concepts involved, relating them to analogous concepts in Pascal. T points out

both the differences and similarities between iteration in Pascal and in LISP. T proceeds to explain DOloops, but is interrupted by a request by S to explore. (The exploration mode of the tutor allows S to

perform experiments of his own in the LISP environment.) T suspends its discourse plan to elaboratefurther on DO and lets S interact with the interpreter and editor. T monitors S during this time. S writes

a simple DO to print out integers and their squares. Next S writes another DO but this one loops forever

since the exit test never becomes true. T's program analyzer, which has been checking S's DO loops for

well-formedness, detects this error. T breaks the infinite loop and explains the problem. The student

edits his DO, retries it, is satisfied, tries some other examples, and finally exits the exploratory mode. T

has also monitored the time involved and would have reacquired the initiative if too much time had been - q

spent exploring or too little progress appeared to result from the exploration. When T reacquires the

initiative, it is satisfied that S has learned those aspects of DO that it was about to explain. Both the

further explanation of DO and an assessment of the student's ability to use DO are now obviated by the

student's demonstrated capability during the exploratory session. T replans to omit these parts of the

lesson and then proceeds to discuss mapping constructs.

15

We pick up the scenario a few lessons later. Before moving onto more advanced topics, such asobject-oriented programming in LISP, T switches to a case-method style of instruction for this particularlesson. Each programming exercise tests some of the skills that S should have acquired and retained. Shas only minor problems with the first two exercises but his solution of the third exercise has multiplebugs. (T uses a program analyzer such as TALUS [5) that can detect multiple bugs in student programs;a PROUST-like [41 analyzer is used when T teaches Pascal.) T plans which bugs to address and in whatorder. It switches to a Socratic question and answer discourse strategy to track down and remediatelikely misconceptions about iteration indicated by the most important program bugs. T focuses on thosebugs related to iteration, and quickly explains and repairs the more simple remaining bugs for thestudent. (T could let S repair the bugs but decides that the time is better spent addressing underlyingmisconceptions.) T switches to expository instruction to review those aspects of iteration the studentappeared to have trouble with and then resumes case-method instruction to assess the student'scapabilities in other areas covered.

At the end of this assessment and review of basic skills, T explains the student's progress so farand what remains to be learned. The remaining curriculum is reviewed and the student is reminded thatit can be revised (within limits) to accommodate his specific interests. The student expresses an interestin covering one of the optional topics (pattern matching in LISP) since he has decided to build a theoremprover for his term project in the AI class he is taking concurrently. Since T cannot delete or compressany of the remaining lessons because they cover material mandated by the computer science depamnentit asks S if it is OK to add a short lesson on this. S agrees, so T inserts into the curriculum plan a lessonon using LISP to write simple pattern matchers. S is now satisfied with the overall curriculum plan, 4requesting only that more time be spent on working through examples in the future.

Still later in the lesson. T is explaining object-oriented programming by focussing on anapplication that is used as a running example throughout the discussion. Much less expositoryinstruction is given since more time is spent motivating concepts through examples, and giving thestudent exercises to solve with the tutor's assistance. At times, S switches to exploratory mode,examines the case library, ,nd requests that T solve one of the cases.6 When this happens, T tutorsopportunistically - it explains new material that an tes in the case or material that it believes the studentis weak on. It also replans to omit any similar cases that it had intended to cover, unless it appears - _necessary based on its assessment of the student's understanding of the tutor's problem solving.

4.2. Scenario 2: Opportunistic Tutoring with a Discourse Management NetworkNow we reconsider the same scenario assuming that only a discourse management network

controls the tutor. In this case no individualized lesson planning can be done without recourse tomechanisms outside of the discourse management network such as an agenda or auxiliary planningmechanism. Alternatively the DMN could be extended so that some states correspond to lesson

6'the available library of cases that the student can select from changes to include only those that the student should also beable to solve at that tiLe.

16

planning decisions that result in the selection of a curriculum plan from a plan library. But then statesnow represent both lesson planning actions and discourse management actions, undercutting theperspicuity of the DMN. Even with this extension the curriculum planning is not nearly so customized

as the blackboard tutor provides. S cannot interact with the DMN tutor T' to formulate a customizedcurriculum that achieves his objectives while satisfying department requirements. T' is also less capableof customizing individual lesson plans than T. Although the student can change the current topic withT', he cannot interact with T' to draw up an intended sequence of topics or cases for a particular lesson.

Let us assume that the DMN tutor is used either as a problem solving monitor that tutorsopportunistically (like GUIDON) or as a Socratic question and answer tutor (like MENO-TUTOR). Ineither case the tutor is not the primary source of instruction [1 ]; instead it assumes that the student issomewhat familiar with the material already but his knowledge can be refined and is likely to havemisconceptions. Thus the tutor must be used with some auxiliary form of instruction such as aworkbook (like Anderson's LISP tutor [7]) or classroom instruction.

The DMN tutor is best suited to handle one particular class of tutorial strategy, either Socraticquestion and answer or another opportunistic approach.7 Its ability to handle expository instruction islimited and its inability to plan dialog hampers it once it leaves the stereotypical discourse situations forwhich DMNs are best suited. When S gives the DMN tutor a program with multiple bugs it cannot planwhich bugs to address, which to ignore, and what order (and how) to discuss the bugs since these are notlocal decisions (e.g., one bug may mask other more serious bugs or two similar bugs could be discussed " -

together as resulting from the same misconception). The DMN tutor also cannot plan a series of casesalong with the topics to discuss for each case to cover a particular set of topics. The ability to deferdiscussing one topic now since a subsequent case will better address the same topic requires foresightthat the DMN does not have.

The DMN tutor does not allow the student as much freedom to express initiative as the blackboard-based tutor. The student cannot interrupt the tutor's presentation with requests that may changeassumptions underlying the instructional plan currently being followed. Thus in the earlier exampleswhere the student interrupted the tutor and entered an exploratory mode, or requested a change in -tutorial strategy, these degrees of flexibility are not readily supported in the DMN tutor. As before, anyspecific example in the scenario could be handled by a special-purpose DMN but in general thecombinatorics prevent a single DMN from handling the possibility of these kinds of interruptions at anytime, for any tutorial situation. The blackboard-based tutor is better equipped for these interruptionssince it has an explicit modifiable plan representation and can apply knowledge about replanning at thetime the interruption occurs.

This discussion is not meant to imply that S cannot benefit from the DMN controlled tutor T', only

7Even though GUIDON can exhibit many different discourse strategies for presenting MYCIN roles or assessing thestudent's knowledge of these, its overall tutorial strategy is opportunitic.

. . . . .. . . . . . . . . . . . .. ..... .. . . . .. . .. .. . . .. . . .. .. .. .. . . .. . . . . .. . . . . .. .. . . . . .. . .

17

that T' is quite limited compared to the blackboard-based tutor T. Most previous work in intelligenttutoring systems adopts an opportunistic tutoring framework similar to that of T', which can be wellsupported by discourse management networks. Many successful tutors have been built in thisframework (e.g., GREATERP [7), WEST [1], WUSOR [91, and GUIDON [2])8 while tutors such as Tare still only hypothetical. The computational overhead in the blackboard architecture is alsoundeniably greater than that of the simpler discourse management network. However, the point of thesescenarios is to illustrate that we can still build much more powerful tutors, encompassing bothopportunistic and highly customized expository instruction, and that research in planners andarchitectures to support them, such as blackboards, is justifiable.

5. SummaryThis paper compares the blackboard architecture to discourse management networks and concludes

that the blackboard architecture better supports dynamic instructional planning. This paper also arguesthat dynamic instructional planning allows more flexible and effective instruction than tutors that haveno plan or only a pre-stored plan. To support these arguments the key architectural features of eacharchitecture were compared and discussed in terms of the support or limitations they provide fordynamic instructional planning. A hypothetical scenario illustrating control by the two architecturesalso points out differences in capabilities.

Since discourse management networks are not sufficient in themselves to support the full range ofbehavior desirable of intelligent tutoring systems, especially the ability to customize lesson plans, togenerate globally coherent instruction, to select the best action based on multiple conflicting goals, andto allow student initiative that may alter a tutor's current plan, research in blackboard-based dynamicinstructional planners is justified. It is also clear that such blackboard-based planners are not strictlynecessary to build opportunistic tutors that do not perform lesson or dialog planning and which limitstudent initiative. In these cases, DMNs are sufficient and sophisticated tutors, such as GUIDON, canand have been built with that architecture.

This paper has not addressed the planning knowledge or the kind of planning required to realize adynamic instructional planner such as the one described in the scenario above. The planning required isan instance of a larger class of problems characterized by an environment that is

" Incomplete - since the tutor does not know the student's knowledge completely at any point,

" Uncertain - since what it does know it does not know with complete certainty,

" Dynamic - since the student's knowledge changes during and between tutorial sessions,

" Multi-agent - since the tutor and student cooperate to facilitate the student's learning,and where

Thm tam all adopt an oppmustic tutoring strategy, even though WEST and WUSOR act a coache fr a shtwho is playing a gune while GUIDON and GREATERP ait audeaf in problem solving for pflicula cues.

...... ....- Ymum I/ mmwlh mm Illtwm mmm m mmtml lra mam tmlk m m-I

Is

Results of actions are uncertain - since the tutor cannot predict with certainty the results ofits actions.

To specify a blackboard architecture as preferable to one based on discourse management networks tohandle these kind of planning difficulties does not provide an ITS designer with a solution to the controlproblem, although it is a step in this direction. Any kind of planner can be implemented in theblackboard architecture. Exactly what planning techniques are best in this domain, how to implement apractical dynamic instructional planner in the blackboard architecture, and the knowledge required forsuch planners are some of the difficult issues that remain to be addressed in future research.

Acknowledgements

I would like to thank Perry Thorndyke and Lee Brownston for reviewing drafts of this paper. This

research has been funded under contract N00014-86-C-0487, by the Office of Chief of Naval Research,Office of Naval Research, Naval Training Systems Center, and the Air Force Human ResourcesLaboratory.

19

References

[I] R. R. Burton, and J. S. Brown.An Investigation of Computer Coaching for Informal Learning Activities.International Journal of Man-Machine Studies (11):5 - 24, 1979.

[2] W. Clancey.Tutoring Rules for guiding a case method dialogue.International Journal of Man-Machine Studies (11):25 - 49, 1979.

[3] W. Clancey.Knowledge-based Tutoring.The MIT Press, 1987.

[4] W. L. Johnson and E. Soloway.Intention-based Diagnosis of Programming Errors.In Proceedings of the National Conference on Artificial Intelligence, pages 162 - 168. American

Association for Artificial Intelligence, August, 1 964.

[5] W.R. Murray.Automatic Program Debugging for Intelligent Tutoring Systems.Computational Intelligence 3(1):1 - 16, 19917.

[6] W. R. Murray.Dynamic Instructional Planning in the BBI Blackboard Architecture.Technical Report R-6168, FMC Corporation, August, 1988.

[7] B. Reiser, J. Anderson, and R. Farrell.Dynamic Student Modelling in an Intelligent Tutor for Lisp Programming.In Proceedings of the Ninth International Joint Conference on Artificial Intelligence, pages 8 -

14. 1985.

[8] D. M. Russell.The Instructional Design Environment: The Interpreter.Intelligent Tutoring Systems: Lessons Learned.Lawrence Erlbaum Associates, Inc., 1987.

[9] J. C. Stansfield, B. Carr, and I. P. Goldstein.Wumpus Advisor I: a First Implementation of a Program that Tutors Logical and Probabilistic

Reasoning Skills.Technical Report AI Lab Memo 381, Massachusetts Institute of Technology, 1976.

[10] S. Vere.Planning.Encyclopedia of Artficial Intelligence.John Wiley and Sons, Inc., 1987.

[11] E. Wenger.Artificial Intelligence and Tutoring Systems.Morgan Kaufmann, 1987.

[12] B. P. Woolf, and D. D. McDonald.Building a Computer Tutor: Design Issues.IEEE Computer 17(9):61 - 73, 1984.

2D-

(13] B. P. Woolf.Representing Comiplex Knowledge in an Intelligent Machine Tutor.Comtputational Intelligence 3(1):45 - 55, 1987. .

ONR Distribution

Personnel Analysis Division, Dr. Patrica BaggettAF/MPXA School of Education5C360, The Pentagon 610 E. University, Rm 1302DWashington, DC 20330 University of Michigan

Ann Arbor, MI 48109-1259Air Force HumanResources Lab Dr. Eva L. BakerAFHRL/MPD UCLA Center for the StudyBrooks, AFB, TX 78235 of Evaluation

145 Moore HallAFOSR, University of CaliforniaLife Sciences Directorate Los Angeles, CA 90024Boiling Air Force BaseWashington, DC 20332 Dr. Meryl S. Baker

Navy Personnel R&D CenterTechnical Director, ARI San Diego, CA 92152-68005001 Eisenhower AvenueAlexandria, VA 22333 Prof. Dott. Bruno G. Bark

Unlta di ri rc dl -..!Technical Director, Intelligenza Artifidale AArmy Human Engineering Lab Universita dl MilanoATTN: SLCHE-D 20122 Mlano -via F. Sforza 23Aberdeen Proving Ground ITALYMD 21005-5001

Dr. Gautam BiswasDr. Beth Adelson Department of Computer ScienceDepartment of Computer Science Box 1688, Station BTufts University Vanderbilt UniversityMedford, MA 02155 Nashville, TN 37235

Dr. Robert Ahlers Dr. John BlackCode N711 Teachers College, Box 8Human Factors Laboratory Columbia UniversityNaval Training Systems Center 525 West 120th StreetOrlando, FL 32813 New York, NY 10027

Technical Director Dr. Deborah A. Boehm-DavisAir Force Human Resources Lab. Department of PsychologyBrooks AFB, TX 78236-5601 George Mason University

4400 University DriveDr. John R. Anderson Fairfax, VA 22030Depatment of PsychologyCanegie-Melon University Dr. Jeff BonarSchenley Park Leaming R&D CenterPittsburgh, PA 15213 University of Pittsburgh 4

Dr. Stephen J. Andriole, Chairman P PA 15260

Department of Information Systems Dr. David Bowersand Systems Engineering Rensls Ukert AssociatesGeorge Mason University 3001 S. State Street4400 University Drive Ann Arbor, MI 48104-7352Fairfax, VA 22030

. . .. . ...I .

ONR Distribution

Dr. Robert Breaux Center for PersonnelCode 7B Security ResearchNaval Training Systems Center Suite E, Bldg. 455Orlando, FL 32813-7100 99 Pacific Street

Monterey, CA 93940-2481Dr. John S. BrownXEROX Palo Alto Research Professor Chu lien-ChenCenter Mathematics Department3333 Coyote Road National Tawan UniversityPalo Alto, CA 94304 Taipei, TAIWAN

Dr. John T. Bruer Dr. Michelene ChiJames S. McDonnell Foundation Learning R & D Center1034 So. Brentwood Blvd., Ste. 1610 University of PittsburghSt. Louis, MO 63117 3939 O'Hara StreetDr. Bruce Buchanan Pittsburgh, PA 15260

Computer Science Department Dr. Raymond E. ChristalStanford University UES LAMP Science AdvisorStanford, CA 94305 AFHRL/MOEL

Brooks AFB, TX 78235Lt. Col. Hugh BumsAFHRLIDI Dr. Willam ClanceyBrooks AFB, TX 78235 Institute for Research

on LearningAssistant for Long Range Rqmts. 3333 Coyote Hll RoadCNO Executive Panel (Op-OOK) Palo Alto, CA 943044401 Ford AvenueAlexandria, VA 22302-0268 Dr. Allan M. Collins

Bolt Beranek & Newman, Inc.Dr. Joanne Capper, Director 10 Moulton StreetCenter for Research into Practice Cambridge, MA 022381718 Connecticut Ave., N.W.Washington, DC 20009 Dr. Stanley Colyer

Office of Naval TechnologyDr. Pat Carpenter Code 222Camegle-Mellon University 800 N. Quincy StrvetDepartment of Psychology Adington, VA 22217-5000Pittsburgh, PA 15213

Dr. Albert T. CorbetDr. John M. Carroll Department of PsychologyIBM Watson Research Center Carnegie-Mellon UniversityUser Interlace Institute Pittsburgh, PA 15213P.O. Box 704Yorktown Heights, NY 10598 CAPT P. Michael Curran

Chief of Naval OperationsDirector, Manpower Program OP-939Center for Naval Analyses Pentagon4401 Ford Avenue Washington, DC 20350-2000P.O. Box 16268Alexandria, VA 22302-0268

10.6 2

ONR Distribution2

Brian Dalman Dr. Andrea di SessaTraining Technology Branch University of California3400 TCHTW/ITTGXC School of EducationLowry AFB, CO 80230-5000 Tolman HallBerkeley, CA 94720

Dr. Robert B. Davis

Curriculum Laboratory ERIC Facility-Acquisitions(Education) 4350 East-West Hwy, Suite 1100University of Ilinois Bethesda, MD 20814-4475Urbana, IL 61801

Dr. Martha EvansChief, Survey & Market Dept. of Computer ScienceAnalysis Division Illinois Institute of TechnologyDefense Manpower Data Center Chicago, IL 606161600 Wilson Blvd., #400Arlington, VA 22209-2593 Dr. Marshall J. Farr, Consultant

Cognitive & Intructiona SciencesDefense Technical Info. Ctr. 2520 North Vernon StreetAttn: TC Arlington, VA 22207Cameron Station, Bldg. 5Alexandria, VA 22314 Dr. Paul Feltovich(12 copies) Southern llinols Univ.

School of MedicineDeputy Dir. Military Personnel P.O. Box 3926Policy Divislon Springfield, IL 62708Office of the DCNO (MPT)(Op-i 3B) Dr. Gerhard FischerDepartment of the Navy University of ColoradoWashington, DC 20370-2000 Department of Computer Science

Deputy Director Total Force Boulder, CO 80309

Training & Education Division Dr. Kenneth D. ForbusOffice of the DCNO (MPT) University of llnois(Op-i l B) Department of Computer ScienceDepartment of the Navy 1304 West Springfield AvenueWashington, DC 20370-2000 Urbana, IL 61601

Head, Leadership Branch Dr. Barbara A. FoxNaval Miitary Personnel Command University of ColoradoAttn: LCDR E. Marits, NMPC-621 Department of UnguisticsDepartmot of the Navy Boulder, CO 80309Washritlon, DC 20370-5620

Dr. John R. FrederdksenHed, MINtery Compensation BBN LaboratoriesPolicy Branch 10 Moulton StreetOffice of the DCNO (MPT) Cambridge, MA 02238(Op-134)Department of the Navy Dr. Michael FriendlyWashington, DC 20370-2000 Psychology Department

York UniverftToointo ONTCANADA M3J 1 P3

1046 3

ONR Distribution

Julie A. Gadsden Dr. Jordan GrafmanInformation Technology Neuropsychology SectionApplications Division Medical NeurologyAdmiralty Rest 'h Establishment Branch-NINCDSPortsdown, Por ,nouth P06 4AA Building 10, Room 5C416UNITED KINGDOM Bethesda, MD 20892

Eric Gaussens Dr. James G. GreenoResearch & Development Dept. School of EducationFramentec S.A. Stanford UniversityTour Flat Cedex 16 Room 311Paris la Defense Stanford, CA 94305F. 92084France Dr. Dik Gregory

Admiralty ResearchDr. Dedre Gentner EstabshmentVAXBUniversity of Illinols Queens RoadDepartment of Psychology Teddington603 E. Daniel St. Middlesex, ENGLAND TWII OLNChampaign, IL 61820

Dr. Henry M. HlffDr. Meg Gerrard Halff Resources, Inc.Psychology Department 4918 33rd Road, NorthIowa State University Arlngton, VA 22207Ames, IA 50010

Dr. Bruce W. HamillDr. Robert Glaser Research CenterLearning Research & The Johns Hopkins UniversityDevelopment Center Applied Physics LaboratoryUniversity of Pittsburgh Johns Hopkins Road3939 O'Hara Street Laurel, MD 20707Pittsburgh, PA 15260

Dr. Barbara Hayes-RothDr. Sam Glucksberg Knowledge Systems LaboratoryDepartment of Psychology Stanford UniversityPrinceton University 701 Welch RoadPrinceton, NJ 08540 Palo Alto, CA 94304

Prof. Clark Glymour Dr. James HolanDept. of Philosophy NPRDC, UCSDCamegWMelon University Code 501Pittsburgh, PA 15213 San Diego, CA 92152

Dr. Shere Gott Dr. Keith HolyoakAFHRL/MOMJ Department of PsychologyBrooks AFB, TX 78235-5601 University Of California

Los Angeles, CA 90024Dr. T. GovndarajGeorgia Institute of Ms. Julia S. HoughTechnology Lawrence Erlbam AssociatesSchool of Industrial 110W. Harvey Streetand Systems Engineering Phladelphia, PA 19144Atlanta, GA 30332-0205

10-8 4

ONR Distribution

Dr. Ed Hutchins Dr. Jeffery L. KenningtonIntelligent System Group School of Engineering &Institute for Applied SciencesCognitive Science (C-015) Southern Methodist UniversityUCSD Dallas, TX 75275La Jolla, CA 92093

Dr. David KlerasDr. Janet Jackson Technical Communication ProgramRijksuniversitet Groningen TIDAL Bldg., 2360 Bonisteel Blvd.BWofoich Centrum, Vleugel D University of MichiganKerklaan 30, 9751 NN Haren Ann Arbor, MI 48109-2108The NETHERLANDS

Dr. Janet L KolodnerDr. Claude Janveb Georgia Institute of TechnologyUniversite' du Quebec a Montreal School of Information &P.O. Box 8888, succ: A" Computer ScienceMontreal, Quebec H3C 3P8 Atlanta, GA 30332CANADADr. Dr. Kenneth Kotovsky

Community College ofHewlett-Packard Laboratories, 3L Allegheny CountyP.O. Box 10490 808 Ridge AvenuePalo Alto, CA 94303-0971 Pittsburgh, PA 15212

Prof. David W. Johnson Dr. Benjamin KulpersCooperative LeOrg Center U1ivOMty of Texas at AustinUniversity of Minnesota Department of Computer Sciences150 Pillsbury Dr., S.E. Taylor Hall 2.124Minneapolis, MN 55455 Austin, Texas 78712

Dr. Marcel Just Dr. Jill LatdnCarnegie-Mellon University Carnegie-Mellon UniversityDepartment of Psychology Department of PsychologySchenley Park Pittsburgh, PA 15213Pittsburgh, PA 15213

Commander J.M. LaRoocoDr. Daniel Kahneman Naval School of Hesith SciencesDepartment of Psychology National Naval Medical CenterUniversity of Calfornia Bldg. 141Berkeley, CA 94720 Washington, DC 20814-5033

Dr. Milton S. Katz Dr. Robert W. LawEuropean Science Coorinatlon Matthews 118Office Purdue UniversityU.S. Army Research institute West Lafayette, IN 47907 ABox 65FPO New York 09510-1500 Dr. Alan M. Lesgold

Learning R&D CenterDr. Wendy Kellogg University of PttbxghIBM T. J. Watson Research Ctr. Pittsburgh, PA 15260P.O. Box 704Yorktown Heights, NY 10598

10-66 5

ONR Distribution

Dr. Jim Levin Dr. Manton M. MatthewsDepartment of Department of Computer ScienceEducational Psychology University of South Carolina210 Education Building Columbia, SC 292081310 South Sixth StreetChampaign, IL 61820-6990 Dr. Joseph C. McLachlan

Code 52Dr. John Levine Navy Personnel R&D CenterLearning R&D Center San Diego, CA 92152-6800University of PittsburghPittsburgh, PA 15260 Dr. James McMichael

Technical DirectorDr. Clayton Lewis Navy Personnel R&D CenterUniversity of Colorado San Diego, CA 92152-6800Department of Computer ScienceCampus Box 430 Dr. Barbara MeansBoulder, CO 80309 SRI International

333 Ravenswood AvenueScience and Technology Division Menlo Park, CA 94025Library of CongressWashington, DC 20540 Prof. George A. Miler

Dept. of PsychologyVern M. Malec Princeton UniversityNPRDC, Code 52 Princeton, NJ 08544San Diego, CA 92152-6800

Dr. James R. MilkrDr. Jane Malin MCCMall Code EF5 3500 W. Balcones Center Dr.NASA Johnson Space Center Austin, TX 78759Houston, TX 77058

Mark L. MilerDr. William L. Maloy Advanced Technology GroupNaval Education and Training Apple Computer Inc.Program Support Activity 20525 Marlan Ave.Code 047 MS 220Building 2435 Cupertino, CA 95014Pensacola, FL 32509-5000

Dr. William MontagueDr. Elaine Marsh NPRDC Code 13Naval Cener for Applied Research San Diego, CA 92152-6800in ArMal IntegenceNaval Research Laboratory Dr. Judy MoraccoCode 5510 Code CEL-MP53Washington, DO 20375-5000 Washington Navy Yard

Bldg. 200Dr. Sandra P. Marshall Washington, DC 20374Dept. of PsychologySan Diego Sta University Dr. Randy MumawSan Diego, CA 92182 Training Research Division

HumRRO1100 S. WashingtonAlexandrl VA 22314

1066 6

ONR Distribution

IDr. Allen Munro UbraryBehavioral Technology NPRDCLaboratories - USC Code P201L1845 S. Elena Ave., 4th Floor San Diego, CA 92152-6800Redondo Beach, CA 90277

Spec. Asst. for Research, Exped-Deputy Technical Director mental & Academic Programs,NPRDC Code 01A NTC (Code 016)San Diego, CA 92152-6800 NAS Memphis (75)

Mlillngton, TN 38054Director, Human Factors& Organizational Systems Lab, Assistant Chief of StaffNPRDC (Code 07) for Research, DevelopmenLSan Diego, CA 92152-6800 Test, and Evaluation

Naval Education andDirector, Manpower and Personnel Training Command (N-5)Laboratory, NAS Pensacola, FL 32506NPRDC (Code 06)San Diego, CA 92152-6800 Director, Instructional Dvlpmt. -

and Educational Pgm. Support Dept.Director, Training Laboratory, Naval Education & Training Pgm.NPRDC (Code 05) Management Support ActivitySan Diego, CA 92152-6800 (NETPMSA)

Pensacola, FL 32509Head, Fleet Uaison Dept.NPROC (Code 31) Technical DirectorSan Diego, CA 92152-6800 Naval Health Research Center

P.O. Box 85122Head, Human Factors Dept. San Diego, CA 92138-9174NPRDC (Code 41)San Diego, CA 92152-6800 Director, Recreational Svcs. Dept.

Naval Military Personnel CommandHead, Manpower Systems Dept. (N-651 C)NPRDC (Code 61) 1300 Wilson Blvd., Room 932San Diego, CA 92152-6800 Arlington, VA 22209

Head, Personnel Systems Dept. Chairman, Department ofNPRDC (Code 62) Administrative SciencesSan Diego, CA 92152-6800 Code 54

Naval Postgraduate SchoolHead, Teting Systems Dept. Monterey, CA 93943-5100NPRDC (Code 63)San Diego, CA 92152-6800 Chairman, Department of

Operations ResearchHead, Training Systems Dept. Code 55NPRDC (Code 52) Naval Postgraduate SchoolSan Diego, CA 92152-6800 Monterey, CA 93943-5100

Head, Training Tech. Dept.NPRDC (Code 51)San Diego, CA 92152-6800 "

1046 7

ONR Distribution

Commanding Officer, Director, BlologlcallHumanNaval Research Laboratory Factors Division (Code 125)Code 2627 Office of the Chief ofWashington, DC 20390 Naval Research*Arlington, VA 22217-5000j

Deputy Director Manpower, _A

Personnel and Training Div. Director, Navy Family Support PgmNaval Sea Systems Command Office of the DCNO (MPT) (Op-156)Atn: Code CEL-MP63 1300 Wilson Blvd., Room 828Washington, DC 20362 Arlington, VA 22209

Head, Human Factors Laboratory Office of Naval Research,Naval Training Systems Ctr. Code 1142Code 71 800 N. Quincy St.Orlando, FL 32813-7100 Arlington, VA 22217-5000

Ubrary Office of Naval Research,Naval Training Systems Center Code 1142B1Orlando, FL 32813 800 N. Quincy Street

Arlington, VA 22217-5000UbraryNaval War College Office of Naval Research,Newport, RI 02940 Code 1142CS

800 N. Quincy StreetCommanding Officer Arlington, VA 22217-5000Navy Personnel R&D Ctr. (6 Copies)San Diego, CA 92152-6800

Office of Naval Research,Technical Director Code 1142PSNavy Personnel R&D Center 800 N. Quincy StreetSan Diego, CA 92152-6800 Arlington, VA 22217-5000

Director, Research & Analysis Div. Biological intelligenceNavy Recruiting Command (Code 223) Code 1142BI4015 Wilson Blvd., Room 215 Office of Naval ResearchArlington, VA 22203-1991 Arlington, VA 22217-5000

Dr. Donald A. Norman (C-01 5) Cognitive ScienceInstitute for Cognitive Science Code1142CsUniversity of California Office of Naval ResearchLa Jolla, CA 92093 Adington, VA 22217-5000

Special Assistant for Marine Director, Cognitive &corps Matters, Neural Sciences (Code 1142)ONR Code 00MC Office of Naval Research800 N. Quincy St. Arlington, VA 22217-5000Arlington, VA 22217-5000

Director, Lfe ScinesChairman, MPT R&D Committee Code 114Office of the Chief of Office of Naval ReearchNaval Research Arlington, VA 22217-5000Code 222Arlington, VA 22217-5000

10488 8

ONR Distribution

Director Research Programs Dr. Douglas PearceOffice of Naval Research 1133 Sheppard WCode 11 Box 2000Arlington, VA 22217-5000 Downsvlew, Ontario

CANADA M3M 3B9MathematicsCodeI I I IMA Office of the Deputy AssistantOffice of Naval Research Secretary of the NavyArlington, VA 22217-5000 Manpower & Reserve Affairs

5D800, The PentagonPsychologist Washington, DC 20350-1000Office of Naval ResearchBranch Office, London Dr. Tjeerd PlompBox 39 Twente University of TechnologyFPO New York, NY 09510 Department of Education

P.O. Box 217Psychologist 7500 AE ENSCHEDEOffice of Naval Researct THE NETHERLANDSDetachment1030 E. Green Street Dr. Martha PoisonPasadena, CA 91106-2485 Department of Psychology

University of ColoradoDr. Harold F. O'Nell, Jr. Boulder, CO 80309-0345School of Education - WPH 801Department of Educational Dr. Steven E. PoltrockPsychology & Technology MCCUniversity of Southern Callfomia 3500 West Balcones Center Dr.Los Angeles, CA 90089-0031 Austin, TX 787594509

Dr. Judith Orasanu Dr. Harry E. PopleBasic Research Office University of PittsburghArmy Research Institute Decision Systems Laboratory5001 Eisenhower Avenue 1360 Scalfe HalAlexandria, VA 22333 Pittsburgh, PA 15261

Military Assistant for Training and Dr. Joseph PsotkaPersonnel Technology, ATTN: PERI-ICOUSD (R & E) Army Research InstituteRoom 3D129, The Pentagon 5001 Eisenhowor Ave.Washington, DC 20301-3060 Alexandria, VA 22333-5600

Assistant for Manpower and Training Mr. Paul S. RauOffice of the CNO (Op-987H) Code U-335D772, The Pentagon Naval Surface Weapons CenterWashlngton, DC 20350 White Oak Labors"a_1

Slver Spring, MD 20903Head, Manpower, Personnel,and Training Branch Dr. Steve RederOffice of the CNO (Op-81 3) Northwest Regional4A478, The Pentagon Educational LaboratoryWashington, DC 20350-1000 400 ndsy Bld Mg.

710 S.W. Second Ave.Portand, OR 97204

10_ 9

ONR Dlstrlbution

2

Dr. James A. Reggla Dr. Walter SchneiderUniversity of Maryland Learning R&D CenterSchool of Medicine University of PittsburghDepartment of Neurology 3939 O'Hara Street22 South Greene Street Pittsburgh, PA 15260Baltimore, MD 21201

Dr. Janet W. SchofieldDr. J. Wesley Reglan 816 LRDC BuildingAFHRI/IDI University of PittsburghBrooks AFB, TX 78235 3939 O'Hara Street

Pittsburgh, PA 15260Dr. Fred ReifPhysics Department Dr. Miriam SchustackUniversity of California Code 52Berkeley, CA 94720 Navy Personnel R & D Center

San Diego, CA 92152-6800Dr. Brian ReiserDepartment of Psychology Dr. Judith W. SegalGreen Hall OERIPrinceton University 555 New Jersey Ave., NWPrinceton, NJ 08540 Washington, DC 20208

Dr. Gilbert Ricard Dr. Colleen M. SeifertMail Stop K02-14 Instltute for Cognitive ScienceGrumman Aircraft Systems Mail Code C-015Bethpage, NY 11787 University of California, San Diego

La Jolla, CA 92093Dr. J. Jeffrey Richardson

Center for Applied Al Dr. Ben ShneldermanCollege of Business Dept. of Computer ScienceUniversity of Colorado University of MarylandBoulder, CO 80309-0419 College Park, MD 20742

Dr. Unda G. Roberts Dr. Randall ShumakerScience, Education, and Naval Research LaboratoryTransportation Program Code 5510Office of Technology Assessment 4555 Overlook Avenue, S.W.Congress of the United States Washington, DC 20375-5000Washington, DC 20510

Dr. Edward E. SmithDr. Willm B. Rouse Department of PsychologySearch Technology, Inc. Unverslty of Michigan4725 Peachtree Corners Circle 330 Packard RoadSulte 200 Ann Arbor, MI 48103Norcross, GA 30092

Program DirectorDr. Eduardo Salas Manpower Reearch &Human Factors Division Advisory ServicesCode 712 Smthsonian insitutionNaval Training Systems Cir. 801 N. Pt St., Suie 120Orlando, FL 32813-7100 Alexandda, VA 22314-1713

10-a6 10 . -

ONR Distribution q

Dr. Al Smode Dr. Douglas TowneNaval Training Systems Ctr. Behavioral Technology LabsCode 71 University of Southern CaliforniaOrlando, FL 32813-7100 1845 S. Elena Ave.

Redondo Beach, CA 90277Dr. Elliot SolowayYale University Headquarters, U. S. Marine CorpsComputer Science Department Code MPI-20P.O. Box 2158 Washington, DC 20380New Haven, CT 06520

Dr. Jerry VogtDr. Richard C. Sorensen Navy Personnel R&D CenterNavy Personnel R&D Center Code 51San Diego, CA 92152-6800 San Diego, CA 92152-6800

Dr. Kathryn T. Spoehr Dr. Ralph WachterBrown University JHU-APLDepartment of Psychology Johns Hopkins RoadProvidence, RI 02912 Laurel, MD 20707

Dr. Kurt Steuck Dr. Keith T. WescourtAFHRI/MOA FMC CorporationBrooks AFB Central Engineerng LabsSan Antonio, TX 78235-5601 1205 Coleman Ave., Box 580

Dr. Albert Stevens Santa Clar CA 950

Bolt Beranek & Newman, Inc. Dr. Barbara White10 Moulton St. BBN LaboratoriesCambridge, MA 02238 10 Moulton StreetDr. John Tangney Cambridge, MA 02238

AFOSR/NL, Bldg. 410 David C. WidnsBoiling AFB, DC 20332-6448 212 Digital Computer Lab

1304 W. Springfield Ave.Dr. Kikumi Tatsuoka Urbana, IL 61801CERL252 Engineering Research Dr. Kent E. WilamsLaboratory Inst for Simulation and Training -.103 S. Maltews Avenue Univesty of Central FloridaUrbana, L 61801 P.O. Box 25000

Dr. Pe" W. Orlando, FL 328160544

FMC Coqfparlon Dr. Robert A. WisherCentrl Engineering Labs U.S. Any Institute for the1205 Coleman Avenue, Box 580 Behavioral and Social Sciences - ISanta Clarm CA 95052 5001 Eisenhower Avenue

Alexandria VA 22333-5600Dr. Martin A. Tolcott3001 Veazey Terr., N.W. Dr. Martin F. WiskoffApt. 1617 Defense Manpower Data CenterWashington, DC 20008 550 Camino E Estero

Suite 200Monterey, CA 93943-3231

10- 11-.

ONR Distribution

Dr. Wallace Wulfeck, IIINavy Personnel R&D CenterCode 51San Diego, CA 92152-6800

Dr. Masoud YazdaiDept. of Computer ScienceUniversity of ExeterPrince of Wales RoadExeter EX44PTENGLAND

Dr. Joseph L. YoungNational Science FoundationA

1800 G Street, N.W.Washington, DC 20550

10-86 12