AD-Ai4i 263 TOMARDS A GENERAL SCIENTIFIC REASONING ENGINE(U) i/iCARNEGIE-MELLON UNIV PITTSBURGH PA DEPT OF PSYCHOLOGYJ G CARBONELL ET AL. 17 NOV 83 TR-84-i-ONR

I UNCLASSIFIED NOeei4-82-C-e?6? F/G 6/4 NL

IIEEEEEEEEEESlflflfll.lllJ

'.4 4. .. TIM-

.-

%1"

ig 1-2 Q

11SM

~- I1

MICROCOPY RESOLUTION TEST CHART

NATIONAL BUREAU OF STANDARDS- 1963-A

.-. g4o'

TOWARDS A GENERAL

SCIENTIFIC REASONING ENGINE

,,. Jaime G. Carbonell

Jill H. Larkin

Department of Psychology

Carnegie-Mellon UniversityPittsburgh, PA 15213

.I F. Reif

University of CaliforniaBerkeley, CA 94720

Approved for public release; distribution unlimited.Reproduction in whole or in part is permitted for any purpose

of the United States government

DTIO..ELECTED

_kMY 1 81984

L.NR No 667-494

_% *.

8 .3.

P..

-14r SECURITY CLASSIFICATION OF THIS PAGE (When Data Entered)REPORT DOCUMENTATION PAGE READ INSTRUCTIONSREPORT__ DOCUMENTATIONPAGE_ BEFORE COMPLETING FORM

I. REPOpT,14.ER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

% 4. TITLE (ad Subtitle) S. TYPE OF REPORT & PERIOD COVERED

Towards a General Scientific Reasoning Engine Technical Report

6. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(s) S. CONTRACT OR GRANT NUMBER(3)

Jaime G. CarbonellJill H. Larkin N00014-82-C-90767F. Reif

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENH. iRCJE--. TASKAREA & *ORK UNIT NUMBERS

Department of PsychologyCarnegie-Mellon University NR 567-494Pittsburgh, PA 15213

I. CONrROL..Nt; OFFICE NAME AND ADDRESS 2. REPORT OArE

Personnel and Training Research Programs 17 November 1983office of Naval Research " NUM8ERGF AG5

Arlington, VA 22217 8714. MONITORING AGENCY NAME & AODRESS(I dillerent from Controlling Offilce) IS. SECURITY CLASS. (of this report)

unclassified

3IS. DECLASSIFICATION/ DOWNGRADINGSCHEDULE

-I II. DISTRIBUTION STATEMENT (of this Report)

" Approved for public release; distribution unlimited

17. DISTRIBUTION STATEMENT (of the abetract entered In Block 20, It dillerent from Report)

II. SUPPLEMENTARY NOTES .

1. KEY WOROS (Continue on reverse aide I necoeear, and identify by block number)

problem-solving computex-simulationknowledge-engineering intelligent CAIexpert systems instructionphysics

20. ABSTRACT (Continue on ,ovorao aid* If necesooaa and Identify by block number)Expert reasoning in the natural sciences appears to make extensive use of arelatively small number of general-principles and reasoiung-strategies, eachassociated with a larger number of more specific inference patterns. Using adual declarative hierarchy to represent strategic and factual knowledge, weanalyze a framework for a robust scientific reasoning engine. It is argued thatsuch an engine could provide (a) the ability to reason from basic principles inthe absence of directly applicable specific information, (b) principled

DOI , 1473 EDITION OF I NOV ,s 1S OSSOLIT1 unclassifiedS/N 0102A014o6601

SECURITY CLASSIFICATION OF THIS PAGE (When Data 9#neered)

L..U4ITY CLASSIFICATION OF THIS PAGE(Whof Date Entered)

20. Abstract (cont'd)

owledge acquisition by using existing general patterns to structure newinformation, and (c) congenial explanation and instruction in terms of generaland familiar patterns of inference.

'S

*q'1

!-.

.

.o

'

I' *," ,,., .. .*.. -,,. .- ,. .,.-.., ..-.- , . . .. .",', .".' .- '-""' ,,,','.....-..'...,,S-%,"'.'..' -. -;' ..- , .'': ,'- ."'-'' -" - -- N'-.

Abstract

Expert reasoning in the natural sciences appears to make extensive use of a

relatively small number of general principles and reasoning strategies, each associated

with a larger number of more specific inference patterns. Using a dual declarative

hierarchy to represent strategic and factual knowledge, we analyze a framework for a

robust scientific reasoning engine. It is argued that such an engine could provide ) the

ability to reason from basic principles in the absence of directly applicable specific

information, A principiea Knowleage acquisiton by using existing general patterns to

structure new information, ana t4) congenial explanation and instruction in terms of

general and familiar patterns of inference.

Acoession For

.TI *GA&IDTIC TABiUntinn ounced El

S ByDistributton/

Availability CodesDs" "Avsil and/or

Dist Special

lJ

1 A 0 ', " & A" " "4 A" "

' . *sQ4 l-. .. ..p .. . .* a -. . . . ° . . . ..

Towards a General Scientific Reasoning Engine

Table of Contents1. Objectives 1

1.1. General Inference Patterns in Natural Science 11.2. Importance of General Inference Patterns to Al Models 1

1.2.1. Help for Common Expert-System Problems 11.2.2. Implications for Intelligent CAI 3

2. The Reasoning Model 32.1. Principles 32.2. Strategies 42.3. Knowledge Representation 5

3. Conclusion 74. References 8

4.:

IA.

.'-.

:p:r ...............

Towards a General Scientific Reasoning Engine

1. ObjectivesNatural scientists believe that there are general patterns of reasoning used repeatedly in a variety of

contexts. These repeated patterns occur both among scientific principles (e.g., conservation,

equilibrium), and among problem-solving strategies (e.g., decomposition, analyzing extremes). Wepresent an analysis of how to capitalize on this apparent redundancy to build a prototype of a

potentially general scientific reasoning engine.

1.1. General Inference Patterns in Natural Science

The natural sciences contain two inter-related hierarchies of knowledge (see Figure 1-1). First,scientific principles often contain Knowledge of varying generality. Conservation of .harge, nas3,

and energy all share the orinciole of conservation and the inference rules it entails. Similarly,

conservation of macroscopic kinetic energy (without dissapation) inherits many of the properties of

conservation of energy. Of course, the more specific principles also provide additional informationnot present in from their more general counterparts. Second, there is a similar hierarchy of problem-

solving strategies. Decomposition is a very general strategy, from which more specific strategies

(vector or Fourier decomposition) inherit knowledge. Third, there are in each hierarchy horizontallinks reflecting analogies among simi!ar concepts in each hierarchy. This enables latteral (partial)

inheritance [3, 7]. Finally there are links between the two ihierarchies in both directions. A reasoning

strategy may be based upon one or more principles, and, conversely, kncwing that a particularprinciple applies to a given problem situation may suggest one or more appripriate strategies. For

instance, noting that certain types of energy are conserved in some subsystems may suggest problem

decomposition along the conservatioh boundaries.

1.2. Importance of General Inference Patterns to Al Models

Explicitly encoding the general inference patterns of the natural sciences should help alleviate

some problems inherent to present expert system architectures, and should provide particularly good

support for intelligent computer assisted instruction (ICAI).

1.2.1. Help for Common Expert-System Problems

While encoding new information for a currently interesting task it is hard to recognize knowledge

that might be generally applicable in very different situations. Therefore general and specific, knowledge are often inextricably encoded into the same rules. Moreover, strategic knowledge is often

intermingled with factual knowledge, or worse yet, convolved with "certainty factors", "subjective

probabilities" or other numerical parameters. This lack of separation gives rise to commondeficiencies in the following areas:

t V

/e= , ,:.,; .. ,,,,.... ,.;..;.:, :. . . .. , . ,..., . . . . . ..- . . . , . . . . .

2 Carbonell, Larkin & Reif.4

Principles Strategies

EquilibriumEq

Conservation Decomposition

U~eS JAnalyze extremes

of charge

of mass of energy Vector

analogy

of macroscopic energy Fourier

Figu re 1 -1: Fragment of the dual hierarchy illustrating the relation betweenprinciples and the strategies.

I..

. Portability. It is hard to apply or mcdicy knowledge en'coded for use in one domain to anotherdomain. Shared general knowledge or similarities among related domains cannotbe exploited.

Robustness. It is hard to make an expert system behave gracefully and sensibly beyond theparticular limited domain for which it was designed. In contrast gracefuldegradaticn rather than catastrophic failure is the hallmark of human expertisewhen faced with problems that depart somewhat from the expected.

Extensibility. It can be extremely difficult to revise past knowledge distributed among a large setA of rules, rather than encapsulated at the mcst general appropriate level (and then

inherited by lower levels of the hierarchy).

Congeniality. The reasoning of expert systems .- linear traces of hundreds or thousands of rules-- can prove extremely frustrating to humans. Moreover, when decisions dependupon combinations of certainty values, the process becomes even more obscure.

In the natural sciences there are general inference patterns that can be stated independently of other

more specific knowledge. Encoding these patterns separately should help alleviate all of the

preceding problems. This general knowledge can be used in extending the system to other scientific

domains that rely on the same inference patterns. When taxed beyond its original limits, the system

.%

• -,i-'-i , ' . 'i . . . . ,. , , ..... . . . o , , pj-,_ _ _ -,.. -.. . . ,, ,- . .-. . -- . . -

- , -" . .-

. . -

Towards a General Scientific Reasoning Engine 3

will at least have this knowledge available, enabling it to ellicit reasonable if suboptimal behavior.

General patterns of inference, of the kind illustrated here, appear explicitly in human discussions of

natural science (e.g., in textbooks) and there is some evidence [8] that experts use them in solving

problems. Thus congeniality to humans should be improved by the explicit use of general inference

patterns.

1.2.2. Implications for Intelligent CAIThe use of general inference patterns has definite benefits for ICAI systems which must satisfy

particularly stringent demands for robustness, extensibility, and congeniality. Instructional systems

must be robust -- deviation from particular anticipated paths is the rule, not the exception for learners.

Instructional domains are always large (by Al standards), yet we must be able both to extend our

systems at reasonable cost and to use instructional material in one domain to design instruction in a

related domain. Lastly, the reasoning used by the expert system must be sufficiently congenial to

students that they can learn to use it themselves.

In addition, encoding scientific knowledge in terms of repeated general inference patterns offers the

opportunity for teaching diractly a kind of reasoning that is ordinarily left implicit. Whereas textbooks

and lecturers in physics and chemistry may use general inference patterns such as additivity or

reduction of dimensionality, they are unlikely to make explicit the repeated use of these pattorns in

very different areas of science. An ICAI system with knowledge based on these patterns could make

them explicit to students and illustrate their utility in solving different classes of problems. This

exp!icit teaching could be an important educational contribution because there is evidence [8] that

students may fail to solve science problems because they cannot reliably construct and use

qualitative relations between scientific quantities. Many such relations (e.g., energies are conserved,

forces balance) are included among the general patterns of inference.

2. The Reasoning Model

As outlined above, the reasoning model contains a linked dual hierarchy of factual or principle

knowledge and strategic or problem-solving knowledge. This section describes each of these kinds

of knowledge and then discusses their representation as linked declarative hierarchies.

2.1. Principles

The same patterns of inference are used repeatedly in different areas of natural science. For

example. a system for reasoning about fluids at rest must include knowledge like the following:

, Pressure drops are additive along any path. Thus to determine the pressure dropbetween any two points, one can add the pressure drops along any sequence of

S. ' ?''- '-..,? ?.'-.', -- '-

4 Carbonell, Larkin & Reif

* ~. . segments joining those points.

" '.** When a fluid is in equilibrium, the forces on any portion of fluid (and on objects in contactwith it) are balanced. The total force in one direction is always balanced by a force ofequal magnitude in the opposite direction.

* The magnitude F of a force due to a fluid on any small surface is proportional to the area*.-A of that surface. Thus we can define a proportionality constant, called pressure p, that is

equal to the ratio F/A of force divided by area, and that is independent of force and area.

If instead we were building a system to reason about chemical equilibrium, the system would need

knowledge like the following:

* Concentrations are additive in any suiution. Thus to determine the concentration oi anyspecies, one can add the concentrations due to any collection of sources for this species.

* When a reaction is in equilibrium, the reaction rates are balanced. The total reaction ratein one direction (e.g., producing a particular species) is always balanced by a reactionrate of equal magnitude in the opposite direction (destroying the species).

* The magnitude R of a reection rate of a species is proportional to the concentration c ofa! that species. Thus we can define a proportionality constant, called the reaction

coefficient k, that is equal to the rato Ric of reaction rate divided by concentration, and. that is independent of reaction rate and of the concentration of that species.

Additivity. bal-ince/equlibrium, and proportionality are examp!es of general inference patterns used

repeatedly in natural science. Other examples are conservation and fields (unique furictions of

'- points). Many scientific prir.ciples thus form a hierarchy with a considerable amount of knowledge

inherited from a relatively few general patterns of inference. By its very nature, an inheritance

hierarchy allows one to encode facts or strategies at their most appropriate level of abstraction.

2.2. Strategies

Scientific strategic knowledge is also hierarchical, centering around a relatively small number of

very general strategies, for example, the following:

=.:'.-"Reduce dimensionsReued..sIi an N-dimensional situation can be stated in N • K dimensions, reformulate in

4 . fewer dimensions. For example, calcuiating a parabolic trajectory oi a projectile

requires only the plane of the parabola. If all that matters is the maximum verticalheight, then the one-dimensional vertical component of the motion will suffice.

.. Find dependenciesConsider using any applicable principle that relates unknowns to knowns or(second best) establishes dependencies among one or more unknowns.

-?,Find analogies to solved problemsIf a similar problem has a known solution, try the same solution procedure on the

.".. .. A .- ...-

., ..,.:. ., :.:,. .... ..:...... . ., ,. , ......: .:.... -.?..... . . .' , .s... .......,..... ... ,......,,1 .. .,. .- .... . . -, . .-.,.,.,,

,' Towards a General Scientific Reasoning Engine 5

new problem (perhaps with minor changes to accommodate relevant differences)[5,41. For example, electrical circuitry was traditionally understood by analogy to

steady.state fluid dynamics in closed systems.

Decompose Many apparently difficult problems become tractable when much simple instancesof the problems are first solved and then composed into a full solution. For

- . - example, complex functions can often be decomposed into a sum or series ofsimple functions.

-.- Analyze extremesAnalyze simple, often limiting, instances of a problem. The solution proceduresfor these simple problems may serve as a basis for the final solution. This methodis essentially equivalent to Polya's "tame auxiliary problem" idea (14].

Perturbation analysisStart from one or more known or easily computable steady-state solutions. thendetermine how the behavior of the system changes in response to minorperturbations of its independent parameters.

Exploit symmetriesSymmetry relations provide useful constraints on the form a result may take. Forexample, spatial symmetry requires that forces between particles depend only onthe distance between them and not on the indivitual positions.

Reasoning strategies of the kind listed above are relatively few .in number. Hdwever, there ars

potentially very many specific strategies (like the examples used above) which are essentially useful

instantiations of the general strategies.

The idea and utility of a hierarchy of heuristics has been proposed by Lenat [9]. We add two new

organizational principles: (a) basing this hierarchy on the available and powerful hierarchy of

strategic methods in the natural sciences; (b) linking the strategic hierarchy to a hierarchy of factual

knowledge as suggested in Figure 1-1. The result is a coordinated knowledge system in which

strategies can call on principles and principles on strategies. Both can be used at any level of

generality.

2.3. Knowledge Representation

The multiplicity of levels of abstraction among both principles and problem-solving strategies

suggests representing both declaratively, allowing the more specific to inherit from the more general.

This representation both allows general knowledge to be accessed naturally (through inheritance) in

a variety of situations and has the following additional advantages: Using a declarative representation

allows a learning system to directly access and so to modify its knowledge. Arranging this knowledge

in a hierarchy allows the system to explain its reasoning at different levels of detail. Moreover, a

• o" ' " " N.

I Carbonell, Larkin & Reif

hierarchy that clusters like strategies into groups with common parents (a parent being the more

general strategy or strategies from which the various instantiations were derived) enables a system to

compare solution attempts and to reason analogically [5].

Implementing this hierarchical knowledge representation across multiple scientific domains

requires a flexible representation, one capable of encoding both problem-solving and principle

knowledge at different levels of specificity and of bringing the appropriate knowledge to bear as new

problems are encountered. In particular, we require a flexible inhertance mechanism [3, 7] that

operates on structured objects representing problem solving methods and scientific principles.

Lateral (or partial) inheritance is also required in analogical aspects of problem solving [15, 5]. We

need to combine the better aspects of semantic networks [2, 6] and schemas (or frames) [12, 1]. The

Schema Representation Language (SRL), presently in use in the Robotics !nstitute at CMU, presents

a good basic structure in which to build declarative knowledge schemas that can be procedurally

interpreted in the problem solving process.

To illustrate the implementational requirements, we describe the information encoded in the general

principle of conservation, which must contain the following statement:

Conserve(P,S) =_ (V t,t 2) (V (si), {sii) E P(si,t1) = E P(si,t2)i I

where,

9 P is the property conserved (such as mass, volume, number, or energy). P(si, t) is thevalue of P over subsystem si maasured at time t.

0 t1 and t 2 are two arbitrary times at which the property P is measured

* S is the closed system where conservation applied (loosely stated, the circumscription ofthe world over which conservation is postulated). S is treated computationally as a set.

o {s,} and {s1 are two partitions of the set S, corresponding to the possibly differentgroupings of elements in S at times t1 and t2 respectively. For instance, if the atoms in achemical process (such as oxidation) are rearranged into different molecules, the fi,'stpartition is the substances before the reaction (e.g., at t,) and the second partition is

s. defined by the products of the reaction (at t2).

-*,- Conservation of mass is represented as an instance of the conservation principle (with P = MASS)

with the set of preconditions necessary for it to apply (e.g., that there be no nuclear reactions with

mass-energy conversion -- E = Mc2 __ in the time interval [ti, t2]). Additionally, information on when it

may prove useful to apply this principle is represented', as is knowledge of which strategic reasoning

-We have some notions of acquiring this heuristic knowledge from experience, in much the same way that Mitchell does inhis LEX system 1131 -. but applied to our more general problem solving context.

-' ! . * . + - '.'-. .*. ' '* °=' ". ' , " 4% '

,",+ .*" * ", "R ' 'R 'q .* , *.' \ ~ .'-, * . ". ",=. ** ,

Towards a General Scientific Reasoning Engine 7

methods can utilize the conservation of mass principle and in what manner.

3. ConclusionWhereas the reasoning system discussed is yet to be built, we believe that our analysis and design

suggest a powerful new framework for building reasoning engines. Most expert systems, such as

MYCIN [16] and R1/XSEL [10, 11], do not exhibit the more significant properties in our design, as

summarized below:

* A separation of strategic problem-solving knowledge from knowledge in the domain.

e Declarative, modifiable, representation of problem-solving strategies as well as of domainprincipies and iacts.

%

* Representation at multiple levels of generality with vertical and lateral inheritance.allowing economy of representation for general knowledge and exhibiting non-brittlebehavior when a problem differs somewhat from the set foreseen by the system designer.

* The ability to use more general and familiar reasoning patterns in generating anexplanation to an external user.

In addition to its intrinsic Al interest, this project is relevant to questions central to plh=!osophy of

science, to psychology, and to educaticn. For example, to. what extent does the structure of science

use a limited n umber of basic inference patterns? A model that explicitly encodes these patterns and

uses them to make scientific inferences can shed some light on this question. Do human scientific

experts use general patterns of inierence? A mcdel of how this process might occur can be

compared with human expert data. Can students be helped to learn science by explicit teaching of

general inference patterns? A general reasoning system could support explanatory ICAI

systematically acquainting students with powerful re-usable patterns of inference.

-7 7 7:7 _1 TV .1

8 Carbonell, Larkin & Reif

4. References

1. Bobrow, D. G. and Winograd, T., "An Overview of KRL, a Knowledge RepresentationLanguage," Cognitive Science, Vol. 1,.No. 1, 1977, pp. 3-46.

2. Brachman, R. J., "On the Epistemological Status of Semantic Networks," in AssociativeNetworks, N. V. Findler, ed., New York: Academic Press, 1979.

3. Carbonell, J. G., "Default Reasoning and Inheritance Mechanisms on Type Hierarchies,"SIGART, SIGPLAN, SIGMOD Joint volume on Data Abstraction. 1980.

S.* 4. Carbonell, J. G., "A Computational Model of Problem Solving by Analogy," Proceedings of theSeventh Inteinational Joint Conference on Artificial Intelligpnce, August 1981 , pp. 147-152.

5. Carbonell, J. G., "Learning by Analogy: Formulating and Generalizing Plans from Past- Experience," ;n Machine Learning, An Artificial 'metiigence Appr ac,-, 3.S ,ichalski, J. G.

Carbonell and T. M. Mitchell, eds., Tioga Press, Palo Alto, CA, 1983.

6. Fahlman, S. E., NETL: A System for Representing and Using Rea! World Knowledge, MITPress, 1979.

7. Fox, M. S., "On Inheritance in Knowledge Representation," Proceedings of the SixthInternational Joint Conference on Artificial Intelligence, 1979, pp. 282-284.

8. Larkin, J. H., "The Role of Problem Representation in Physics," in Mental Models, D. Gentner& A. Collins, eds., Lawrence Erlbaum Associates, Hillsdale, N. J., 1983.

9. Lenat, D. B., "The Role of Heuristics in Learning by Discovery: Three Case Studies," inMachine Learning, An Artificial intellience Approach, R. S. Michalski. J. G. Carbonell and'r. M. Mitcholl, eds., "ioga Press, Palo Alto, CA, 1983.

10. McDermott, J., "RI: A Rule-Based Cou~figurer of Computer Systems," Tech. report, Dept. ofComputer Science, Carnegie-Mellon University, 1980.

11. McDermott, J., "XSEL: A Computer Salesperson's Assistant," in Machine Intelligence 10,Hayes, J., Michie, D. and Pao, Y-H., eds., Chichester UK: Ellis Horwood Ltd., 1982", pp.325-337.

12. Minsky, M., "A Framework for Representing Knowledge," in The Psychology of ComputerVision, P. Winston, ed., McGraw-Hill, New York, 1975.

13. Mitchell, T. M., Utgoff, P. E. and Banerji, R. B., "Learning by Experimentaticri: Acquiri.ig andS..-. Refining Problem-Solving Heuristics," in Machine Le.arning, An Artificial Intel!igence

Approach, R. S. Michalski, J. G. Carbonell and T. M. Mitchell, eds., Ticga Press, Palo Alto. CA,:-" 1983.

- 14. Polya, G., How to Solve It, Princeton NJ: Princeton U. Press, 1945.

15. Polya, G., Induction and Analogy in Mathematics, Princeton NJ: Princeton U. Press, 1954.

16. Shortliffe, E., Computer Based Medical Consultations: MYCIN, New York: Elsevier, 1976.

_%-

-U • '

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 1

Navy Navy

I Robert Ahlers 1 Dr. Ed HutchinsCode N711 Navy Personnel R&D CenterHuman Factors Laboratory San Diego, CA 92152NAVTRAEQUIPCEN

Orlando, FL 32813. 1 Dr. Norman J. KerrChief of Naval Technical Training

I Dr. Ed Aiken Naval Air Station Memphis (75)Navy Personnel R&D Center Millington, TN 38054San Diego, CA 92152

I Dr. Peter Kincaid1 Dr. Meryl S. Baker Training Analysis & Evaluation GroupNavy Personnel R&D Center Dept. of the NavySan Diego, CA 92152 Orlando, FL 32813

I Dr. Nick Bond i Dr. William L. Maloy (02)Office of Naval Research Chief of Naval Education and TrainingLiaison Office, Far East Naval Air StationAPO San Francisco, CA 96503 Pensacola, FL 32508

1 Dr. Richard Cantone 1 Dr. Joe McLachlanNavy Research Laboratory Navy Personnel R&D CenterCode 7510 San Diego, CA 92152Washington, DC 20375

1 Dr. James McMichael1 Dr. Fred Chang Navy PersonnelR&D CenterNavy Personnel R&D Center San Diego. CA 92152San Diego, CA 92152

1 Dr. George MoellerI Dr. Stanley Collyer Director, Behavioral Sciences Dept.Office of Naval Technology Naval Submarine Medical Research Lab

, 800 N. Quincy Street Naval Submarine Base9 Arlington, VA 22217 Groton, CT 06349

1 CDR Mike Curran 1 Dr William MontagueOffice of Naval Research NPRDC Code 13800 N. Quincy St. San Diego, CA 92152Code 270Arlington, VA 22217 1 Library. Code P201L

Navy Personnel R&D Center1 Dr. Jude Franklin San Diego, CA 92152

Code 7510Navy Research Laboratory 1 Technical DirectorWashington. DC 20375 Navy Personnel R&D Center

San Diego, CA 92152

I LT Steven D. Harris, MSC, USNRFD 1, Box 243 6 Commanding OfficerRiner, VA 24149 Naval Research Laboratory

Code 26271 Dr. Jim Hollan Washington, DC 20390Code 14Navy Personnel R & D Center I Office of Naval ResearchSan Diego, CA 92152 Code 433

800 N. Quincy SStreetArlington, VA 22217

* V - 9. tF'ft -- f

CMU/Larkin Carbonell (NR 667-494) 17-Nov-83 Page

Navy Navy

6 Personnel & Training Research Group I Mr John H. WolfeCode 442PT Navy Personnel R&D CenterOffice of Naval Research San Diego, CA 92152Arlington, VA 22217

1 Dr. Wallace Wulfeck, IIII Dr. Gil Ricard Navy Personnel R&D CenterCode N711 San Diego. CA 92152NTECOrlando, FL 32813

1 Dr. Robert G. SmithOffice of Chief of Naval OperationsOP-987HWashington, DC 20350

1 Dr. Alfred F. Smode, DirectorTraining Analysis & Evaluation GroupDept. of the NavyOrlando, FL 32813

1 Dr. Richard SnowLiaison ScientistOffice of Naval Research

Branch Office, LondonBox 39FPO New York, NY 09510

1 Dr. Richard SorensenNavy Personnel R&D CenterSan Diego, CA 92152

A 1 Dr. Frederick SteinheiserCNO - OP115Navy Annex

% Arlington, VA 20370

1 Dr. Thomas StichtNavy Personnel R&D CenterSan Diego, CA 92152

1 Dr. James TweeddaleTechnical Director

Navy Personnel R&D CenterSan Diego, CA 92152

1 Roger Weissinger-BaylonDepartment of Administrative SciencesNaval Postgraduate SchoolMonterey, CA 93940

1 Dr. Douglas WetzelCode 12Navy Personnel R&D CenterSan Diego. CA 92152

CHU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 3

* Marine Corps Army

I H. William Greenup 1 Technical DirectorEducation Advisor (E031) U. S. Army Research Institute for theEducation Center, MCDEC Behavioral and Social SciencesQuantico, VA 22134 5001 Eisenhower Avenue

Alexandria, VA 22333I Special Assistant for Marine

Corps Matters 1 Dr. Beatrice J. FarrCode lOOM U. S. Army Research InstituteOffice of Naval Research 5001 Eisenhower Avenue800 N. Quincy St. Alexandria, VA 22333Arlington, VA 22217

1 Dr. Harold F. O'Neil, Jr.

1 DR. A.L. SLAFKOSKY Director, Training Research LabSCIENTIFIC ADVISOR (CODE RD-i) Army Research InstituteHQ, U.S. MARINE CORPS 5001 Eisenhower AvenueWASHINGTON, DC 20380 Alexandria, VA 22333

1 Commander, U.S. Army Research Institutefor the Behavioral & Social Sciences

ATTN: PERI-BR (Dr. Judith Orasanu)5001 Eisenhower AvenueAlexandria, VA 22333

1 Joseph Psotka, Ph.D.ATTN: PERI-ICArmy Research Institute5001 Eisenhower Ave.Alexandria, VA 22333

I Dr. Robert SasmorU. S. Army Research Institute for theBehavioral and Social Sciences5001 Eisenhower AvenueAlexandria, VA 22333

.J

i.

•

CLU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 4

Air Force Department of Defense

1 U.S. Air Force Office of Scientific 12 Defense Technical Information CenterResearch Cameron Station, Bldg 5

Life Sciences Directorate, NL Alexandria, VA 22314

Bolling Air Force Base Attn: TCWashington, DC 20332

1 Military Assistant for Training and

1 Dr. Earl A. Alluisi Personnel Technology

HO. AFHRL (AFSC) Office of the Under Secretary of Defens

Brooks AFB, TX 78235 for Research & EngineeringRoom 3D129, The Pentagon

1 Bryan Dallman Washington, DC 20301AFHRL/LRTLowry AFB, CO 80230 1 Major Jack Thorpe

DARPAi Dr. Genevieve Haddad 1400 Wilson Blvd.Program Manager Arlington, VA 22209

N Life Sciences DirectorateAFOSR 1 Dr. Robert A. Wisher

Bolling AFB, DC 20332 OUSDRE (ELS)The Pentagon, Room 3D129

1 Dr. T. M. Longridge Washington, DC 20301AFHRL/OTEWilliams AFB, AZ 85224

1 Dr. John TangneyAFOSR/NLBolling AFB, DC 20332

1 Dr. Joseph YasatukeAFHRL/LRT

Lowry AFB, CO 80230

4.-,

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 5

Civilian Agencies Civilian Agencies

1 Dr. Paul G. Chapin I Dr. Joseph L. Young. DirectorLinguistics Program Memory & Cognitive ProcessesNational Science Foundation National Science FoundationWashington, DC 20550 Washington, DC 20550

I Dr. Susan ChipmanLearning and DevelopmentNational Institute of Education1200 19th Street NWWashington, DC 20208

I Edward EstyDepartment of Education, OERI

*MS 40

1200 19th St., NWWashington. DC 20208

I Dr. Arthur Melmed724 BrownU. S. Dept. of EducationWashington, DC 20208

41

1 Dr. Andrew R. MolnarOffice of Scientific and EngineeringPersonnel and Education

National Science FoundationWashington, DC 20550

1 Dr. Everett PalmerMail Stop 239-3NASA-Ames Research CenterMoffett Field, CA 94035

1 Dr. Ramsay W. SeldenNational Institute of Education1200 19th St., NWWashington, DC 20208

1 Dr. Mary StoddardC 10, Mail Stop B296Los Alamos National Laboratories

11 1Los Alamos. NH 87545

1 Chief, Psychological Reserch BranchU. S. Coast Guard (G-P-1/2/TP42)Washington, DC 20593

I Dr. Edward C. WeissNational Science Foundation1800 G Street, NWWashington, DC 20550

4..

°.

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 6

Private Sector Private Sector

1 Dr. Erling B. Andersen 1 Dr. John S. BrownDepartment of Statistics XEROX Palo Alto Research CenterStudiestraede 6 3333 Coyote Road1455 Copenhagen Palo Alto, CA 94304DENMARK

I Dr. Glenn Bryan

1 Dr. John R. Anderson 6208 Poe RoadDepartment of Psychology Bethesda, MD 20817Carnegie-Mellon UniversityPittsburgh, PA 15213 1 Dr. Bruce Buchanan

Department of Computer Science1 1 Psychological Research Unit Stanford UniversityNBH-3-44 Attn Stanford, CA 94305Northbourne HouseTurner ACT i601 I Bundministerium der VerteidigungAUSTRALIA -Referat P II 4-

.J Psychoiogical ServiceI Dr. Alan Baddeley Postfach 1328Medical Research Council D-5300 Bonn 1Applied Psychology Unit F. R. of Germany15 Chaucer RoadCambridge CB2 2EF I Dr. Pat CarpenterENGLAND Department of Psychology

Carnegie-Mellon University1 Dr. Patricia Baggett Pittsburgh, PA 15213

.i Department of PsychologyUniversity of Colorado I Dr. William ChaseBoulder, CO 80309 Department of Psychology

Carnegie Mellon University1 Eva L. Baker Pittsburgh, PA 15213DirectorUCLA Center for the Study of Evaluation 1 Dr. Micheline Chi145 Moore Hall Learning R & D CenterUniversity of California, Los Angeles University of PittsburghLos Angeles, CA 90024 3939 O'Hara Street

1 . r BPittsburgh. PA 15213• 1 Mr. Avron Barr

Department of Computer Science 1 Dr. William ClanceyStanford University Department of Computer ScienceStanford, CA 94305 Stanford University

Stanford, CA 943061 Dr. Menucha BirenbaumSchool of Education I Dr. Michael ColeTel Aviv University University of CaliforniaTel Aviv, Ramat Aviv 69978 at San DiegoIsrael Laboratory of Comparative

Human Cognition - D003AI Dr. John Black La Jolla, CA 92093Yale UniversityBox 11A, Yale Station 1 Dr. Allan M. CollinsNew Haven, CT 06520 Bolt Beranek & Newman, Inc.

50 Moulton StreetCambridge, MA 02138

CHU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 7

Private Sector Private Sector

I Dr. Emanuel Donchin 1 Dr. Michael Genesereth

Department of Psychology Department of Computer ScienceUniversity of Illinois Stanford UniversityChampaign, IL 61820 Stanford, CA 94305

I Dr. Thomas M. Duffy 1 Dr. Don GentnerDepartment of English Center for Human Information ProcessingCarnegie-Mellon University University of California, San DiegoSchenley Park La Jolla, CA 92093Pittsburgh, CA 15213

1 Dr. Dedre Gentner1 ERIC Facility-Acquisitions Bolt Beranek & Newman

4833 Rugby Avenue 10 Moulton St.Bethesda, MD 20014 Cambridge, MA 02138

I Dr. Paul Feltovich I Dr. Robert GlaserDepartment of Medical Education Learning Research & Development CencerSouthern Illinois University University of PittsburghSchool of Medicine 3939 O'Hara StreetP.O. Box 3926 PITTSBURGH, PA 15260Springfield, IL 62708

1 Dr. Marvin D. GlockI Professor Reuven Feuerstein 217 Stone HallHWCRI Rehov Karmon 6 Cornell UniversityBet Hakerem Ithaca, NY 14853JerusalemIsrael I Dr. Josph Goguen

SRI InternationalI Mr. Wallace Feurzeig 333 Ravenswood AvenueDepartment of Educational Technology Menlo Park, CA 94025Bolt Beranek & Newman10 Moulton St. 1 Dr. Daniel GopherCambridge, MA 02238 Faculty of Industrial Engineering

& Management1 Univ. Prof. Dr. Gerhard Fischer TECHNIONLiebiggasse 5/3 Haifa 32000A 1010 Vienna ISRAELAUSTRIA

I Dr. Bert Green1 Professor Donald Fitzgerald Johns Hopkins UniversityUniversity of New England Department of PsychologyArmidale, New South Wales 2351 Charles & 34th StreetAUSTRALIA Baltimore, MD 21218

1 Dr. Dexter Fletcher I DR. JAMES G. GREENOUniversity of Oregon LRDCDepartment of Computer Science UNIVERSITY OF PITTSBURGHEugene, OR 97403 3939 O'HARA STREET

PITTSBURGH, PA 152131 Dr. John R. Frederiksen

Bolt Beranek & Newman 1 Dr. Barbara Hayes-Roth50 Moulton Street Department of Computer ScienceCambridge, MA 02138 Stanford University

Stanford, CA 95305

Q4U/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 8

Private Sector Private Sector

1 Dr. Frederick Hayes-Roth 1 Dr. Walter KintschTeknowledge Department of Psychology525 University Ave. University of ColoradoPalo Alto, CA 94301 Boulder, CO 80302

1 Dr. Joan I. Heller 1 Dr. Stephen KosslynGraduate Group in Science and 1236 William James Hall

Mathematics Education 33 Kirkland St.c/o School of Education Cambridge, MA 02138University of CaliforniaBerkeley, CA 94720 1 Dr. Pat Langley

The Robotics InstituteI Dr. James R. Hoffman Carnegie-Mellon UniversityDepartment of Psychology Pittsburgh, PA 15213University of DeiawareNewark, DE 19711 1 Dr. Jill Larkin

Department of Psycnoiogy

I American Institutes for Research Carnegie Mellon University

1055 Thomas Jefferson St., N.W. Pittsburgh, PA 15213Washington, DC 20007

I Dr. Alan Lesgold

I Glenda Greenwald, Ed. Learning R&D CenterHuman Intelligence Newsletter University of PittsburghP. 0. Box 1163 3939 O'Hara StreetBirmingham, MI 48012 Pittsburgh, PA 15260

1 Dr. Earl Hunt I Dr. Jim LevinDept. of Psychology University of SaliforniaUniversity of Washington at San Diego

- Seattle, WA 98105 Laboratory fof ComparativeHuman Cognition - D003A

I Dr. Marcel Just La Jolla, CA 92093Department of PsychologyCarnegie-Mellon University 1 Dr. Michael LevinePittsburgh, PA 15213 Department of Educational Psychology

210 Education Bldg.

I Dr. Steven W. Keele University of Illinois. Dept. of Psychology Champaign, IL 61801

University of OregonEugene, OR 97403 1 Dr. Marcia C. Linn

Lawrence Hall of Science1 Dr. Scott Kelso University of CaliforniaHaskins Laboratories, Inc Berkeley, CA 94720270 Crown StreetNew Haven. CT 06510 1 Dr. Don Lyon

AFHRL/OT (UDRI)I Dr. David Kieras Williams AFB. AZ 85225

Department of PsychologyUniversity of Arizona 1 Dr. Jay McClellandTuscon, AZ 85721 Department of Psychology

MITCambridge, MA 02139

a(. i+ I I: " I i 'I i I + i i+I llm ,m l~ ldi m~ l ~ m-m ~

..... . :. -.. . . . . . . . . . . . .

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 9

Private Sector Private Sector

1 Dr. James R. Miller 1 Dr. Steven E. PoltrockComputer*Thought Corporation Bell Laboratories 2D-4441721 West Plano Highway 600 Mountain Ave.Plano, TX 75075 Murray Hill, NJ 07974

I Dr. Mark Miller 1 Dr. Mike PosnerComputer*Thought Corporation Department of Psychology1721 West Plano Parkway University of OregonPlano, TX 75075 Eugene, OR 97403

I Dr. Tom Moran 1 Dr. Lynn Reder

Xerox PARC Department of Psychology3333 Coyote Hill Road Carnegie-Mellon UniversityPalo Alto, CA 94304 Schenley Park

Pittsburgh, PA 15213

I Dr. Allen MunroBehavioral Technology Laboratories Dr. Fred Reif1845 Elena Ave., Fourth Floor Physics DepartmentRedondo Beach, CA 90277 University of California

Berkeley, CA 947201 Dr. Donald A NormanCognitive Science, C-015 1 Dr. Lauren ResnickUniv. of California, San Diego LRDC

La Jolla, CA 92093 University of Pittsburgh3939 O'Hara Street

1 Dr. Jesse Orlansky Pittsburgh, PA 1521Institute for Defense Analyses1801 N. Beauregard St. 1 Dr. Jeff RichardsonAlexandria, VA 22311 Denver Research Institute

University of DenverI Prof. Seymour Papert Denver, CO 80208

20C-109

Massachusetts Institute of Technology 1 Mary S. RileyCambridge, MA 02139 Program in Cognitive Science

Center for Human Information ProcessingI Dr. Nancy Pennington University of California, San DiegoUniversity of Chicago La Jolla, CA 92093Graduate School of Business1101 E. 58th St. 1 Dr. Andrew M. RoseChicago, IL 60637 American Institutes for Research

1055 Thomas Jefferson St. NWI LCOL F. Pinch Washington, DC 20007Directorate of Personnel Applied

Research 1 Dr. Ernst Z. RothkopfNational Defence HQ Bell Laboratories101 Colonel By Drive Murray Hill, NJ 07974OTTAWA, CANADA KIA

I Dr. William B. RouseI DR. PETER POLSON Georgia Institute of TechnologyDEPT. OF PSYCHOLOGY School of Industrial & SystemsUNIVERSITY OF COLORADO EngineeringBOULDER, CO 80309 Atlanta, GA 30332

I.1

I~ -7;7--- -;. 77-7777

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 10

Private Sector Private Sector

I Dr. David Rumelhart I Dr. Robert SternbergCenter for Human Information Processing Dept. of PsychologyUniv. of California, San Diego Yale UniversityLa Jolla, CA 92093 Box 11A, Yale Station

New Haven, CT 065201 Dr. Michael J. Samet-Perceptronics, Inc 1 Dr. Albert Stevens6271 Variel Avenue Bolt Beranek & Newman. Inc.Woodland Hills, CA 91364 10 Moulton St.

Cambridge. MA 02238

I Dr. Roger SchankYale University 1 David E. Stone, Ph.D.Department of Computer Science Hazeltine CorporationP.O. Box 2158 7680 Old Springhouse RoadNew Haven, CT 06520 dcLean, VA 22102

I Dr. Walter Schneider 1 DR. PATRICK SUPPESPsychology Department INSTITUTE FOR MATHEMATICAL STUDIES IN603 E. Daniel THE SOCIAL SCIENCESChampaign, IL 61820 STANFORD UNIVERSITY

STANFORD, CA 94305

1 Dr. Alan SchoenfeldMathematics and Education 1 Dr. Kikumi TatsuokaThe University of Rochester Computer Based Education Research Lab

4 Rochester, NY 14627 252 Engineering Research LaboratoryUrbana, IL 61801

1. Mr. Colin SheppardApplied Psychology Unit 1 Dr. Maurice TatsuokaAdmiralty Marine Technology Est. 220 Education BldgTeddington, Middlesex 1310 S. Sixth St.United Kingdom Champaign, IL 61820

1 Dr. H. Wallace Sinaiko I Dr. Perry W. ThorndykeProgram Director Perceptronics, Inc.Manpower Research and Advisory Services 545 Middlefield Road, Suite 140Smithsonian Institution Menlo Park. CA 94025801 North Pitt StreetAlexandria, VA 22314 1 Dr. Douglas Towne

Univ. of So. California1 Dr. Edward E. Smith Behavioral Technology LabsBolt Beranek & Newman, Inc. 1845 S. Elena Ave.

. .50 Moulton Street Redondo Beach, CA 90277

Cambridge, MA 021381 Dr. Kurt Van Lehn

1 Dr. Eliott Soloway Xerox PARCYale University 3333 Coyote Hill RoadDepartment of Computer Science Palo Alto, CA 94304P.O. Box 2158New Haven, CT 06520 1 Dr. Keith T. Wescourt

Perceptronics, Inc.I Dr. Kathryn T. Spoehr 545 Middlefield Road, Suite 140Psychology Department Menlo Park, CA 94025Brown UniversityProvidence, RI 02912

..'

CMU/Larkin & Carbonell (NR 667-494) 17-Nov-83 Page 11

Private Sector

1 William B. WhittenBell Laboratories2D-610Holmdel, NJ 07733

I Dr. Thomas WickensDepartment of PsychologyFranz HallUniversity of California405 Hilgarde AvenueLos Angeles, CA 90024

1 Dr. Mike WilliamsIntelliGenetics

124 University Avenue0. Palo Alto, CA 94301

1 Dr. Joseph WohlAlphatech, Inc.2 Burlington Executive Center111 Middlesex TurnpikeBurlington, MA 01803

4m*

SliS

-AN,

x .: *

VIt

V:i.k