— -~ r -r

-. —--,I,.__

,/ f Novembor 1978(:9 Report No. 1802

C ,1/

—-~~

Naomi Miyake and Donald A. Norman

TO ASK A QUESTION, ONE MUST KNOW ENOUGHTO KNOW WHAT IS NOT KNOWN

~~0

C..37 i~

-~ ~1 t

UNIVERSITY OF CALIFORNIA , SAN DIEGO CENTER FOR HUMAN INFORMATION PROCESSJN

This research was mpported by the Advanced Research Projects Age and the Office of Naval Research, Personnel and 7)viningResearch Programs and was monitored by ONR under Contract N000 28 NR 134-387, ~~~~ terms of ARPA OnierNo. 2284.The views and conclusions contained tn this document are thus o t ie author an u WôTlie interpreted ~iiiecess ily representingthc ~rnclal policies, either expressed or implied, of the Advanced Research Projects Agency, of the Office of Naval Research, or thethii’~’d States Government.Approved for public release; distribution unlimited. Reproduction in whole or part is permitted for any pw’pose of the UnitedSM:es Government.

IIn~~1~~~~4 flailsEcu~~I r’t CL. ASS IYICA f l O N Q~ TNIS PAGE (Wh mi Oat. tnI .r.~)

0 0~~~~T r~r~~ l , A ?IAIJ 0A(~ READ INSTRUCTI ONS

,I~i ~~ ~~~~~~~~ ‘~~ I ~~~~~ ~ BEFORE COMPL.ETIN G FORM_______

NUMBER ‘2 GOV1 ACCESSION NO 3. RECIPIENVS CAT A~ OO NUMSER

78,0 2/ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

r i~ii (mid SubUUs) .J~YPE O~ RV~ ñ 1 COVERED

E ~Techn I ca ~~~~~ ~~,~~~ To Ask A Question, One Must Know Enough I _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

to Know What Is Not Known _~~~~~~~ 6. PERFORMING ORG. REPORT dUMBER

7. AUT$~~R(,) -

S. CONTRAC T QR GRANT P4UMSER(.)

(3~ aom~i~~ ke~~~~OonaidAJNonMn / _~~ i/i/ A ~~~t9i i r~~~~~~’~~~ f9. P FORMING ORGA NIZA 11ON NAM E AND ADDRESS - ~ 1~ . PROGRAM CLEMENT. P!diC~T. ~~~ç~riter for Human Information Processing ~7 A REA & WO RK uNIT NUMBERS

University of California, San Diego MR 154 387La Jol la , CA 92093 ________________________

II. CONTROLLING OFFICE NAME AND AO ORESS Il. REPORT DATEPersonnel and Training Research Programs (‘~j ~~~~~~~ t~7i7 ~~Office of Naval Research (Code 458) ~~~~~ u.-,, uMIEw ~ ir-!r*ats- J

Arlington, VA 22217 2014. MONITORING AGENCY NAME 4 AOORESS(It ditf.rmi.t from Can •) IS. SECURITY CL ASS. (at tAd. ,.po$)

/ . Unclassifi edl5~ DECLASS I FICATIO H/ DOWNGRA DING

SCHEDULE

I4. QI5Tp~9U1~IQp4 S~TATEMEN1 (of this R.por t)

Approved for public release; distribution unlim i ted.

17. DISTR IBUT ION STATEMEN T (at A. .b.tr.eI antsr d In at.ck 20. ii diU.,mit from R.po?t)

IS. SUPPLEMENTARY HOlES

IL KEY WORDS (CantInu• an ,,at •. aide ii n.c.. .atv W td.nui ~ ~~ block nombat)

Question asking, complex learning, knowledge levels.

~~~~ASSTRA C t (C.nhInu. an ro’~~o. lid . It n~c•• v mid IdmIlSIV by bI..k I~~~~~at)

~ Is some appropriate level of prior ~~owledge a prerequisite for asking ques-tions about a new topic matter? The hypothesis that learners should ask themost questions when their ~~owledg. is well matched to the level of presentationtaa tested using two levels of background knowledge and two levels of difficultyqLffia ~~~~a~~~~~b~~~~~~ped ,~ . ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~vhile learning. Novice learners asked more questions on ihe easier material t~~~Learners with some training; trained learners asked more questions than noviceson the harder material. Accordingly , theories of question asking in learningshould address interactions between knowledge levels of askers and material.

DD , ~~~~~~ 1473 (DIllON OF I NOV SI IS OSIOt IT.SIN 0102.LP.0144101 tInr1~~cci flail

SECuRItY CL.AUIFICA 1IOH OF THIS PAGE (UP,.,, Dat. ~~Iat.I3

_________ — _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _- _ _ _

TO ASK A QUESTION ,

‘I ONE MUST KNOW ENOUGH TO KNOW

WHAT IS NOT KNOWN

Naomi Miyake and Donald A. Norman

Center for Human Information Processing Report No. 7802University of California , San Diego November 1978La Jolla , California 92093

The views and conclusions contained in this document are those of theauthors and do not necessarily reflect the policy of any agency of the UnitedStates Government. Partial, research support was provided by a contract f rcm theOffice of Naval Research and the Advanced Research Project s Agency , monitored byONR under contract NO0 1~ .-76—C—O628 , MR 15~1—387 . Support was also provided byNational Institutes of Mental Health grant MH—15828 to the Center for HumanInformation Processing. N. Miyake is supported by the Japan Society for thePromotion of Science.

The report is approved for public release ; distribution unlimited . Repro-duction in whole or in part is permitted for any purpose of the United StatesGovernment .

To Ask a Question , One Must Know Enough to Know What is Not Known

Naomi Miyake and Donald A. Norman

To ask a question of someone implies more than a need for informa-

tion. It also implies a proper structure of knowledge with which to

formulate the question and to int erpret the response. Thus , the ability

of a person to think of an appropriate question on a topic matter is a

complex function of the knowledge of that topic.

At a research seminar on computer techniques , we noted that

beginners at programming (to whom the seminar was addressed) asked few

questions and generated few comments. More expert programmers, however,

had many questions and, eventually, dominated the discussion. Reflec-

tion on this observation indicated that a general principle might be

involved: in order to be able to ask questions about new material, one’s

knowledge of the topic must be at an appropriate level: the number of

questions should be an inverted U—shaped function of the relationship

between student knowledge and task complexity.

Consider the novice exposed to expert level material: although

there should be many difficulties in understanding , the novice does not

even have the proper framework within which to ask questions. Thus ,

although at first glance , one might expect novices to be filled with

questions , there are strong theoretical reasons not to expect this.

Consider the expert exposed to novice material. Here, there should

be good structural framework for the material on the part of the expert,

- -~~~

- - -~~~~~

-~~~

-~~~~~~~~~~~

2

but a lack of questions: there are no gaps or inconsistencies in the

knowledge that need to be filled. The maximum number of’ questions

should occur when a person is well matched to the level of presentation.

There should be sufficient background knowledge, but the new material

should indeed be new, with inconsistencies or incomplete information

that causes the learner to question the interpretation, to seek more

information, or to confirm the existing understanding.

The ability to ask questions during the learning of new material,

therefore, seems to be a function of two variables: the existence of

appropriate knowledge structures, and the level of completeness of those

structures with respect to the new material. When there are two presen-

tation levels of the same material, easy and hard , the learner with the

ample knowledge structure should ask more questions on the harder

presentation and less on the easier one; the Learner with little

knowledge should ask most questions with the easier presentation and not

many with the harder presentation.

Considerable research has been done on question answering, but we

have been unable to find relevant experimental or theoretical studies on

the generation of questions. Some relevant issues are discussed in the

speech—act literature, but the relevance is indirect. Grosz (Note 1)

and L.evin and Moore (1977) have looked at a naturalistic dialogue situa-

tions, but they did not perform any analyses relevant to the issues

being raised here .

This paper tests the predicted interaction. The topic to be

learned is a computer text editor. This is a topic for which it is easy

I

3

to find subjects with no prior knowledge. Thus, it is relatively easy

to manipulate the amount of prior knowledge, as well as the difficulty

level of the material.

Method

Desian

We used two levels of background knowledge and two levels of diffi-

culty of the material to be learned. Subjects were preselected to have

little or no prior knowledge about text editors or computers. Trained

subjects received a pre—experimental training session. Other subjects

(novices) did not. There were two levels of the learning task: the

level called ~~~ was quite Less advanced than the level called ~~~~

There were tour groups of subjects, with 15 subjects per group:

trained, Easy; trained, Hard; novice, Easy; novice, Hard. The order of

the conditions were:

1. training (for subjects in trained conditions only);

2. a practice task;

3. a baseline task;

4. the learning task.

Trainini

During the training session , subjects learned a task related to the

final learning task. Using a simplified training manual , the subjects

in the two “trained ” groups learned three commands of the text editor

called ED: how to print, how to delete , and how to add a line of text.

After practicing these three commands on the computer terminal , the sub-

jects were given three new texts, each of which had mistakes marked:

they were asked to fix the mistakes . They could refer back to the

training manual. During the training , the experimenter was available

for help, but no more information than that given in the training manual

was provided . The amount of training was selected to be sufficient for

subjects to gain a sense of the text editor . The training session took

some 25 to 50 minutes, depending on the ability of each subject. All

the trained subjects reached the level where they could fix at least one

text out of three without any help .

The Dracti~e ~~~~~~~~ baseline tasks

The goal of the experiment was to collect freely occurring thoughts

and questions of the subjects who were learning different levels of

materials. Accordingly, it was felt important to train the subjects in

giving protocols naturally. (It takes a little bit of training to speak

thoughts and questions aloud freely, especially when the thoughts are

not responded to.) In addition , because different subjects would have

different baseline rates of asking questions and speaking , we wished to

get a calibration of their baseline rate.

All the materials for the practice , baseline , and learning tasks

were presented on individual index cards, held in a loose-leaf binder .

Each card had one or two sentences. Subjects read each card and were

instructed to say their thoughts and questions aloud.

The practice material was three sentences taken from an article in

a popular science magazine (Mahoney, 1978) . Each sentence was on a

~~~~~~~

_______ -~~~~~~~~~~~~~~

5

separate index card. After each subject finished reading and responding

to the third sentence, all the quesquestions up to that point were

answered by the experimenter. There was only one subject who did not

say anything, and so he was given the two most common questions and

their answers as examples.

The baseline task was to learn to play Owari , an African board and

counter game , quite different from most board games in the United

States. The explanation was made deliberately obscure. It consisted of

six very short paragraphs ( only one or two sentences each) , each on a

separate index card. Subjects’ questions were again answered at the end

of this session. (All subjects asked questions in the baseline task.)

I~t lear’nina ±.ask

Two versions of the instructional manual f o r the text editor NED

were constructed. The ~~~~ Manual was readable for people with no

experience with computers; it used only non—technical language, gave

detailed explanation for concepts, and taught five commands by first

presenting concrete examples and later more abstract explanations. The

~~~ Manual was more technical , written for experienced readers . It

taught seven commands and three special characters . It present ed

abstract conceptual explanations first , followed by a specification of

how to do each comnand . Both the Easy and Hard Manuals were divided

into 30 steps, each of which consisted of one or two sentences, each

step on a separate index card . (NED differs from ED, the editor used in

the training . The instruction manual used for ED was considerably dif-

ferent from either of the manuals used in the learning task for NED. )

_ _ _ ~~~~~~ -- -~~~~~

6

During the learning task, subjects were asked to read the material

page by page and to say aloud all the questions and thoughts which came

to mind. The instructions were specifically aimed at making the task

natural. Al]. the utterances were tape recorded. In order to keep the

amount of information available the same from subject to subject , the

experimenter never responded to these questions.

Sub lects

Sixty undergraduates in introductory psychology courses at the

University of California, San Diego served as subjects, either for

course credits or pay. They were pre—selected to have no knowledge of

either text editors or computers. It was hoped that no subjects would

be familiar with the base line task, Owari. However, 12 subjects recog-

n.tzed the game while reading the explanation. They were disqualified and

replaced with new subjects.

Results

Codina ~~~

Transcriets

All the protocols recorded during the baseline and learning tasks

were transcribed . There does not yet exist an adequate method for

analyzing naturally occuring questions. Considerable syntactic and

semantic variations were observed in these studies. Accordingly, we

first devised a rough classification of the utterances into ten dif—

ferent categories. It was difficult to make unambiguous classifications

of all the questions into these ten categories, however, and so these

categories were grouped into four composite, more conceptual categories

r~ ~~~~~~~~~~~~ - -

— -~~~~~~~ -~~

7

(plus a fifth category, looking back) . Four of the transcripts , one

from each experimental group, were randomly selected and coded by an

independent judge and the results were compared with the codings by one

of the authors (N.M.). Between coder reliability was only about 0.7 on

the ten categories, but was 0.95 on the composite classifications.

Examples of the questions and the categorizations used are shown in

Table 1. Only the five categories are considered in this paper. They

are:

1. Overall questions: The total number of questions, including

repetitions , paraphrases of previously asked questions , and inargi—

rial questions (an interrogative , but with no substantial content) .

2. Number ~~ ~onceots: The number of concepts for which further

clarification was asked . Operationally , this is the total number

of questions , minus repetitions , paraphrases and narginal ques-

tions.

3. Rvnotheses: The number of specific hypotheses or subjects ’ own

interpretations——a measure of how easily subjects could speculate

about the possible answers to their own questions.

The composite categories “number of concepts” and “hypotheses”

are both subsets of the composite category “overall questions”:

moreover , the two overlap in sharing some common subcategories .

~4. Confirmations: The number of utterances (other than questions)

in which subjects stated their own interpretation of the material.

Mere read ing aloud was not included . This category reflects sub—

je cts’ understanding of the material , but with an apparent need for

confirmation of the interpretation.

r~~~~r~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

8

Table 1

Coding Categories, Example statements , and Composite Categories.

(In parentheses are shown previously asked questions.)

~~~~~2ries Example statements

Questions

With contents

Original

With hypotheses “Is NED a person?”

Without hypotheses “What does NED stand for?”

Paraphrased

With hypotheses “(Is NED a person?) Or a computer?”

Without hypotheses “(What is the commnad?) What type of

thing do you want to type in, that

would be considered as conmand?”

Repeated “(What is the buffer?) Again, what is

the buffer?”

Without ccntents “What...?”

Judgements “I am confused .”

Confirmations “Oh , so insert goes backward , instead

of forward like append does. ”

Verbatim reading aloud

Composite Categories:

Overall questions = Al]. statements categorized as “Questions” above.

Number of concepts = All statements categorized as “Questions -

With contents — Original” above. - —

Hypotheses = “Questions — With contents — Original — With hypotheses”

+ “Questions - With contents — Paraphrased — With hypotheses.”

Confirmations = All statements categorized as “Confirmations” above.

- --

~

- . - .- - - -~~—-~~~~~~~~~~~~--- _ _ _

r _ _ _ _

9

5. J.,~g~—backs: The total number of the times subjects turned back

to previous pages . The number of pages turned back was riot

counted. (The data were recorded by the experimenter during the

baseline and learning tasks.)

Ad lustment ~~~ tendency ~g j~g~ auestions

There were considerable individual differences in the tendency to

speak aloud. To compensate for this propensity to speak , the rr.imber of

utterances in the experimental condition was adjusted by using the

number of questions observed in the baseline condition . The argument

for this adjustment is simple. In~the baseline condition , all subjects

were novices. Therefore , differences in the number of questions could

only result from individual differences in speaking styles. Moreover,

there were no significant differences in mean baseline frequency among

the tour experimental groups. The correlation between overall questions

in the baseline condition and overall questions in the learnIng task was

.66 . The regression equation for the number of overall questions was :

TQ 6.2 4. 1 . 7 BQ,

where TQ is the learning task questions and SQ is the baseline ques-

tions. Accordingly, the slope of the regression equation was used to

determine the adjustment factor: the number of overall questions , number

of concepts, and hypotheses were divided by 1.78Q . All statistical ana-

lyses were done on both raw scores and adjusted scores , and both values

will be reported. (We used the slope of the regression equation for our

The mean number of overall baseline questions was J4 .7 for the novicesubjects with the Easy Manual , ~4.9 for the trained subjects with theEasy Manual, 7.2 for the novice subjects with the Hard Manual, 5.5 forthe t rained subjects with the Hard Manual .

_ _ _ _ _

______ -

10

correction because it provided a satisfactory and easy estimate. Note,

howev er , that our results are not sensitive to the form of the adjust-

ment used . The primary point of the paper is confirmed by using the raw

data , without adjustment . Performing the adjustment does allow some

secondary results to reach significance.)

The number of “confirmations” and “look—backs ” were not adjusted

because the baseline trequencie5 for more than t~O subjects were zero

(this is about 2/3 of the subjects). Moreover, the correlation between

the baseline and the learning phase for these two categories was low Cr

= .23 and .38, for confirmations and look—backs respectively).

Findinas

The results for questions confirm expectations. All the adjusted

questioning scores show significant interactions between the training

conditions and the levels of the instructional manuals: with the easier

manual , novice subjects asked more questions than trained ; with the

harder manual , t rained subjects asked more questions than novices. Fig-

ure 1 shows the pattern based on the number of concepts , the most rigid,

qualitative measure of questioning. The L values for the interactions

calculated from the 2 X 2 analysis of variance (training conditions by

manual levels) are , £ (1 , 56) = 32.3k, ~~. < .01 for overall questions; .F.

(1 , 56) = 32.35 , ~ < .01 for the number of concepts ; F. (1 , 56) 6.59, .~~.

< .05 for hypotheses. 1

1. MaIn effects were not significant , except for training conditions onoverall questions (.~. (1, 56) = 6 . 93 , ~~. ( .05) and on concepts (.~~ (1 , 56)= ~~~~ ~ < .05). Main effects themselves do not convey much informa-tion , because the experimental hypotheses clearly call for the signifi-cant interactions.

—

~

-

~

-- .- --—---- -.-

~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

r.~Y - --- -

11

3 0-a4,

VI

a

0. 2.0~~ • Trained

~

..-.--‘.- ._~

___•

— -~

Navice

I

Easy Ha rd

Manual

Figure 1. Interactional pattern of adjusted scores for number ofconcepts asked during learning task.

_ _ _ _ _ _ _ _ _ _ _ _ _

______

12

The mean questioning scores (both raw and adjusted) are shown in

Table 2. With the exception of the hypotheses, all the raw frequencies

show the interacting pattern in the predicted order. The £ values for

the interactions are £ (1, 56) = 12.39, ~ ( .01 for overall questions; £(1 , 56) 11.72, ~ < .01 for the number of concepts; ~ (1 , 56) = 1.90,

for hypotheses. 2

Simple main effects for the factor of manual levels are mostly sig-

nificant. All, the questioning scores on the Easy Manual are signifi-

cantly greater for novices than for trained subjects. The £ values are

£ (1 , 56) = 34.61, ~~~, < .01 for overall questions; Z. (1 , 56) = 30.50, .a <

.01 for concepts; £ (1 , 56) = 13.76 , ~, < .0 1 for hypotheses. On the

Hard Manual , where the question is whether the trained subjects asked

more questions than the novices, only the category of hypotheses fails

to reach the significant level difference. The £ values for overall

questions, concepts, and hypotheses are £ (1 , 56) = ~4. 73 , ,~ < .05 ; Z (1 ,

56) 6.39 , ~ < .05 ; and ~ (1 , 56) = O.~5, respectively.

Mean number of confirmations and look-backs are shown in Figure 2

and Table 3 (these are raw, unadjusted data) . Although the interaction

of confirmation does not reach significance, CL (1 , 56) = 2.22) , the

data do show the appropriate pattern of interaction. Look-backs have a

unique pattern, with neither significant interaction CL (1 , 56) = 0.30),

nor any significant differences between the novices and the trained sub-

jects . The Hard Manual elicits more look-backs than the Easy Manual

(main ,~~~ for manual levels is £ (1, 56) = 4.92 , ~~. < .05).

2. Only one main effect , the training condition factor on overall ques-tions (L (1, 56) 5.44, ~g. < .05) is significant.

-~~~~~~- - - - -—~~~~~~~~~~~—-~~~

,~~~

______ .- . . — - -~~

-- •~~

- ---

~~~1~~

13

Table 2

Mean Number of Questions Asked

While Learning the Easy Manual or the Hard Manual.

(Standard deviations are given in parentheses.)

Easy Manual Hard Manual

Novice Trained Novice Trained

Raw frequencies

Overall 20.9 6.5 16.2 19.1

(11.9) C L~.5) (10.1) (10.1)

Concepts 15.1 5.9 12.5 15.3

( 7.3) ( 3 .9) ( 7.6) ( 7.6)

Hypotheses 7.0 1.5 7.5 7.2

C 7.5) C 1 . 7) ( 9.1) C 8.5)

Adjusted scores

Overall 2.8 0.9 1.5 2.2

( 1.2) C 0.7) ( 0.5) C 0.9)

Concepts 2.1 0.8 1.1 1.8

( 0.8) ( 0.6) C 0.4) ( 0.7)

Hypotheses 0.9 0.2 0.6 0.7

C 0.9) ( 0.2) ( 0.3) C 0.8)

~

14

Table 3

Mean Number of Confirmations and Look—backs.

(Standard deviations are given in parentheses.)

Easy Manual Hard Manual

Novice Trained Novice Trained

Confirmations 2.5 1.3 1.0 2.9

C 4.5) C 2.4) C 2.1) C 6.0)

Look—backs 1.3 1.7 2.7 3.9

C 1.3) ( 2.2 ) C 1.9) C 5.3)

__________________________ - - _ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - —

15

A3 0 ’

• • Tra ined

E 2.0

1.0’ ~~~

Novice

-

Easy HardManual

4.0 ’

~~~ Trai ned

~~.1

~ 3.0C) Novice

I ——

—-p

o 20’ p

0_. w ——

10

Easy HardManual

Figure 2. Interactional patterns of confirmations and look—backs .A: Confirmations. R: Look—backs.

—.— - - - - . - —— - _

L. -

• -‘~~ -~~~~-~~~~~~—--_ - _

—~~

--- • --- . - - — • — -

~---

__________

16

Discussion

The most interesting result is that novices do not ask many ques-

tions on material that is too difficult. This finding shows that a

theory of questioning that suggests that people ask questions to fill in

their knowledge structures is too simplistic. People do not appear to

be able to cope with material, too far beyond their present knowledge.

The number of overall questions gives a quantitative measure of the

general tendency to ask questions. The number of concepts and

hypotheses can be viewed as reflecting more qualitative aspects of ques-

tioning. To ask about a concept, the subjects must know what is missing

and what is necessary for further understanding; questions asking about

concepts thus imply the asker is at an appropriate distance from the

given material. To create a hypothesis, subjects should nave scme

expectation or inferred understanding beyond the given material;

hypotheses thus imply the asker is active in constructing a knowledge

structure. In this sense, the novices who learned on the easier manual

and the trained subjects on the harder manual had the greater tendency

to ask questions, not only quantitatively but qualitatively.

Because the two inatruotional manuals differed in structural pro-

perties, it is possible to inter what subjects in each group were doing.

The Easy Manual first presented concrete examples and later , more

abstract explanations; the Hard Manual presented the abstract concepts

first and then , later , the detailed specification of how to use each

co and.

~~~~~~~~~~~~~~~~~~~~~~~~ -

j

17

The novices given the Easy Manual started without knowledge of the

text editor task , but the manual provided concrete examples which could

be used as the “core” to start learning. Thus, these subjects could

follow the manual, constantly expanding the scope of their knowledge.

The novices on the Hard Manual also started without any specific

knowledge, but had to cope immediately with the abstract framework. It

was as if they did not know what they should know to know further. By

the time they came to more concrete examples, it was too late.

Look—backs can be regarded either as a measure of the complexity of

the material or as a measure of memory loss . There should be more

memory loss for the novices on the Hard Manual , because they had the

greatest trouble in following the material. Similarly, trained subjects

on the Hard Manual should need to check back on the materials , beca use

they were taking in the largest amount of information. These two types

of look—backs are indistinguishable with the present data. The data for

look—backs are consistent with either interpretation. However, the

relatively high score of the novices with the Hard Manual implies that

they were actually trying to understand the material, not just flipping

the page with boredom; they simply did not know what they should ask.

This paper is one of the first experimental attempts to manipulate

question asking in a learn.tng task. The results show that at least some

aspects of complex learning are accessible through the measures of ques—

tion asking. The results also show that any theory of question asking

in learning cannot simply use the gaps in a person’s knowledge as the

source of questions. To ask a question, knowing too little is just as

disadvantageous as knowing too much.

- - -.--

~

--— --

~

- -— -~~~~ -- - .-- - — ---~~~~- ---~~.--— -- --- - — - - _ •-

18

Reference Note

1. Grosz, 8. J. ~~~~~~~ reoresentation ~~~~~~~~ use ,Q~~ focus ,j~ dialoaue

imderstandiria. (Technical Note 15 1) . Menlo Park , Calif.: Stanford

Research Institute, July 1977.

19

References

Levin, J. A., & Moore, .J. A. Dialogue—games: Metacommunication struc-

tures for natural language interaction. Cognitive Science, 1977 ,

.1, 395—420.

Mahoney, M. J. Publish and perish. Human Behavior, February, 1978.

_ _ _ _ -- - -- - ---- -- . • - I

~~-~~-•-~~- ~~—.- -~~~~ •-,— —

• 20

Acknowledgement

We thank the LNR Research Group for their support and guidance.

Partial research support was provided by a contract from the Office of

Naval Research and the Advanced Research Projects Agency, monitored by

ONR under contract N0014—76—C—0628, NR 154—387. Support was also pro-

vided by a National Institutes of Mental Health grant MH—15828 to the

Center for Human Information Processing. N. Miyake is supported by the

Japan Society for the Promotion of Science. Address reprint requests to

Naomi Miyake, Department of Psychology, C—009, University of California

San Diego, L.a Jolla, California 92093. -

~~~~ -~~~~~~~~~~~

-~~~~~~~~ -••~~~~~~~~ _ _

r~~~~v - -

~~~~~~~~~~

504? 41? Ft•*

I (0’ 04 01104 I P*73 10(Ij; 3 *01 000,3.0147 0004rOry 3 0,’ . 00...1l 5000•( I 04? . 0 IIorr.* 0 *1* 0 4005234 1~ -••fl 700001(002 II~ /0*000 447101 44 (*9(2. 004424*08 144404 s.o.s. 30/00 3 0,8000,0. for S,7o,..01.. 010.01,. 64~4,. 1 •, 007 0. _fl - , 31.040 - 4 - - 07 0 0 / 0 4 4— 00045 . /4 52032 22 0 0.0 04flLu 4 .440 40000 *42 40., *0, 5 6 3 4 2 004000.4 *43 Of 1.0110730. . 00/. 000(o 4 lo . r .ooy l I to’...,,Lola . S. 0375 (~~ _4 2.. 4020. 44 92093

~,._,, 44 s on, .j , :4/ . •3 0?. J90A 001.0093 3 0~ . 5. 4. 00(31*4*0P?50S0 I *0.0,010 D0~0, I P0130.104004 920.04./u 1 0,. Jot, S. to..U.S. 0*092 P0019204o.0. 02000 0 440 I?8~0l 0f711• fl1041.PUflh J500 (It , 14 0 3 * 3 1 *•154 1*0/ 4..OIL - ,..l00,, 00. 404.03/SI. 4* 4 ,4 340* 0 0030 8401 00,00 50,0,4 537 3 0,730. 4* 00 *0.4 , ,,s ,,, — , /,*4*00009 . /4 9 0 3 0 ) ~~~. oat. . 7*4. 4200 . 4 00 003 ..~~~o ,.., *•0 1 00 1I 0,• ‘0• Or 000004 000*0* 0*03 Li AIfl Pro/U5000.10110 000.40 *., UIiSL,PLPIIO 3*401000 2*0 . I a. c. vociog nposa ( 1 so - lu, o, • - : 00 .31 0 0 0 7,00 JUL00 3/f 04500 0500(304 4 0L1.10400 API . Al *3229 4 I CAT SIC ‘,00/ 7.3 . 0,. 4 / .0/4*01. 220 9 0,0.041110 04.1,05 O,ooop( to~~~ 0(10U0Ofr fl_ ALA , SO01~ 2 /104.1. . 13 .4 ..o’:’~o . , 0t~ ’t,o ,, S,ot04.014*3.00.. 904*05, I 0?. 0000 C. 00.0(000 ) AII L,A3I I 000 59 37474 03. 0 0 1 / 0 0 4 0 7 1 0 * 0,. . ’ O . O . ’ / O f . 0 . . ’ . 4 1 . / A• 00. 04000 • .0.5. 4,0 008 P0 0*01.., , .4 04303 0004001 ‘P.10.0000 40. (4,0. 07 40031 0.l~~O ,~~/ 0 4 4 0 .40.03/ ( 0 0 / / P s L400fltos’, 0000 0540 3 0 00‘4 0 5 4 : / ~~0.t /•3/0~ ? I (.44 . 0000013/0 . 00*0000.0*0 *04 31.41..

3 00 0 0,1 — 0 - • .. .0 , • .0 -, 0 10’ 0. -• , 0_.410 0/. ’ (0P 020 • 0 *4’ *00101 (*3/nAy ( 810*0/fl) %n0400000 rO3/ 0.40 . 5 • - . l~~~~ ,• . — . .oflo,001004. *0*, 01 oI.44 - 1.0. 3.00.4’,? .‘I, •_~ 4~ L0~~ On *0 0.4 .000*44144(10. . 00 20310 00101*40 02 31 0*01. 9*11 0 / 4 * 411/0 3/.’.. 0 0 0 - . •000 / 140.0 01 20007

004*10.00 40. 01 7 00 III 04 f I .0. 00 ~ 044*4 ~ 04 —a *44/ ,34400* 044. 00 ‘0 00 03/000

:0. 15 0003. Ci... •— ,~~ Is 05*4 0? *0 20332 *14 000 (~o4 403 *0 /0 /0

.0 3 05 00* 2U °‘ 0-’ 0 ’- ~°’:O~ /0441 03/5.. •0 . ‘C_ ‘ .4 0 /0 00033 b0 / , , , * , 1 0 0 10.4 .2,7/50.50/ 0000/0 ’. *1, .0.00/ -- u*(?0L10 . 00 . 402 I 92. 11.35.53 134,0013 I 00 40020,, Ca.., “ C’ O0 i•001. .10. 0.34/ 000. 4-~~~~~ * 00*010*0 ~1034*C0 00044013 92P(C3 1.00002 64 4 405* 1 . 01/ U . /4 0 0! 00304033..A000/0000 . 304 o .‘ ‘ o fl 741 073.. . ) *00.0.0(01 1000 (ILS0( *0.00 . 000 3/ 00.040/000 0 4 0 , 0 0 . 0 3 0 c_A Ol10

~ ~,, 1000 ~~ 209 .0/o0 0 ....... 44. ... ’ .. 3000 04 . / 3 0 / 0 . P/So. PD Il 0*103/.. 000 .000010# COlt.? / 0, *40, /23 /0 /0/ 2000 00400.04 0,05‘4” /4

0/ /10* 09 4/~ .0. 000 4

~ 4 4CII

34(00 03/2 00 30 9 00*, .3/ •~ .13 4 . ...

1*3/00 • 0 73. 12504 000

~~~~~ 73. 0250* P0041073 I0~~~ 3 92 ~~~~~~~ A / SOPS 1401.0 . .2 ‘ o oI “~~~~~~~~

‘ - l~

- /4• 04 300/513/0 41L140044 , 0* 22200 Co 0$• 0 5*00 ‘‘ 0 01 (*100( 3 / / t o o I I •I 4. 0. 040000. 5900.?000y 07 0003.00.7 0.50,0*03/I 01 / 0 7 0 . 4 3

00 .0.00 7 (004 pg C~~ 400 4. 0. .0

10 ~~~~ 9 00000* 10 P 40/4/ ~~ 00 003/0. *1 4 to, CO11.09330., 1* .0.07/’ ~~~ 04 02 52 0/rI o. or 00.1., 3007ot*,j Of 0.?,o*.

to, ,....,‘ot & r.soo..rorg I 0’ hh1~~ 0. _~~ ‘~ ‘ I 0. 00,10 *0.,.. • —• ol 007 00 40 o OP.. *0 *0 00 0003/ 0.4. 93 ,* 07

10 so

‘1 4* *00 °0~ 4(0/ 31 00 2 4 10 40

0/ *0 01303 0*30 0.03/ *40 00 /4 50 Mt 44

34 40 00 Or 310400 03 7000 3/4 /0 0.(0 44 •_ 00 00 .304 ~~

~~~03/ ~• 00 00

04 .3/ 9.0*, 40104 0020037

04 3400 Ctat0.. 5 Th ola l J O . 30/ . 000 4.0011400,, 0/ 2020* 3 0,. *400,3 3/1.7(00 00 )0 /30 / /030/0 / /100

/0*00 0.000.. *30 nO 0 0.0* 30*, P 00*4 .3/0?. 0(001 II, 000.10 *,t /.00., 0 04000 03/ 0( 0.0.4,. *40.3/40103 3

• — .4*3. 097

P01.30,0 .. , 0 .~~~ . •~~. (.000,01 0.0.43/. P0004* 0103/ • 090400 -700.00 .5 10 05 030 00 .. PD a• or 00/04/0 L 93 • ,D 0000004o , 00 20013 0 0 / 0 4 / 4 / 0 4 07 _ 0 . . f . • , ,o .01 ‘ lolL 3,3 00 ,070 ’ .4-. /1,0/1:00 2 5 0?’ 04

0* 0 11.0 4,0 0 ~~ 3/0 00• — ,.0*0 03 .100’ ‘ ‘ 4.0031 44003/ , 0’, 2020* £4301.4 3*34.000, 0400 - .4000/103 P40 ,3/ / I . ,

‘00 09 0* 01 0/ 0,3/

000 4 9

0 0 * 0 0

01

4000 3(10 *0 2~5~~ 70 1000 7 114

~.

500

3 *01 *0 014.11100 0 5 5 00 00 /2 “30 0 0 / 0 / 000, . 13 5 , . 0 0 , 1 1 / 3 . 04 0 2 1 ) 1 VIP? . /7 7s30001043 50/,.?’ sp o t 4 ( , MD 2031.

P05*20.31.. 3/ o, , ,, .0 (300 /0 34004.’ 0 1 ) 0 ./ 0 • x ‘ 0 3 . - 1? ‘ 0000 *0*1*4’ 04.2 1041310* 0*164*0103 I 0 . . 2030 4 . r,.3.,o n,,

3/ 4’09 30 3 5 8 43/ O0 ,o. o. 15* ~,,

Dr 4.0 0 *3/ 53/ 3/40 40 344 38 0~s 3136400r’,.,0l 07 P0 1333/104, ‘00, - .

‘ 0/ 00/0/10 4 ‘*4/ 0 , 2 ’ - toront o 110m 04100r.lt, 0 0!. P0*0*7 001007 P.0,1., . 0. 0 0 , . 4)50 /10 * .. * p030.00? ,, . PA 1 52 ) 4 1300

400 10• 500 3 0/5030 , 000 4 340/1 04134*3073 01 3130000303 ‘ ‘ ‘~~ -.P., 33 3 444 75643/ 3039 II4~4 ~~

03 C-.40100/ 0*3 *7 007* 07730*107* P 5 3• *7. 23/0530 90,0 00 0*8000 P1030. C_I07 4000 £o, 4 * 3 0 , 00 0 / , /0.0. *4.3 *0,00roO / 0 3 0 1 / l o t . 19 33 0, 18*0111* 0/0*0* I 0 4 . 1?. 0000.0.4.- * .- 500 1 *00*0*000, *0.30. ~~~~493/43 CO *0310 i.r.s 04*0 , , p. n- - • 3 0/000021 10 . 04 223 33 4 333 Corol. 3010 01. ‘ °“•‘

~ 0I.’ ’ 000/01 3 / 7 3 )01/ ’o -

010 ,, ,., P.73/I.00010540 0.00*100 07414 Polo 0000 . /4 9033/0 0*3/3/0.0 n

-‘ . 0 (0 4 / 1 / 0 3 0.0030*

0.70. of 0.1.30. (47.7 0(10*.) ~~1~~~ l01 L I

1, / O r 0(301 , ACT 2000 , 00,3/1.10. IPX 0, 00*090

C 0r0440, 34 02)39- . 204. 73/ 0*3/

0(03000033 0? 1070754109393 7 400 00 Co 0...

~~ 4. 4 939 0 370141 0’ 000443041 0 . 0( 22 3 11,I**r5037 0? 4104(0 1, P0ft3W10033, 75 ‘724 3 0, 0034 1

30 0*7 I I 100! 0 °‘ 0*0.4 444 1 330

‘~ “‘ 00 104 00 00100 44S. 09 00 0 7*711.1.00000130700 . 00 20380 3 *3’ . *1,002.. A. 03.0 4*3 •1. 0* 00002 20 3? 0. ~~~1VT 7403*03O 7,,040..1 0 00./3/33 !000~~ ol 1700( 0 100.000000 0.90 . of P.?00.01747 10000070*. 740 22207• 5240 459/ 30l,~~0410 34.3. 001014. 2 0 4 . wboro 0.3*0.3,030 .. 905 J .? 34,0.1 64. 0*05 Co,’p.,.IIo. I *0 . P37*0 0,1544IP. ’ l•4 , 0 . £ 0 .0 50 J SOPII J . 1140*. CIII? 003r 00410 . 1* 33013 3330 9007 *or.*t 0*17 . 0? 7410740.043

000*OPCO (04.1/621 3.01. It*oo~~, 08 80000 0(1/0*3433 3/1 004045910(30 10(0* MS I 03’ . 032. 00.040 *00.004! . 00 40342

((3310/744, DC 20*50 7*74,3.07 * 4? P0?3S0O(%/M030’lIt? If 441033004000040? , 00 00702

- “ _ _ - - - - , —---.~~~~-- .. .~~~~~~~~~~, ~~~~ . —---