Margje van de WielFaculty of Psychology, Universiteit Maastricht Thinking-Aloud & Protocol Analysis...

Margje van de Wiel Faculty of Psychology, Universiteit Maastricht

Thinking-Aloud &

Protocol Analysis

Margje van de WielDepartment of Experimental Psychology

Universiteit Maastricht

Unit 3.7.1Macro-Ergonomics: Evaluating human

work


Overview

• Research on cognitive processes underlying cognitive task performance:– Problem solving – Knowledge representation

• Focus on the think-aloud method– Collecting think-aloud protocols– Analyzing think-aloud protocols


Applications of cognitive psy-chological research methods• Fundamental psychological research

– Explorative– Testing theoretical hypotheses– Compare tasks or groups

• Cognitive task analysis• Building knowledge based systems• Usability research


Usability research

• How easy is it for people to set the time and alarm on a clock without a manual?

• How easy is it for them with the manual?• Do people understand the manual?• Do they use the manual?

• Video, computer program, web site etc. Compare conditions or groups


Problem solvingTower of HanoiThe objective of this game is to move all of the rings to the right most peg.You may only move one ring at a time; you must never allow a large ring to rest on a smaller ring.


Problem space (Newell & Simon, 1972)

Initial state

Goal state

Subgoalstate

Subgoalstate

Inter-mediate

states

Inter-mediate

states

Inter-mediate

states

Inter-mediate

states


Research on problem solving• Product evaluation• Observation and registration of

behavior (real-life, video, log files)

But Gives no direct information on cognitive processes


Research on problem solving• Introspection= Direct observation of mind

ButParticipants may give their interpretation of their cognitive processes when they prefer during task performance


Research on problem solving• Retrospection

ButParticipants may give their interpretation of their cognitive processes after task performance


Research on problem solving• Free recall • Stimulated recall

ButParticipants give their memories of the thoughts they had during problem solving after task performance


Instructions

• Retrospection– How did you proceed in solving the

problem?– What did you think during solving the

problem?

• Stimulated / Free recall– Report everything you can remember

about your thoughts during solving the problem


Research on problem solving• Questioning and prompting

ButParticipants may give their interpretation of their cognitive processes when prompted so that natural task performance may be disturbed


Research on problem solving• Dialogues in explanation and collaboration

ButThe cognitive processes might be different than in natural task performance


Research on problem solving• Thinking aloud=Saying everything that comes to mind

Advantages:• Concurrent verbalization• Close to natural task performance• Discourages interpretation and

explanation


Theoretical framework / verbalization theory (Ericsson & Simon, 1993)

=the information processing model

LTM

Verbal report

STM Taskperformance

Inputinfo


Theoretical framework

Verbal report

STM Taskperformance

Verbal encoding

VocalizationType of task- modality- verbalizability- instruction


The type of problem and the role of knowledge

Well-defined problems Ill-defined problems

Weak problem solving methods

Strong problem solving methods

Novices Experts

Knowledge lean task Knowledge rich task


Focus on knowledge representationThe nature and organization of knowledge is central in problem solving and not the strategies used (e.g. Chi, 1997)

The representation of a problem determines the problem solving process


Research on knowledge representation• Same methods as for investigating

cognitive processes, but analysis of the knowledge used

• Free recall– Report everything you remember about ...

=Reflection of the representation of (prior) knowledge, a text or a problem in the mind


The think-aloud method

• Collecting think-aloud protocols• Transcribing the verbal protocols Verbal reports as data• Segmenting the transcribed

protocols• Analyzing the transcribed protocols


Research questions

Gaining insight in cognitive processes and knowledge structures

• Explorative• Testing hypotheses

• Comparing types of problems• Comparing groups


Collecting think-aloud protocolsWhat do you need?• Experimenter• Participant• Problem• Tape-recorder and tape• Instructions


Instructions

• Make participant feel comfortable• Explain the purpose of the research• Explain the task• Explain thinking aloud• Emphasize that it is no test situation • Emphasize that the data are confidential• Practice problem(s) and talking out loud• Give feedback


Transcribing the protocols

• Type out the protocols as literally as possible, including – Natural punctuation– Eh, ehs– Thinking pauses; with dots or even better

to measure length– Indicate when something is said mumbling

so you don’t understand well

• Indicate interventions of experimenter


Example Tower of Hanoi (Anzai & Simon, 1979)

I’m not sure, but first I’ll take 1 from A and place it on B. And I’ll take 2 from A and place it on C. And then, I take 1 from B and place it n C. Experimenter: If you can, tell me where you place it there?Because there was no place else to go, I had to place 1 from B and C. Then, next, I placed 3 from A to B. …..


Segmenting the protocol

• Use natural pauses indicated by punctuation

• Indicate each segment by a line number

• No interpretation


Example Tower of Hanoi (Anzai & Simon, 1979)

1. I’m not sure, 2. but first I’ll take 1 from A and place it on B. 3. And I’ll take 2 from A and place it on C. 4. And then, I take 1 from B and place it on C. Experimenter: If you can, tell me where you place it there?5. Because there was no place else to go,.. …..


Analyzing verbal protocols

(Transcribed) and segmentedverbal protocols

Interpretation

RAW DATA

DATA


Interpretation

• Qualitative analysis

• Analyzing qualitative data in an objective and quantifiable way

Compare experimental conditions– Groups– Tasks– Instructions


Research question --> qualitative analysis• What comprehension strategies do

skilled readers use in reading a text?

Thinking aloud while reading a text Analysis: What strategies are used?

– Exploratory: describe the strategies used by the participants


Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991)• What strategies do skilled readers use in

dealing with degraded text?• Are these comparable to comprehension

strategies used in reading normal text? Thinking aloud while reading text with 3

types of instruction:– Just reading– + ‘Adults skim this type of material first’– Skim text for 60 sec and report the topic


Research questions --> testing hypotheses (Rauenbush & Bereiter, 1991) Analysis: what strategies are used?• Normal comprehension strategies

– Rereading – Reading ahead– Summing up

• Other comprehension strategies– (Letter generating strategy)– (Determining word type)


Analyzing verbal protocols 1

HypothesesCodingscheme

Verbalprotocols

Explore Test

Psychological theoryCognitive task analysis


Analyzing verbal protocols 1

HypothesesCodingscheme

Verbalprotocols

Problem solving <--> Knowledge representationprocesses (E&S, 93) (Chi, 97)


Cognitive task analysis

• What knowledge is available in the problem?

• What knowledge is necessary for problem solving?

• What steps need to be taken to solve the problem?

• What is the most efficient problem solving process?

• What strategies can be used?


Coding scheme

• Operationalizing hypotheses in coding categories

• Clearly defining coding categories and providing examples from pilot protocols

• Categories may differ in level of detail– Segments– Aggregations of segments


Analyzing verbal protocols 2• Code the verbal protocols• Determine interrater reliability (% of

correspondences, kappa, correlation)• Depict the results• Seek and interpret patterns in results

Eventually• Repeat process


Example medical diagnosis• Clinical case


Example of a case

A 70-year old female is admitted into hospital because of increasing shortness of breath. She has been very tired lately and tolerates her food badly. Sometimes she has chest pain, especially after dinner.Et cetera


Example medical diagnosis• Clinical case• Thinking aloud while diagnosing

the case– Problem solving process– The nature of the knowledge used


Analysis of think-aloud protocol

Problem solving process• Data examination• Data exploration• Data explanation• Hypothesis generation• Hypothesis evaluation• Discrepancy

processing• Meta-reasoning• Summarization

Nature of knowledge used

• Data interpretation• Clinical reasoning• Biomedical reasoning• Hypotheses (C or B)• (Repetitions)• Plans for actions• Metacognitive remarks


Example medical diagnosis• Clinical case• Thinking aloud while diagnosing

the case– Problem solving process– The nature of the knowledge used

• Free recall– Case representation


Analysis of free recall protocolsProposition analysis• Number of correct propositions• Number of inferences

(summarizing two or more propositions)


Proposition analysis

Proposition = single idea unit

Concept ConceptQualifier

CausalConditionalAttributeSpecificationIs a


Propositions in the case

Female - 70-year old (attribute)

- is admitted into hospital (attribute) - (because of) shortness of breath (attribute)

- increasing (specification)

- very tired (attribute)

- lately (specification)

- tolerates her food badly (attribute)


Other literature

• Chi, M.T.H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. The journal of the learning sciences, 6(3), 271-315. SGT 3179

• Ericcsson, K. A. & Simon, H. A. (1993). Protocol analysis: verbal reports as data. Cambridge, MA: MIT Press (ch 6 & 7). SL BF 180


Other literature

• Reitman Olson, J. & Biolsi, K. J. (1991). Techniques for representing expert knowledge. In: K. A. Ericsson & J. Smith (Eds.) Toward a general theory of expertise: Prospects and Limits (pp. 240-285). Cambridge: Cambridge University Press. ME EAM 374 / Margje

Margje van de WielFaculty of Psychology, Universiteit Maastricht Thinking-Aloud & Protocol Analysis...

Documents

Transcript of Margje van de WielFaculty of Psychology, Universiteit Maastricht Thinking-Aloud & Protocol Analysis...