Think-Aloud Protocols February 5, 2010. Today’s Class Probing Question Think-Aloud Protocols...

Think-Aloud Protocols

February 5, 2010

Today’s Class

• Probing Question• Think-Aloud Protocols• Assignments

Probing Question for Friday, February 5

• Let’s say that the NSF has just given you a $100,000 grant to study student note-taking during classroom lecture– i.e. you can assign one minion to the task full-time for

a year

• Which qualitative method would you prefer to use, and why?– What factors would you be especially attentive to, to

ensure validity of your results?

Today’s Class


I need a volunteer

• A fast typist

• To try to record things people say while thinking aloud

• Only record the person thinking aloud

I need two more volunteers

I need two more volunteers

• Volunteer #1: Will think-aloud

• Volunteer #2: Will run the think-aloud

What should Volunteer #2 rememberwhile running the think-aloud?

What should Volunteer #2 rememberwhile running the think-aloud?

• Use appropriate instructions• Prompt the subject• Write down interesting events• Focus on what the subject does and says, not

“why”• Don’t help the subject

Long division

• Volunteer #1 will think-aloud

• Volunteer #2 will run the think-aloud

• So, Volunteer #2, give Volunteer #1 some instructions…

Long Division

16, 429, 782, 6007140

How did it go?

• Volunteer #1:

• How hard was the task?

• Did thinking aloud make it harder?

How did it go?

• Volunteer #2:

• How hard was it to get him/her to think-aloud?

• Did you worry about disrupting him or her?

• Audience:

• Any (friendly and constructive!) criticism for Volunteer #2?

Thoughts/comments

• From anyone

• On why this task might have been difficult/easy for the person thinking aloud, or the experimenter

• (or anything else)

Two more volunteers!

Drawing a scatterplot

• Volunteer #1 will think-aloud

• Volunteer #2 will run the think-aloud

Draw a scatterplot of this fake data

City Population(in 1000)

Number of Brazilian Restaurants

Worcester 155 4

Fitchburg 65 0

Boston 650 6

Providence 150 0

Springfield 70 1

Manchester 130 2

Hartford 220 4

New Haven 120 0

New Bedford 55 3

Arapiraca, Brazil 140 80

How did it go?

• Volunteer #1:



How did it go?

• Volunteer #2:



• Audience:


Thoughts/comments

• From anyone



The 9-Dot Problem(Kershaw & Ohlsson, 2004)

The 9-Dot Problem(Kershaw & Ohlsson, 2004)

Volunteer #2 will connect all 9 points with just 4 straight lines(you don’t already knowthe answer, right?)

The 9-dot problem

How did it go?

• Volunteer #2:



How did it go?

• Volunteer #1:



• Audience:


Thoughts/comments

• From anyone



Insight Problems• The biggest problem with thinking-aloud as a technique.

• When people are trying to figure something out, and it requires “insight”, thinking-aloud can be very disruptive.

• Therefore, if your participant is really baffled,• Now might not be exactly the time to push them to “KEEP

TALKING”

• In general, it’s best to avoid using think-aloud in these situations

• What else might you use to study process in these situations?


• One more time

Trogdor!

• http://www.homestarrunner.com/trogdor.html

http://www.homestarrunner.com/trogdor.html

How did it go?

• Volunteer #1:



How did it go?

• Volunteer #2:



• Audience:


Thoughts/comments

• From anyone



OK enough for now

• What have we learned about think-alouds from all of this?

• Bonus points for expressing your thought as a Beyer & Holtzblatt-style aphorism

What could you do?

• If you want to do a think-aloud on some uninterruptable process?

Example

• Take, for example, a helicopter pilot who needs to drop a ladder to someone while still flying

Crashing computer OK,Crashing helicopter NOT OK

What could you do?

Task-splitting

• You might want to do a think-aloud of the pilot’s processes as (s)he does each of these tasks separately: ie, a think-aloud as (s)he lowers a ladder, and a think-aloud as (s)he flies the plane

• This would show you the pilot’s major sub-goals, decision-points, and evaluation criteria for each task

Retrospective Think-Aloud

• Videotape the participant as they are doing the task

• Take a 10 minute break

• Then show the video to the participant and have them do colour-commentary.

• Take notes on their colour-commentary

• (Or even tape-record it)

Retrospective Think-Aloud is Also Useful…

• When you are studying someone who is so expert at an action, that their actions are completely automatized.

• In this case, thinking-aloud can disrupt or change someone’s behavior.

Volunteer #1

• Volunteer #1: Please tell me the cash value of the change you have in your pocket. Don’t think aloud.

Volunteer #2

• Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.

Volunteer #2

• Volunteer #2: Please THINK ALOUD while you determine the cash value of the change you have in your pocket.

• Volunteer #1, is this the procedure you used?

Thoughts? Questions?

Analysis

Analysis

• Let’s take the longest of the protocols that our noble live transcriber has produced

• Can you email it to me?– I’ll put it up on the screen

Segmentation

• It’s better to do segmentation from an audio recording than a transcript in general, but for this time…

Segmentation

• Your thoughts on where we should segment?

Cognitive Processes

• What are the cognitive processes found in these utterances?

Cognitive Processes

• Do we have an “irrelevant to task” category?– Recommended by van Someren

Coding

• Let’s code the whole transcript with regard to the list of cognitive processes

• For each segment, quickly write down your code, and then read it out loud– What level of agreement do we obtain?– Why do we disagree?

Question

• Let’s think about the coding scheme we used

• How did the coding scheme in Gobert (1999) differ from this?

Question

• Let’s think about the coding scheme we used

• How did the coding scheme in Gobert (1999) differ from this?

• Atheoretical .vs. theoretical

Advantages/Disadvantages

• What are the advantages and disadvantages of– atheoretical coding schemes– theoretical coding schemes

Thoughts? Questions?

What kind of participants could you involve in a think-aloud?

What kind of participants could you involve in a think-aloud?

• Complete novice – to see learning• Expert – to see how “it should be done”• Recent learner – to see process of someone

who has just acquired skill (may be more applicable for designing instruction)

For each think-aloud we did,what kind of participant did we have?

• Complete novice• Expert• Recent learner• Other

Where do think-alouds fit on this diagram?

ENTITATIVE

HOLISTIC

ESSENTIALISTEXISTENTIALIST

Janice

• Do you have anything you’d like to add, as one of the world’s foremost experts on think-alouds?

Today’s Class


Assignment #1

• Very high quality submissions overall!• Complete grading and commentary for

assignment #1 will be a little bit delayed

Assignment #2

• Any questions?

Assignment #3

• Will be handed out Monday (by email)

Think-Aloud Protocols February 5, 2010. Today’s Class Probing Question Think-Aloud Protocols...

Documents

Transcript of Think-Aloud Protocols February 5, 2010. Today’s Class Probing Question Think-Aloud Protocols...