1 CMT 4240 week 7 Qualitative Evaluation Methods.
-
date post
19-Dec-2015 -
Category
Documents
-
view
220 -
download
3
Transcript of 1 CMT 4240 week 7 Qualitative Evaluation Methods.
![Page 1: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/1.jpg)
1
CMT 4240 week 7
Qualitative Evaluation Methods
![Page 2: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/2.jpg)
2
Objectives
At the end of this lecture you will be able to carry out and analyse the following evaluation methods: Observation verbal protocols cognitive walkthrough
![Page 3: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/3.jpg)
3
Observation
Direct observationrecorded observationsoftware logging
![Page 4: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/4.jpg)
4
Direct observation
Observation in the real world and in the lab researcher makes notes of user behaviourquestions:
where do you position yourself? What do you watch the screen or the user? which user behaviour is relevant? How fast and how much can you write? Do you have to note conversation literally?
![Page 5: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/5.jpg)
5
Individual Exercise Observation
Imagine you are in the science museumYou evaluate the prototype “secrets
behind your screen”The design team of the museum has
asked you to find out how long on average visitors use the exhibit problems that visitors experience when using
the exhibit
![Page 6: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/6.jpg)
6
Exercise Observation
What do you do (step by step)?Which problems do you expect?What are you going to take with you?5 minutes
![Page 7: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/7.jpg)
7
What are you going to take with you?
Notepad and penscardboard with clip(stop-) watchobservation forms...
![Page 8: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/8.jpg)
8
Potential Problems
Where to position yourself? Need to see the user’s actions and his non-verbal behaviour.
User may feel disturbed, if you watch to closely.How to stop the time and to observe potential
usability problems at the same time.Hawthorne effect
![Page 9: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/9.jpg)
9
Potential Problems
Only one pass at the data collection observer has to decide in real time what is
important and what is not observer may miss important data observer may misconceive events or
misunderstand the user observer has to know from the outset what to
look out for
![Page 10: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/10.jpg)
10
Potential Problems
You may get distractedYour handwriting may be illegibleYou may not remember what you meant
with certain wordsYou may jot down too few words
![Page 11: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/11.jpg)
11
So what do you do?
Preparation: make observation sheets
start time, end timeuser age (approximately) and genderactionsproblems
![Page 12: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/12.jpg)
12
What do you do?
Position sideways between exhibit and user so that you can see both
Write down everything as explicitly as possible
conversations need to be written down as literal as possible
Keep to the observable FACTS not your interpretation of them
![Page 13: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/13.jpg)
13
Recorded observation
Advantage: you can repeatedly look at the recordings and
discover new things every time
Disadvantage: technology is fine, if it works... You need electricity, extension leads, several
cameras, tapes, etc. it can be costly
![Page 14: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/14.jpg)
14
Analysing Observation Material
Recordings: make transcripts transcripts and notes should be coded Informal analysis
what are the problems?
Task based analysis how does the user tackle a given task?
Performance based analysis obtain data on predefined performance measures
![Page 15: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/15.jpg)
15
Verbal protocols
Verbal protocols capture the user’s thoughts and spoken observations
Two types of verbal protocols think aloud protocol post event protocol
verbal protocols are often used in combination with video recording
![Page 16: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/16.jpg)
16
Think aloud protocol
The user is being asked to say out loud what he or she think while he or she is carrying out a task
![Page 17: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/17.jpg)
17
Post event protocol
The user is being asked to think out loud after the task has been completed
![Page 18: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/18.jpg)
18
Exercise Think Aloud Protocol:
Get together in pairsOne student is the observer and one
student is the subjectThe observer makes notesThe subject solves the following problem
while thinking out loud: how does the following series continue:
0, 1/3, 2/4, 3/5, 4/6, ?
![Page 19: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/19.jpg)
19
Solution?
![Page 20: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/20.jpg)
20
Analysis of the protocol
Go through your notes and ask the following questions:
How many different attempts to solve the problem can you identify?
How many different problems did your subject encounter?
Which statement in the protocol supports which problems solving activity and which problem encoutered?
![Page 21: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/21.jpg)
21
Cognitive Walkthrough
Evaluation method based on the idea that… the user learns about the interface in an
exploratory way, the user has goals, the user is applying simple means-ends
reasoning.
![Page 22: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/22.jpg)
22
Cognitive Walkthrough
Evaluation by an expert, who goes through a set task while imitating user performance
![Page 23: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/23.jpg)
23
Preparation: We need four things
1. User description including level of experience with computers and any assumptions made by the designer
2. System description including operations and performance (e.g. paper design)
3. Task description specifying the task that the expert has to carry out from users point of view
4. Action sequence describing the system display and the user actions needed to complete the given task. One system display and one user action together are one step.
![Page 24: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/24.jpg)
24
Process of carrying out a Cognitive Walkthrough
Prototype, user description, system description, task description and action list has to be prepared by the designer/developer
Designer gives these document to an expert.The expert reads the descriptions.The expert carries out the task by following the
action list.The expert asks the following questions with
EACH step of the action list:
![Page 25: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/25.jpg)
25
Questions Cognitive Walkthrough
1. Is the next goal clear at this stage?
2. Is the appropriate action obvious?
3. Is it clear that this action leads to the goal?
4. What problems are there in performing the action?
![Page 26: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/26.jpg)
26
Example: the Automatic Nail Painter
The automatic nail painter is a machine with which one can have one’s nails painted.
![Page 27: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/27.jpg)
27
User Description Automatic Nail Painter
The users are assumed to have no prior knowledge of how to operate the Automatic Nail Painter. However, they are assumed to have experience in putting nail varnish on nails manually
![Page 28: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/28.jpg)
28
System Description Automatic Nail Painter
The system allows one finger to be placed into the machine at a time. The user programmes the machine for use by choosing:- the number of fingers to be varnished- the colours of the varnish to be used- the number of coatings to be applied
![Page 29: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/29.jpg)
29
Task Description Automatic Nail Painter
The user must get all ten fingers varnished with one coat of purple varnish. The system is already switched on when the user begins his/her task. The task is completed, when the tenth fingernail is varnished.
![Page 30: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/30.jpg)
30
Action list Automatic Nail Painter
System: System displays ‘How many fingers do you want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shownSystem: System responds by displaying the current number of fingers selected to be varnished.User action: User presses ok when the correct number has been displayed
![Page 31: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/31.jpg)
31
Action list Automatic Nail Painter (cont.)
System: System displays ‘choose the colour of varnish you wish to apply.’User action: User must select the colour purple by choosing from the colour menu. This includes scrolling up and down until s/he finds purple. S/he then has to press the OK button.System: Systems displays ‘how many coats do you want to apply?’User action: User selects the number of coats by pressing the ‘+’ or ‘-’ button until the 1 is shown.
![Page 32: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/32.jpg)
32
Action list Automatic Nail Painter (cont.)
System: System displays the number of fingers to be varnished, the colour of the varnish and the number of coats to be applied. System asks for confirmationUser action: User presses OK to confirmSystem: System displays ‘Press start to begin varnishing.’User action: User presses start buttonetc.
![Page 33: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/33.jpg)
33
Step 1- Question 1 Automatic Nail Painter
From the action list: Step 1System: System displays ‘How many fingers do you
want to varnish?’User action: user selects number 10 by pressing the + or the - button until the appropriate number is shown
Question 1:Is the goal clear at this stage? Not completely.The user is asked to state the number of fingers to be varnished. He could be thinking that the system deals with one hand at a time.
![Page 34: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/34.jpg)
34
Step 1 - Questions 2, 3, and 4Automatic Nail Painter
Question 2:Is the appropriate action obvious? No.The system dos neither indicate that the user has to use the ‘+’ and ‘-’ buttons nor explain their use.
Question 3:Is it clear that this action leads to the goal? YesIf the user presses the ‘+’ and ’-’ button the numbers are clearly displayed.
Question 4: What problems are there in performing the action?
The action is simple to perform once the user has discovered the use of the buttons.
![Page 35: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/35.jpg)
35
Step 2 – Question 1Automatic Nailpainter
?
![Page 36: 1 CMT 4240 week 7 Qualitative Evaluation Methods.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d3b5503460f94a15aff/html5/thumbnails/36.jpg)
36
Summary
Direct and recorded ObservationVerbal Protocol
Think aloud protocol
Cognitive Walkthrough Preparation: System description, user
description, task description, Action sequence Process: for each step in the action sequence
four questions to be answered