5 Subjective and Objective Measurement Slides
-
Upload
mythili-anbazhagan -
Category
Documents
-
view
226 -
download
0
Transcript of 5 Subjective and Objective Measurement Slides
-
8/3/2019 5 Subjective and Objective Measurement Slides
1/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Subjective and Objective Measurement
Kai Romer
TU-Munchen
Department of Informatics
Chair for Computer Aided Medical Procedures & Augmented Reality
30. April 2007
Kai Romer Subjective and Objective Measurement
http://find/ -
8/3/2019 5 Subjective and Objective Measurement Slides
2/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Introduction
different types of user interfaces
need for empirically test these user interfaces
ability to improve the quality of user interfaces
Kai Romer Subjective and Objective Measurement
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
3/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Overview
= need for measuring usability.
What is usability?
How do I measure usability for time-critical user interfaces ? subjectively objectively
How do I evaluate results of measurement?
Kai Romer Subjective and Objective Measurement
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
4/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
What is usability?
EN ISO 9241-11 European Standart for measuring usability
a lot of prior art mainly based on ISO 9241-11
Kai Romer Subjective and Objective Measurement
I d i
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
5/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Defining Usability
The usability of a product is the degree to which specificusers can achieve specific goals in a particular
environment with effectiveness, efficiency andsatisfaction. [1]
really common definition based on ISO 9241-11
seems not to be really helpfull for time-critical 3D UIs
not mentioning the main issues of time-critical 3D UIs
so new : not yet standardise.
Kai Romer Subjective and Objective Measurement
I t d ti
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
6/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Finding the Main Issues of Usability Testing
These problems should be in mind, when testing time-critical 3D
UIs Information Overload (IO)
Change Blindness (CB)
Perceptual Tunneling (PT)
Cognitive Capture (CC)
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
7/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
IntroductionObjective MeasurementsSubjective Measurements
Measuring Distraction
time-critical separation between main task and second task
main task: driving, surgery, ...
secondary task: using the interface
interface may not disturb the main task!
Influence of usage of supportive interface on the main task needsto be measured.
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
8/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
IntroductionObjective MeasurementsSubjective Measurements
Overview
Objective Measurements
Subjective Measurements
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
9/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
IntroductionObjective MeasurementsSubjective Measurements
Objective Measurement
task time
eye tracking
occlusion test
quality of main task
peripheral detection
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
10/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
IntroductionObjective MeasurementsSubjective Measurements
Task Time
time for application of secondary task
main method for tests results:
tsecondarytask ratio:
tsecondarytasktmaintask
danger of distraction
length of distraction
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
11/41
abakus
IntroductionWhat is Usability?
Measuring UsabilityEvaluating results of measuring usability
Conclusion
IntroductionObjective MeasurementsSubjective Measurements
Eye Tracking: Glance Off Time
time to
focus on an instrument + read an instrument + focus back on the main task
time to mentally process information not included
out of the loop effect
ratio between time used for main task and secondary task
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
12/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Occlusion Test
measuring of time for secondary task
main task neglected
shutter glasses: open for secondary task (1,5s) closed for main task (about 4s)
speed of perception, understanding, transcription
interruptibility/resumability/time dependence
problem: cheating by the test person
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
13/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Occlusion Test
Kai Romer Subjective and Objective Measurement
Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
14/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Quality of Main Task
find distraction by measuring the quality of main task
automotive
track/speed keeping
lane changing
steering wheel reversal rate
Kai Romer Subjective and Objective Measurement
IntroductionWh i U bili ? I d i
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
15/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Track/Speed Keeping Ability
usage of interfaces measured in simulators or real cars
cameras, tracking devices (GPS)
cognitive capture, perceptual tunneling
Kai Romer Subjective and Objective Measurement
IntroductionWh t i U bilit ? I t d ti
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
16/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Lane Change Task
development by DaimlerChrysler
car simulator
road course with 3 lanes and indication, which lane to changeto
difference between drivers and reference trajectory
parallel task execution
testing stress and workload while performing additional tasks disadvantage: unrealistic, unnatural driving
advantages: simple, quick, standardised
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
17/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Lane Change Task
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
18/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
steering wheel reversal rate
rate of intense steering wheel corrections usually after looking at a instrument
peak detection algorithm, counted
workload
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
19/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Peripheral Detection Task
extra task: respond to LED
LEDs on horizontal line in different angels perceptual tunneling (errors)
cognitive capture (reaction time)
performance indices (workload) (average reaction time, errors)
PDT is a concept, not a standardised test
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
20/41
abakus
What is Usability?Measuring Usability
Evaluating results of measuring usabilityConclusion
IntroductionObjective MeasurementsSubjective Measurements
Peripheral Detection Task
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
21/41
abakus
yMeasuring Usability
Evaluating results of measuring usabilityConclusion
Objective MeasurementsSubjective Measurements
Subjective Measurements
subjective measurement means asking the users
several given questionnaires
most common: NASA-TLX, SWAT
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
22/41
abakus
yMeasuring Usability
Evaluating results of measuring usabilityConclusion
Objective MeasurementsSubjective Measurements
Different Questionairs
NASA-TLX: NASA Task
Load Index mental demand physical demand temporal demand performance
effort level frustration level
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
23/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Objective MeasurementsSubjective Measurements
Subjective Measurement
SWAT: Subjective Workload Assessment Techniques loads for time mental effort psychological stress
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability? Introduction
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
24/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Objective MeasurementsSubjective Measurements
Questionaire
Questionaire is answered directly after the test. advantages
ease of implementation low cost limited intrusion on task performance
disadvantages: lack of precision due to repetition,
understanding information overload
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
M i U biliIntroductionObj i M
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
25/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Objective MeasurementsSubjective Measurements
Cognitive Walkthrough
test user tells what he is doing/feeling
work flow test
thought flow
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
M i U bilit I f ti l St ti ti
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
26/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Inferential Statistics
Evaluation of Test Results
Introduction
What is Usability?
Measuring Usability
Evaluating results of measuring usability
Conclusion
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
27/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Inferential Statistics
What does a Test look like?
group of representative participants
define structure of tests order of tests and participants
within subject between subject
what values should be measured
evaluate the results
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
28/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Inferential Statistics
Goals
Many numbers few numbers Measures of central tendency:
Mean: average Median: middle data value Mode most common data value
Measures of variability / dispersion
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
29/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Inferential Statistics
Descriptive Statistics
Mean:
X =
XN
Mean absolute deviation:
|X X|N
= MS
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
30/41
abakus
Measuring UsabilityEvaluating results of measuring usability
Conclusion
Inferential Statistics
Descriptive Statistics
Variance:
s2
=(X X)2
N 1
Standard deviation:
s =(X X)2N 1
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
31/41
abakus
g yEvaluating results of measuring usability
Conclusion
Hypothesis Testing
Goal: conclude characteristics from samples Is system A significantly better than System B?
Is there any siginigcant difference anyway?
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usability Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
32/41
abakus
g yEvaluating results of measuring usability
Conclusion
Testing Hypothesises
1. hypothesis
2. develop testable hypothesis H1
3. develop null hypothesis (logical opposite) H0
4. calculate:H0 : p(X|H0)
5. proves: Probability of logical opposite is small!6. common significance levels are:
-
8/3/2019 5 Subjective and Objective Measurement Slides
33/41
abakus
Evaluating results of measuring usabilityConclusion
Methodes
two variances: t-test
several variances ANOVA
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring Usabilityi f i i i
Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
34/41
abakus
Evaluating results of measuring usabilityConclusion
t-Test
The t-test tells us, if the variation between two groups issignificant
.
2 pairs, t-distribution
implemented in e.g. Excel, SPSS n1,n2: sample count
y1,y2: mean value
s: mean variance
compare to value table of distribution
t =1
1n1
+ 1n2
y1 y2
s
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring UsabilityE l ti lt f i bilit
Inferential Statistics
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
35/41
abakus
Evaluating results of measuring usabilityConclusion
ANOVA
analysis of variance
2 or more groups
f-distribution
F =n1n2(x1 x2)
2
(n1 + n2)varg
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?
Measuring UsabilityEval ating res lts of meas ring sability
Distraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
36/41
abakus
Evaluating results of measuring usabilityConclusion
Summary
introduced several surveys of testing 3D UIs in time-criticalenvironments
presented, how tests can deliver reliable results
showed ways of processing gathered data
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?Measuring Usability
Evaluating results of measuring usabilityDistraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
37/41
abakus
Evaluating results of measuring usabilityConclusion
Thank you for your attention!
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?Measuring Usability
Evaluating results of measuring usabilityDistraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
38/41
abakus
Evaluating results of measuring usabilityConclusion
NPL Report DICT 163/90: A preliminary design for a
methodology for experimentally measuring usability, R ERenger, 1990
http://www.hf.faa.gov/Webtraining/
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?Measuring Usability
Evaluating results of measuring usabilityDistraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
39/41
abakus
Evaluating results of measuring usabilityConclusion
Distraction from main Task
main issue: distraction from main task. information overload (IO)
too much information large amount of information high rate of new information contradictions in available information low signal to nosie ratio problem to remain informed problem to make a decision
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?Measuring Usability
Evaluating results of measuring usabilityDistraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
40/41
abakus
Evaluating results of measuring usabilityConclusion
Distraction from main Task
perceptual tunneling (PT) fucusing on one stimulus like flashing signal leading to neglection of main task
cognitive capture (CC) lost in thought caused by e.g. instruments requireing highly cognitive
involvment leads to a loss of situational awareness
Kai Romer Subjective and Objective Measurement
IntroductionWhat is Usability?Measuring Usability
Evaluating results of measuring usabilityDistraction from main Task
http://find/http://goback/ -
8/3/2019 5 Subjective and Objective Measurement Slides
41/41
abakus
g g yConclusion
Situation Awareness
clear understanding of: what is going on in the environment
what is going to happen in the nearest future
Kai Romer Subjective and Objective Measurement
http://find/http://goback/