5 Subjective and Objective Measurement Slides

download 5 Subjective and Objective Measurement Slides

of 41

Transcript of 5 Subjective and Objective Measurement Slides

  • 8/3/2019 5 Subjective and Objective Measurement Slides

    1/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Subjective and Objective Measurement

    Kai Romer

    TU-Munchen

    Department of Informatics

    Chair for Computer Aided Medical Procedures & Augmented Reality

    30. April 2007

    Kai Romer Subjective and Objective Measurement

    http://find/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    2/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Introduction

    different types of user interfaces

    need for empirically test these user interfaces

    ability to improve the quality of user interfaces

    Kai Romer Subjective and Objective Measurement

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    3/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Overview

    = need for measuring usability.

    What is usability?

    How do I measure usability for time-critical user interfaces ? subjectively objectively

    How do I evaluate results of measurement?

    Kai Romer Subjective and Objective Measurement

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    4/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    What is usability?

    EN ISO 9241-11 European Standart for measuring usability

    a lot of prior art mainly based on ISO 9241-11

    Kai Romer Subjective and Objective Measurement

    I d i

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    5/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Defining Usability

    The usability of a product is the degree to which specificusers can achieve specific goals in a particular

    environment with effectiveness, efficiency andsatisfaction. [1]

    really common definition based on ISO 9241-11

    seems not to be really helpfull for time-critical 3D UIs

    not mentioning the main issues of time-critical 3D UIs

    so new : not yet standardise.

    Kai Romer Subjective and Objective Measurement

    I t d ti

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    6/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Finding the Main Issues of Usability Testing

    These problems should be in mind, when testing time-critical 3D

    UIs Information Overload (IO)

    Change Blindness (CB)

    Perceptual Tunneling (PT)

    Cognitive Capture (CC)

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    7/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Measuring Distraction

    time-critical separation between main task and second task

    main task: driving, surgery, ...

    secondary task: using the interface

    interface may not disturb the main task!

    Influence of usage of supportive interface on the main task needsto be measured.

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    8/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Overview

    Objective Measurements

    Subjective Measurements

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    9/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Objective Measurement

    task time

    eye tracking

    occlusion test

    quality of main task

    peripheral detection

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    10/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Task Time

    time for application of secondary task

    main method for tests results:

    tsecondarytask ratio:

    tsecondarytasktmaintask

    danger of distraction

    length of distraction

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    11/41

    abakus

    IntroductionWhat is Usability?

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Eye Tracking: Glance Off Time

    time to

    focus on an instrument + read an instrument + focus back on the main task

    time to mentally process information not included

    out of the loop effect

    ratio between time used for main task and secondary task

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    12/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Occlusion Test

    measuring of time for secondary task

    main task neglected

    shutter glasses: open for secondary task (1,5s) closed for main task (about 4s)

    speed of perception, understanding, transcription

    interruptibility/resumability/time dependence

    problem: cheating by the test person

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    13/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Occlusion Test

    Kai Romer Subjective and Objective Measurement

    Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    14/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Quality of Main Task

    find distraction by measuring the quality of main task

    automotive

    track/speed keeping

    lane changing

    steering wheel reversal rate

    Kai Romer Subjective and Objective Measurement

    IntroductionWh i U bili ? I d i

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    15/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Track/Speed Keeping Ability

    usage of interfaces measured in simulators or real cars

    cameras, tracking devices (GPS)

    cognitive capture, perceptual tunneling

    Kai Romer Subjective and Objective Measurement

    IntroductionWh t i U bilit ? I t d ti

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    16/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Lane Change Task

    development by DaimlerChrysler

    car simulator

    road course with 3 lanes and indication, which lane to changeto

    difference between drivers and reference trajectory

    parallel task execution

    testing stress and workload while performing additional tasks disadvantage: unrealistic, unnatural driving

    advantages: simple, quick, standardised

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    17/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Lane Change Task

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    18/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    steering wheel reversal rate

    rate of intense steering wheel corrections usually after looking at a instrument

    peak detection algorithm, counted

    workload

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    19/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Peripheral Detection Task

    extra task: respond to LED

    LEDs on horizontal line in different angels perceptual tunneling (errors)

    cognitive capture (reaction time)

    performance indices (workload) (average reaction time, errors)

    PDT is a concept, not a standardised test

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    20/41

    abakus

    What is Usability?Measuring Usability

    Evaluating results of measuring usabilityConclusion

    IntroductionObjective MeasurementsSubjective Measurements

    Peripheral Detection Task

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    21/41

    abakus

    yMeasuring Usability

    Evaluating results of measuring usabilityConclusion

    Objective MeasurementsSubjective Measurements

    Subjective Measurements

    subjective measurement means asking the users

    several given questionnaires

    most common: NASA-TLX, SWAT

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    22/41

    abakus

    yMeasuring Usability

    Evaluating results of measuring usabilityConclusion

    Objective MeasurementsSubjective Measurements

    Different Questionairs

    NASA-TLX: NASA Task

    Load Index mental demand physical demand temporal demand performance

    effort level frustration level

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    23/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Objective MeasurementsSubjective Measurements

    Subjective Measurement

    SWAT: Subjective Workload Assessment Techniques loads for time mental effort psychological stress

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability? Introduction

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    24/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Objective MeasurementsSubjective Measurements

    Questionaire

    Questionaire is answered directly after the test. advantages

    ease of implementation low cost limited intrusion on task performance

    disadvantages: lack of precision due to repetition,

    understanding information overload

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    M i U biliIntroductionObj i M

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    25/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Objective MeasurementsSubjective Measurements

    Cognitive Walkthrough

    test user tells what he is doing/feeling

    work flow test

    thought flow

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    M i U bilit I f ti l St ti ti

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    26/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Inferential Statistics

    Evaluation of Test Results

    Introduction

    What is Usability?

    Measuring Usability

    Evaluating results of measuring usability

    Conclusion

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    27/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Inferential Statistics

    What does a Test look like?

    group of representative participants

    define structure of tests order of tests and participants

    within subject between subject

    what values should be measured

    evaluate the results

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    28/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Inferential Statistics

    Goals

    Many numbers few numbers Measures of central tendency:

    Mean: average Median: middle data value Mode most common data value

    Measures of variability / dispersion

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    29/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Inferential Statistics

    Descriptive Statistics

    Mean:

    X =

    XN

    Mean absolute deviation:

    |X X|N

    = MS

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    30/41

    abakus

    Measuring UsabilityEvaluating results of measuring usability

    Conclusion

    Inferential Statistics

    Descriptive Statistics

    Variance:

    s2

    =(X X)2

    N 1

    Standard deviation:

    s =(X X)2N 1

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    31/41

    abakus

    g yEvaluating results of measuring usability

    Conclusion

    Hypothesis Testing

    Goal: conclude characteristics from samples Is system A significantly better than System B?

    Is there any siginigcant difference anyway?

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usability Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    32/41

    abakus

    g yEvaluating results of measuring usability

    Conclusion

    Testing Hypothesises

    1. hypothesis

    2. develop testable hypothesis H1

    3. develop null hypothesis (logical opposite) H0

    4. calculate:H0 : p(X|H0)

    5. proves: Probability of logical opposite is small!6. common significance levels are:

  • 8/3/2019 5 Subjective and Objective Measurement Slides

    33/41

    abakus

    Evaluating results of measuring usabilityConclusion

    Methodes

    two variances: t-test

    several variances ANOVA

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring Usabilityi f i i i

    Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    34/41

    abakus

    Evaluating results of measuring usabilityConclusion

    t-Test

    The t-test tells us, if the variation between two groups issignificant

    .

    2 pairs, t-distribution

    implemented in e.g. Excel, SPSS n1,n2: sample count

    y1,y2: mean value

    s: mean variance

    compare to value table of distribution

    t =1

    1n1

    + 1n2

    y1 y2

    s

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring UsabilityE l ti lt f i bilit

    Inferential Statistics

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    35/41

    abakus

    Evaluating results of measuring usabilityConclusion

    ANOVA

    analysis of variance

    2 or more groups

    f-distribution

    F =n1n2(x1 x2)

    2

    (n1 + n2)varg

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?

    Measuring UsabilityEval ating res lts of meas ring sability

    Distraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    36/41

    abakus

    Evaluating results of measuring usabilityConclusion

    Summary

    introduced several surveys of testing 3D UIs in time-criticalenvironments

    presented, how tests can deliver reliable results

    showed ways of processing gathered data

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?Measuring Usability

    Evaluating results of measuring usabilityDistraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    37/41

    abakus

    Evaluating results of measuring usabilityConclusion

    Thank you for your attention!

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?Measuring Usability

    Evaluating results of measuring usabilityDistraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    38/41

    abakus

    Evaluating results of measuring usabilityConclusion

    NPL Report DICT 163/90: A preliminary design for a

    methodology for experimentally measuring usability, R ERenger, 1990

    http://www.hf.faa.gov/Webtraining/

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?Measuring Usability

    Evaluating results of measuring usabilityDistraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    39/41

    abakus

    Evaluating results of measuring usabilityConclusion

    Distraction from main Task

    main issue: distraction from main task. information overload (IO)

    too much information large amount of information high rate of new information contradictions in available information low signal to nosie ratio problem to remain informed problem to make a decision

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?Measuring Usability

    Evaluating results of measuring usabilityDistraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    40/41

    abakus

    Evaluating results of measuring usabilityConclusion

    Distraction from main Task

    perceptual tunneling (PT) fucusing on one stimulus like flashing signal leading to neglection of main task

    cognitive capture (CC) lost in thought caused by e.g. instruments requireing highly cognitive

    involvment leads to a loss of situational awareness

    Kai Romer Subjective and Objective Measurement

    IntroductionWhat is Usability?Measuring Usability

    Evaluating results of measuring usabilityDistraction from main Task

    http://find/http://goback/
  • 8/3/2019 5 Subjective and Objective Measurement Slides

    41/41

    abakus

    g g yConclusion

    Situation Awareness

    clear understanding of: what is going on in the environment

    what is going to happen in the nearest future

    Kai Romer Subjective and Objective Measurement

    http://find/http://goback/