Quality of Web UsabilityEvaluation Methods:

an empirical study on MiLE+Davide BolchiniUniversity of Lugano (Switzerland)

Franca GarzottoPolitecnico di Milano (Italy)

2

Outline

MotivationGoalsMiLE+Empirical study:

Experiment 1: „quick inspection“Experiment 2: „usability project“

Results and Discussion

3

Motivation

A proliferation of Usability Evaluation Methods (UEMs)Different philosophies - conception of „quality“, „usability“ and their interrelationships -, thus approaches and techniques

Every method should support the improvement of theusability of the application

However, ongoing discussion (and little agreeement) on the salient „quality attributes“ for UEMs…

4

Motivation - 2Quality attributes concerning the output of the evaluation

Effectiveness?Number of usability problems discovered …Thoroughness (found problems vs existing problems) …Reliability (consistency of results across different inspectors)…Validity (correcting predicting user‘s behaviour, no or minimized false positives…)Productivity…Scope…

Quality attributes concerning acceptability and adoptionLearnabilityApplicability and Compatibility in current practiceVerticalization on domainsReusabilityCost-effectiveness

Supporting arguments with empirical data

5

Goals

Evaluate MiLE+Ongoing development from 2000Used in consultancy projects, in training professionals + ca. 200 students a year.

Focus on few key attributes that we could measure in a realistic setting and that support effective adoption:

PerformanceEfficiencyCost-effectivenessLearnability

6

Goals

Verifying key „acceptability“ requirementsfor a UEM:

Become able to use the method after a reasonable time (1-3 person/days) of „study“Being able to detect the largest amount of usability problems with the minimum effortProducing a first set of results in a fewhours and a complete analysis in a few days

7

MiLE+ : Milano-Lugano Evaluation

MiLE+ is a usability inspection methodEvolution of two previous usability methods:

SUE (Systematic Usability Evaluation)MiLEBorrow general concepts from mainstream usabilityinspection approachesFoster a systematic, structured approach to the analysis, yet aimed at being particularly suitable to noviceevaluators

8

MiLE+ Activities Framework

TECHNICAL

INSPECTION

USER TESTING

EXPERT

EXPERT

APPLICATION

TechnicalHeuristics

NAVIGATION

CONTENT

TECHNOLOGY

SEMIOTICS

GRAPHICS

COGNITIVES

APPLICATION INDEPENDENT

USABILITY

APPLICATION DEPENDENT USABILITY

Validate / Invalidate

User ExperienceIndicators

(UEIs)

SC

EN

AR

IOS

Scenarios(as a tool,

not mandatory)

USER EXPERIENCE INSPECTION

UEIs

90% Framework

10% Specific

Library of Heuristics

Library of Scenarios

Library of UEIs

9

MiLE+ technical heuristics

(82) Technical Heuristics, coupled by a set of operational guidelines that suggest the inspection tasks to undertake in order to measure the various heuristics.Organized by design dimensions

Navigation: (36) heuristics addressing website info architecture and navigation structure Content: (8) heuristics addressing the general principles of content qualityTechnology/Performance: (7) heuristics addressing technology-dependent features of the applicationInterface Design: (31) heuristics that address the semiotics of the interface, the graphical layout, and some “cognitive” aspects

10

Operationalized Attributes - 1

Performance:Performance indicates the degree at which a method supports the detection of all existing usability problems for an application.It is operationalized as the average rate of the number of different problems found by an inspector (Pi) in given inspection conditions (e.g. time at disposal) against the total number of existing problems (Ptot)

Performance = avrg (Pi)/Ptot

11

Operationalized Attributes - 2

Efficiency:Efficiency indicates the degree at which a method supports a “fast” detection of usability problems.It is operationalized as the rate of the number of different problems identified by an inspector in relation to the time spent, and then calculating the mean among a set of inspectors:

Pi is the number of problems detected by the i-th inspector in a time period ti.

)(i

i

tPavrgEfficiency =

12

Operationalized Attributes - 3

Cost-effectiveness:Cost-effectiveness denotes the effort - measured in terms of person-hours - needed by an evaluator to:

carry out a complete evaluation of a significantly complex web applicationproduce an evaluation documentation that meets professional standards, i.e., a report that can be proficiently used by a (re)design team to address the usability problems.

13

Operationalized Attributes - 4

Learnability:Learnability denotes the ease of learning a method.We operazionalize it by means of the following factors:

the effort, in terms of person-hours, needed by a novice to become “reasonably expert” and to be able to carry on an inspection activity with a reasonable level of performancethe novice’s perceived difficulty of learning, i.e., of moving from “knowing nothing” to “feeling reasonably comfortable” with the method and “ready to undertake an evaluation”the novice’s perceived difficulty of applying the method in a real case.

14

Empirical study: general conditions

The overall study involved 42 participantsStudents from HCI course, Politecnico di Milano (Milano and Como campus)„novice inspectors“Preconditions:

No previous exposure to usabilityBasic background in web developmentEtherogeneous profile in terms of age and technical background

Preparatory conditions:5 hours classroom training on usability and MiLE+Assignment of learning material to study (MILE+ overview, technical heuristics library, 2 real-life case studies, excerptsfrom „USABLE“ - an online course on usability - and MILE+)

15

Exp.1: „quick inspection“Inspectors

16 graduate students (Como)Purpose

measure efficiency and performanceLearnability hypothesis: study effort to become proficient <= 2 days

Assigned Inspection Goals:Inspect website (Cleveland Museum of Modern Art) with MiLE+ technical inspection

Setting:Synchronus and syntopic individual inspection3 hour timeLimited inspection scope (2 main sections, around 300 page instances)One week after MiLE+ classes

Output produced:inspection notes including, for each usability problem, name, designdimension, description (max 3 lines), page URL

16

Key Results: from zero to hero…

Experiment 1Avg number of problems discovered: 14.8Hourly efficiency: avg 4.9 problems per hourExisting usability problems (team of experts): 41Performance: 36%

After 6 hours of training and a maximum of 15 hours of study of MiLE+, a novice can become able to detect more than one third of the existing usability problems.

17

Exp.2: „usability project“

Inspectors26 graduate students (Milano)

Purposemeasure perceived difficulty in learning and using MiLE+ andeffort needed to produce a professional evaluation report

Assigned Inspection Goals:Inspect website at choice (full application)

Setting:Asynchronous, team inspectionTwo months period

Output produced:Complete usability evaluation report

18

Key Results: Learning Effort

8%

75%

10% 7%

0%

10%

20%

30%

40%

50%

60%

70%

80%

<10 hours 10-13 hours 14-17 hours >18 hours

% students

Time invested in learning MiLE+ before and during the project

19

Key Results: Learning Difficulty

1%

73%

25%

1%3%

74%

22%

1%0

61%

37%

2%0%

10%20%30%40%50%60%70%80%

Very simple Simple Diff icult Very diff icult

Overall dif f iculty Technical Inspection User Experience Inspection

% students

Perceived difficulty in learning the various MiLE+ tasks

20

Key Results: Difficulty in Using theMethod

% students

0

47% 44%

9%0

79%

21%

0%0

19%

57%

24%

0

45% 41%

14%

0%10%20%30%40%50%60%70%80%90%

Very simple Simple Difficult Very difficult

Overall difficulty Technical InspectionScenario definition User Experience Inspection

Perceived difficulty in using the various MiLE+ tasks

21

Key Results: Individual Effort

32%

54%

12%

2%

0%

10%

20%

30%

40%

50%

60%

20-30 30-40 40-50 50-60

% students

Person/hours

Estimated effort for the entire evaluation process

22

Key Results: Individual Effort per Task

0%

20%

40%

60%

80%

100%

3-5 5-10 10-1515-2020-25

3-5 94% 8%

5-10 3% 69% 1% 4% 51%

10-15 39% 31% 23% 41%

15-20 40% 0% 34%

20-25 18% 0% 42%

Technical Inspection

Scenario definition

User Experience Inspection

Infra-team agreement

Documentation

% students

Person/hours

Estimated effort for the entire evaluation process BY task

23

Conclusions

Promote usability evaluation methods for adoptionFostering learnability and cost effectiveness

We have empirically substantiated the adoptionsuitability of MILE+, with encouraging results

Performant, efficient, cost-effective, easy to learn and use

Next steps:Comparative evaluation

Nielsen‘s heuristics, more unstructured approachesOngoing effort for transfering of MiLE+ to the industry:

Courses for professionalsGoogle - CH