UsabilityTes,ng:WhatHaveWeOverlooked?

HyoungtaeCho

CMSC737So>wareTes,ng

Whatisusability?•  “How well users can learn and use a product to achieve their 

goals and how sa5sfied they are with that process” 

•  Cannotbedirectlymeasured;quan9fiedbymeansofindirectmeasuresora<ributes:–  Thenumberofreportedproblems

–  Easeoflearning–  Efficiencyofuse–  Memorability

–  Errorfrequencyandseverity–  Subjec9vesa9sfac9on

Terminology•  UsabilityTes9ng:toevaluateaproductbytes9ngiton

representa9veusersfromthetargetaudiencethroughspecifictasks

•  ExpertReview:performedbyusabilityexpertstoiden9fypoten9alusabilityproblems

•  ReturnOnInvestment(ROI)ofUsability:Thereturnsfromusabilityimprovementagainstaddedefforts/costs(9me&money)

UsabilityTes9ngvs.ExpertReviewUsabilityTes,ng ExpertReview

Whoconducts? Representa9vepeoplefromtargetaudience

AcoupleofUsabilityExperts

Useful Whenfindingrealusabilityproblems

Whenfindingviola9onsofusability/designstandards

Length 4‐6weeks 1‐3weeks

Disadvantage ‐ ExpensiveandTimeconsuming‐ Heavilydependsoniden9fyingrighttargetgroup,accuracyoftes9ngprotocol

‐ FalsePosi9ves‐ Misstherealproblemsthatcauseuserstofailtasks

JakobNielsen:theMagicNumber5

•  Usabilitytestsareverycostlyandcomplex?–  NotTrue–  “Elaborate usability tests are a waste of resources” 

•  Nomorethan5usersandrunningasmanysmalltestsasyoucanafford

Threetestsetswith5users

each

15testsetswith1user

Atestsetwith15users

misled by a spurious user

same findings

JakobNielsen:theMagicNumber5

•  “Discountusability”model

•  Thepropor9onofUsabilityproblemsfound(%)=–  L:usabilityproblemsfoundbyasingleuser

–  n:numberoftestusers

L=31%84.36%

5

Conflictsagainstthemagicnumber

•  Arguingagainstthefive‐userguidelineintermsoftheclaimonsta9s9calmethods:–  AtypicalL,31%exists?(variesfrom8%to51%)–  Asthenumberofpar9cipantsisincreasing,isthepropor9onof

usabilityproblemsfoundincreasing?

•  Someempiricalresearchesshowthattes9ngthefirstfiveuncoveredmuchlowerpercentages(Spool:35%,Faulkner:55%)ofusabilityproblemsthan85%asNielsenclaimed.

UsabilityTes9ng:WhatHaveWeOverlooked?

•  Gi<eLindgaard&JarineeChar<ra9chart•  CUE:Compara9veUsabilityEvalua9on,tocollectdatafor

usabilitymethodsandtechniques

•  CUE‐4:17professionalteamstoevaluatewww.hotelpenn.com(9teams:usabilitytes9ng,8teams:expertreview)

UsabilityTes9ng:WhatHaveWeOverlooked?

ResearchQues9ons

•  Thereisacorrela9onbetweennumberofusersandthepropor9onofproblemsfound.

•  Thereisacorrela9onbetweennumberofusertasksandthepropor9onofproblemsfound.

Methods

•  Think‐aloudMethod

•  9teamswithdifferentnumberofusers

•  AnalyzeTasksandScenarios–  Taskgoal,e.g.,Findanavailableroom

–  Usertask,e.g.,Checkroomavailabilityofapar9cularroomtypeonacertaindate;checkroomavailabilityforthefollowingyear,…

–  Usertasktoken,e.g.,goingbacktothehomepage;makingareserva9onforafamilyofthreefromJune28toJuly5

•  Analyzetheproblemsreportedbyeachusabilitytestteam

Team A H J K L M N O S

#ofUsers 6 12 7 5 6 15 13 6 6

Results

•  NumberofUsers&UsabilityProblemsfound–  Nosignificantcorrela9on

Results

•  NumberofUsers&UsabilityProblemsfound–  Correla9onexists

r=0.731,p<0.05(n=9) r=0.821,p<0.01(n=9)

AnalysisonResults

•  Correla9onbetweenthenumberofusersandthepropor9onofproblemsfoundwasnotsupported

•  Correla9onbetweenthenumberoftasksandsthepropor9onofproblemsfoundwasupheld

•  5userclaimwasnotsupported(85%oftheproblemsfound)•  Roleofpar9cipantrecruitment(TeamA&TeamL)

Team A H J K L M N O S

#ofUsers 6 12 7 5 6 15 13 6 6

#ofusertasks 14 11 5 11 12 10 6 10 8

ProblemsFound(%) 42 43 7 22 27 29 23 24 30

%Newproblems 12 8 0 4 4 3 2 5 4

Summary

•  Whatwehaveoverlooked:Othercontribu9ngfactorstoimproveusabilitysuchastaskcoverageandpar9cipantrecruitmentratherthansamplesize–  Usabilityinves9ga9ononimportanttargetusers&themostcri9cal

tasks

•  Ifthegoalofusabilitytes9ngistogatherqualita9veinsightstoimproveproducts,oneortwoindividualobserva9onscanprovidethem

•  Nielsen’sclaimisuseless?–  Inoverall,themagicnumber5canbeappliedasoneofquan9ta9ve

assessment–  Todemys9fytheconceptthatusabilitytes9ngisverycostlyatthat

9me

References

•  Gi<eLindgaard,JarineeCha<ra9chart,Usabilitytes9ng:whathaveweoverlooked?,ProceedingsoftheSIGCHIconferenceonHumanfactorsincompu9ngsystems,April28‐May03,2007,SanJose,California,USA

•  Nielsen,J.Whyyouonlyneedtotestwith5users,JakobNielsen'sAlertbox,March19,2000,h<p://www.useit.com/alertbox/20000319.html

•  Faulkner,L.Beyondthefive‐userassump9on:Benefitsofincreasedsamplesizesinusabilitytes9ng.BehaviorResearchMethods,Instruments&Computers,35,3,PsychonomicSociety(2003),379—383

•  JaredSpool,WillSchroeder,Tes9ngwebsites:fiveusersisnowherenearenough,CHI'01extendedabstractsonHumanfactorsincompu9ngsystems,March31‐April05,2001,Sea<le,Washington