User Evaluation
Transcript of User Evaluation
![Page 1: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/1.jpg)
UserEvaluation
AlarkJoshi
![Page 2: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/2.jpg)
OpticalIllusions
Same Length
Same Color perceived differently
Different Color perceived as same
![Page 3: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/3.jpg)
MotionInducedBlindness• Fixateonthecenter– Theyellowspotsdisappearonceinawhile(sometimes1andsometimesall3)
http://www.michaelbach.de/ot/mot_mib/index.html
![Page 4: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/4.jpg)
Inattentional Blindness
http://www.youtube.com/watch?v=Ahg6qcgoay4
![Page 5: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/5.jpg)
PreattentiveProcessing
Closure
SizeOrientationShape and Color
ColorShape
Credits: http://www.csc.ncsu.edu/faculty/healey/PP/
![Page 6: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/6.jpg)
NeedforEvaluation• Visualsystemisextremelycomplex• Misrepresentationofdatacanhavecatastrophicresultsinsomecases– Challengerdisastercouldhavebeenaverted‐ Tufte
• Needtoensureconsistentinterpretationofdatausingavisualizationtechniqueinvariousscenarios
• “Userstudiesofferascientificallysoundmethodtomeasureavisualizationtechnique’sperformance”– Kosara etal.
![Page 7: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/7.jpg)
NeedforEvaluation
• Novelvisualizationtechniquesprovideauniquerepresentationofdata
• Howdoyoumeasuretheeffectivenessofthetechnique?
• Howcanwebesurethatthenewtechniqueisnotmisrepresenting/misleadingtheuser?
• Howcanweconfidentlysaythatyourvisualizationtechniqueis‘better’thanexistingtechniques?
![Page 8: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/8.jpg)
UserEvaluations
• Conductauserstudytoevaluatewhichvisualizationtechniqueis‘better’
• Auserstudyinvolvesusersperformingtasksusingavarietyofvisualizationtechniques
• Theideaistoevaluatethevisualizationtechniquesbasedontheperformanceoftheusers– Forexample,YostandNorth’06(ThePerceptualScalabilityofVisualization)
• Howdoyoumeasuretheperformance ofauser?
![Page 9: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/9.jpg)
UserStudy
• ComparePieChartsandBarGraphs• Learnaboutthecomponentsofauserstudy• Fillouttheworksheetaswegoalongwithwhatyouthinkeachcomponentis
• Weshallthenconducttheuserstudywithallofyouas‘willing’participantsattheend
• Analyzetheresultsafterclass
![Page 10: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/10.jpg)
DependentVariables
• Howdoyoumeasuretheperformance ofauser?• Performancemetrics
– UserAccuracyatperformingthetasks– Timerequiredtoperformthetasks– Emotionalresponse
• Confidencethattheusers’haveintheiranswers(whichindicatestheeffectivenessofatechnique)
• Stresscausedbyaparticulartechnique– Qualitativefeedbackintheformofaquestionnairethatletstheuserprovideadditionalfeedback
![Page 11: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/11.jpg)
ConductingaUserStudy
• Whatareyoutesting?– Efficacyofanovelvisualizationtechnique– Evaluatingpre‐existingtechniquestofindthebestoneforacertaintask
• IdentifyyourNullHypothesis– Forexample,theNullHypothesiswouldbethatthatthenovelvisualizationtechniqueisnotmoreeffectivethancurrentlyusedtechniques
![Page 12: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/12.jpg)
IndependentVariables
• Whatarethevariablesinthestudythatareindependent?
• Variablesthatwemanipulateinthestudy• Thevisualizationtechniquesthatyoushowtheuserareyourindependentvariables– E.g.YouhavethreeindependentvariablesifyouarecomparingTreemaps vs BarChartsvs PieCharts
![Page 13: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/13.jpg)
Datasets
• Thedatasetschosenforthestudyshouldbestandardandanonymous
• Mock/Testdatasetsareacceptableaslongastheydonotfavoracertaintechnique
• Ideally,thestudyshouldevaluatethevisualizationtechniqueonacoupleofreal‐worlddatasets
• Provideslegitimacytotheresults
![Page 14: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/14.jpg)
Beforeyoustartthestudy• Identifythetasksthatusers/expertswoulddoonaregularbasis– Trytokeepthetasksgeneralandreasonablyshort
• Createfair visualizationsofthedataforallthevisualizationtechniquesinvolved
• Identify/Solicitunbiasedparticipantsforthestudy– Attempttohaveabalancedgender,agepoolforthestudy
– Forexpertstudies,identifyappropriateparticipants
![Page 15: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/15.jpg)
Beforeyoustartthestudy
• Inadditiontoverballydescribingthestudyanditspurpose,haveawrittendescriptionofthereadyfortheparticipants– Providesperspectivetotheparticipants– Thisallowsparticipantstoleaveiftheyfeeluncomfortablewiththestudy
• Makesureyouhavethequestionnairereadyforuserstofilloutafterthestudyiscompleted
• Ensurethatyouhavewaystorecorddataabouttheparticipantsbeforethestudycommences
![Page 16: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/16.jpg)
EthicalConsiderations
• Sometimestestscanbedistressingandtiring• Ensurethatyouareclearlycommunicatingtotheparticipantsthat– Theycanleaveatanytime– Theyarenotpressuredtoparticipate– Thedatacollectedwillbeanonymous
• EveryuserstudyhastobeapprovedbyanInstitutionalReviewBoard
![Page 17: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/17.jpg)
CapturingUserData
• Automaticallyrecordingperformancemetrics• Requestingparticipantsto‘thinkaloud’
– Telluswhattheyaretryingtodo– Telluswhatquestionsariseastheywork– Telluswhatproblemsarearisingastheywork
• Videorecordingtheparticipantsstudy• Recordingon‐screenactivity
![Page 18: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/18.jpg)
PilotEvaluation• Beforeyoustartthestudy,itisalwaysagoodideato
– Takethestudyyourselftoidentifypotentialproblemswiththestudy
– Performapilotevaluationwith2/3participantstomakesurethetasks,visualizationsanddescriptionisnotambiguous
• Invariablyendupfindingproblemsandallowsyoutofixthembeforeyoustartthestudy
![Page 19: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/19.jpg)
Conductingtheuserstudy
1. Welcomeandintroduceyourselftotheparticipant
2. Explaintheentireprocesstotheparticipant3. Clearlyindicatetotheparticipantthats/heis
allowedtoleavethestudyatanytime4. Allowtheparticipanttoreadthewritten
descriptionandobtainconsentfromtheparticipant
5. Introducetheparticipanttothebasicsofthestudy
![Page 20: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/20.jpg)
Conductingtheuserstudy
6. Informtheparticipantthatyouwillbemeasuringmetrics(time,accuracyetc.)
7. Patientlyanswerallthequestionsthattheymayhave
8. Lettheparticipantstartthestudy9. Oncethestudyhasbeencompleted,requestthe
participantstofilloutthequestionnaire10.Thanktheparticipantforcompletingthestudy
![Page 21: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/21.jpg)
OrderingEffects
• Orderingofconditionsisavariablethatcanconfoundtheresults
• Randomization– Toensurethatthereisanorderingeffectdoesnotaffecttheoutcomeofthestudy
• Control– Toensurethatavariableisheldconstantforallcases– SameDatasetsformultiplevisualizationtechniques
![Page 22: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/22.jpg)
OrderingSubjects• Twotypesoforderingtheparticipants• Within‐Subjectsdesign
– Allparticipantsseeallvisualizationtechniques– Orderingandfatigueeffects
• Between‐Subjectsdesign– Onesetofparticipantsseesonetechniqueandperformstaskspertainingtothattechniques
– Cannotisolateeffectsduetoindividualdifferences– Requiresmoreparticipants
![Page 23: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/23.jpg)
MeasuringEffectiveness
• Performancemetricscanprovideinsight– Consistentlyaccurateresultsusingatechnique– Significantlyfasterusingatechnique– Higherconfidencewhenusingatechnique
• Donotcombinetimewith‘thinkaloud’– Talkingwillaffectspeed
• Caveats– Fasterisnotalwaysbetter– Higherconfidence/speedbutlessaccurate?
![Page 24: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/24.jpg)
PieChartsvs BarGraphs
• Completethestudybasedontheinstructionsgiven– 10mins
• Turninyourstudywhenyouaredone
![Page 25: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/25.jpg)
Quantitativeanalysis
• Identifystatisticalsignificanceforperformancemetrics
• Studentt‐testcomputesstatisticalsignificanceiftwoquantitiesarebeingcompared
• ANOVA(AnalysisofVariance)computesstatisticalsignificanceifmorethantwoquantitiesarebeingcompared
• Outputintheformofp‐value– P<0.05=statisticallysignificant– P<0.01=highlystatisticallysignificant
![Page 26: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/26.jpg)
QualitativeAnalysis
• ‘Thinkaloud’feedbackuseful• AnswerstotheQuestionnaireprovidecrucialfeedbackaboutpreferences,difficultieswithacertaintechnique,ideasforimprovementetc.
• UseFeedbackwithQuantitativeresults– Todeterminewhichtechniqueis‘better’– Makealistofpositiveandnegativeevents– Refinethetechnique/productforasecondroundofstudies
![Page 27: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/27.jpg)
ReportyourResults
• Reportyourresultsinfulldetail– Researchpaper– Marketingbrochure
• Providesufficientdetailtoallowanotherteamtoconductthesamestudy
• Resultsprovidealegitimacytoyourclaimsofamoreeffectivevisualizationtechnique/system– Example:YostandNorth,Laidlawetal.andsoon
![Page 28: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/28.jpg)
IssuesofPerceptionandCognition
• Adventofhugedisplaysandcomputingpoweroverwhelmingourperceptualsenses
• Visualizinghugemultimodaldatasetsonthesedisplaysmaycauseperceptualandcognitiveoverload
• Needtoevaluateanddesignvisualizationsthatareeffectiveandeasytounderstand
![Page 29: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/29.jpg)
Nowforsomethingcompletelydifferent!
• PixarShort– Lifted– http://www.youtube.com/watch?v=pY1_HrhwaXU
![Page 30: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/30.jpg)
UserStudies• RobertKosara,ChristopherG.Healey,VictoriaInterrante,DavidH.Laidlaw,andColinWare.UserStudies:Why,How,andWhen?.IEEEComput.Graph.Appl.23,4(July2003),20‐25.
• Increasetheawarenessoftheneedforuserevaluationinvisualization
• Discussscenarioswhereuserstudiesmaybeappropriateandsituationswhereothertechniquesmaybemoreappropriate(videotapingusersusingthesystem,etc.)
![Page 31: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/31.jpg)
Blogcomments
• Danny– “colorsequencesportionwasextremelyhelpfulandisnotasemphasizedasitshouldbeindesign”
• Peter– “thetestsubjectmustbepresentedwithacleartask”
• Tim– “biggerconcernisthatsuchuserevaluationscouldeasilyyieldmisleadingresults…andthatisfarworsethanjustresultsof“limitedvalue”.
![Page 32: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/32.jpg)
OtherEvaluationMethods
• Expertevaluation– Evaluatewiththehelpofasmallgroupofdomainexperts
• Interview‐basedevaluation(ManyEyes paper)• Multi‐dimensionalIn‐depthLong‐termCasestudies(MILC)
![Page 33: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/33.jpg)
Multi‐dimensionalIn‐depthLong‐termCasestudies(MILC)
• BenShneiderman,CatherinePlaisant,Strategiesforevaluatinginformationvisualizationtools:multi‐dimensionalin‐depthlong‐termcasestudies,BELIV'06.
• Assessinginsight,discoveryishard• Efficacycanbeassessedbydocumenting
1. usage(observations,interviews,surveys,loggingetc.)2. expertusers’successinachievingtheirgoals
![Page 34: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/34.jpg)
Blogcomments
• Peter– “ multipledimensionsandlong‐termnatureofthestudy…decreasingthelikelihoodofresultsbeinginfluencedbybeliefsystems”
• Archana – “appreciatetheauthorsforacceptingthatMILCisnot“the”strategyanditalsohassomedrawbacks”
• Tim– “Iliked theguidelineslaidoutinSection6…particularly,thoseregarding theincorporationoffeedbackfromwithinthetoolitself”
![Page 35: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/35.jpg)
ChallengeofInformationVisualizationEvaluation
• Plaisant,C.TheChallengeofInformationVisualizationEvaluation.AdvancedVisualinterfaces,Italy,2004,ACMPress.
• Discussotherevaluationapproachesthattakeintoaccountthelongexploratorynatureofuserstasks
• Addresstheproblemsassociatedwithevaluatingbenefits
![Page 36: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/36.jpg)
InformationVisualizationEvaluation
• “Discoveryisseldomaninstantaneousevent”– Observeusersoveralargerperiodinsteadofashortuserstudywithasingletool
• “Waytoanswerquestionsyoudidn’tknowyouhad”– Allowfreeexplorationofdataandreportonwhattheylearnedandunderstood(lesseffectivewithnon‐experts)
– Lettingsubjectsusetheirowndatacanincreaseengagementandinterest
![Page 37: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/37.jpg)
InformationVisualizationEvaluation
• “Factoringinthechancesofdiscoveryandthebenefitsofawareness”– Potentialadoptershavetoconsiderrisksassociatedwitherrorscausedduetovisualization
– Streamliningrepetitivetasksisalsoabenefitthatadopterslikeinplaceofnoveldiscovery
• “Successcannotbetrackedbacktovisualization”– Freire andSilvaproposetheuseofprovenancetotrackusertasksand“quantify”insight
![Page 38: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/38.jpg)
ExamplesofTechTransfer
From treemap to the Map of the Market and other treemap tools
![Page 39: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/39.jpg)
Blogcomments
• Eddie– “ needtobalancethegeneralapplicationofatoolwithrequirementstosolveproblemsinspecificdomains”
• Danny– “Buildinggenericandreusabletoolkitsshouldbeencouraged,buttheclient‐facinginterfaceshouldbeverytailoredtotheproblemtheyaretryingtosolve”
• Tim– “neveroccurredtomejusthowharditcanbetofindearlyadopters”
![Page 40: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/40.jpg)
ExpertReviews
• Tory,M.,MoellerT.EvaluatingVisualizations:DoExpertReviewsWork?IEEEComputerGraphicsandApplications,25(5),2005,8‐11.
• Askingafewfriendsfortheiropinionisn'tsufficientandmaymissvaluableinformation
• Expertreviewsidentifyusabilityproblemsandareefficient(5expertsfind75%problems)– Comparedtoastudyof50participants
![Page 41: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/41.jpg)
Comparinglightcontrolwidgets
• HeuristicevaluationthatfocusesonGUIissues,genericvisualizationtasksandtasksspecifictotheirtool
• Lightdial(rightimage)facilitatedfasterexploration• Slidersbetterforunderstandinglightcontributions
![Page 42: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/42.jpg)
Comparinglightcontrolwidgets
• Usabilityexpertshadfarmoreinsight• Oftheexperts,onlytwofocusedondataanalysistasks– Duetotheirtraininginmedicalimaging
• Involvingusabilityexpertswithdomainknowledgeismoreimportantthanjustusabilityexperts
![Page 43: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/43.jpg)
Comparingvolumerenderinginterfaces
• Expertreviewsontwovolumerenderinginterfaces
• Tableinterfacevs parallel‐coordinatesstyleinterface
• Usedusabilityexpertswithdomainknowledgeforthisstudy
![Page 44: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/44.jpg)
TableInterface
Image credits: TJ Jankun Kelly and Kwan‐Liu Ma
![Page 45: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/45.jpg)
Parallel‐CoordinatesStyleInterface
Image credits: M. Tory and T. Moeller
![Page 46: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/46.jpg)
Twotasks
1. Exploreseveraldatasets2. Searchforanidentifiableobjectusingthe
interfaces
• Expertsprovidedwrittenfeedbackalongwithratings
• Observedparticipantsandrecordedtheiropinionthroughoutthestudy– Ledtoidentificationofmisconceptions
![Page 47: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/47.jpg)
Comparingvolumerenderinginterfaces
• Tableinterface– Usefulforquickexplorationofsettings
• Parallelcoordinatesinterface– Identifyingavailabledisplayoptionsandmanipulatingdisplaysettings
• Smallimageswereseenasaproblem
![Page 48: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/48.jpg)
Summary
• Recordingandanalyzingtheobservationsishard– Explorevideorecording
• Highlyvaluableandapplicablefeedback• Expertsprovidequickandvaluableinsight• Notasubstituteforuserstudies• “Problems”foundbyexpertsmaynotaffectendusers
• Expertswithusabilityexperiencecanprovideapplicablefeedback
![Page 49: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/49.jpg)
Blogcomments
• Danny– “judgesoftwarepoorlyatfirstglanceife.g.thewidgettoolkitorwindowdecorationsappearedoldorblock”
• Tim– “providedsomeguidelinesregardingwhenitmakessensetoacquireexpertreviews(earlyon)andwhen ”
• Bill– “ …inclinedtodoanexpertreviewinpreferencetoalabstudywhiledevelopingaproduct.”
![Page 50: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/50.jpg)
SevenScenarios‐ InfoVis Evaluation
• HeidiLam,EnricoBertini,PetraIsenberg,CatherinePlaisant,Sheelagh Carpendale,EmpiricalStudiesinInformationVisualization:SevenScenarios,TVCG,December2011.
• ProvideadescriptiveviewofthevariousevaluationscenariosinInformationVisualization
• Surveyed850paperstocategorizethemintosevendistinctclasses(scenarios)
![Page 51: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/51.jpg)
Scenarios
• Identifyyouruserevaluationgoals• Provideaninsightintothekindsofquestionsthatcanbeaskedforspecificscenarios
• Eachscenariocontainsadescriptionofthesituationinwhichyoucouldusespecifiedquestionsandmethodstoevaluateresults
![Page 52: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/52.jpg)
Scenarios1. UnderstandingEnvironmentsandWorkPractices
(UWP)– MILCfallsintothiscategory– Spendtimeatthesiteoftheclienttryingto
understandthekindsoftasks,visualizationsandbarriersbeingused
2. EvaluatingVisualDataAnalysisandReasoning(VDAR)
– Studythedataanalysisorientedtasks(hypothesisgeneration,knowledgediscovery,decisionmaking)
![Page 53: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/53.jpg)
Scenarios3. EvaluatingCommunicationthroughVisualization
(CTV)– Conveyinginformationtoalargegroupofpeople– Aimtoevaluateefficacy,usefulnessandinteraction
patterns4. EvaluatingCollaborativeDataAnalysis(CDA)
– Evaluatetheabilitytofacilitateseamlesscollaboration5. EvaluatingUserPerformance(UP)
– Identifylimitsofhumanperception,comparedvisualizationtechniquesbyexaminingperformance
![Page 54: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/54.jpg)
Scenarios
6. EvaluatingUserExperience(UE)– Usabilitytestingintermsofusefulfeatures,
missingfeatures,improvements,learningcurve7. EvaluatingVisualizationAlgorithms(VA)
– Evaluatetheefficacyofalgorithmsinbeingabletohighlightpatterns,produceleastclutteredview,abilitytoscale(withlargedisplays,hugedatasets)
– Examinetheperformanceintermsofefficiencyofanalgorithm
![Page 55: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/55.jpg)
Many‐to‐manymapping
• Scenariosdon’tmakedirectlytoasingleevaluationmethod
• Situationswhereyourevaluationgoalsmaybebroadmayincludemultiplescenarios– ExploringdataanalystsprocessofknowledgediscoverymayincludeUWP,VDAR,UE,VAandsoon
![Page 56: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/56.jpg)
Blogcomments
• Peter– “ …importanceofeffectiveevaluation,andbycontrast,theproblemscreatedbyineffectiveevaluation.”
• Danny– “Thepaperdoesdoagoodjobindescribingthederivedcategoriesandexplainshowtheyaredifferent.”
• Eddie– “providesahandyframeworkwithinwhichtocategorizeevaluationscenariostoaidintheselectionof theproper evaluationmethod”
![Page 57: User Evaluation](https://reader031.fdocuments.net/reader031/viewer/2022013005/61cd27a3aa930d26f832e16d/html5/thumbnails/57.jpg)
Userstudy:2DVectorVisualization
• D.H.Laidlaw,M.Kirby,C.Jackson,J.S.Davidson,T.Miller,M.DaSilva,W.Warren,andM.Tarr (2005).Comparing2Dvectorfieldvisualizationmethods:Auserstudy.TVCG,11(1):59‐70,2005.
• Conductedathoroughevaluationofexpertandnon‐expertusersforvisualizationtechniques
• Comparedsixvisualizationmethodsfor2Dvectorvisualization
• PresentationbyTimManess