Lecture 3 - User centered designtlatoza/teaching/swe632s18... · User-Centered Design SWE 632 ......
Transcript of Lecture 3 - User centered designtlatoza/teaching/swe632s18... · User-Centered Design SWE 632 ......
User-Centered Design
SWE 632Spring 2018
© Thomas LaToza
Looking ahead• Examined human cognition
• Have 2 ways to identify usability issues
• But... is HCI just identifying usability issues?
• What does design mean?
• How do we learn about user needs?
• How do we build designs?
• How do we evaluate designs?2
User-centered design
3
User-centered design
4
What problems may users
encounter w/ current ways of
doing things?
Who are the users?
What are the user’s needs?
What are the user’s tasks and
goals?
What extreme cases may
exist?
How does the product fit into the broader context of their lives?
Technology-centered design
5
What can this
technology do?
How might users use it?
What features
does it have?
Double diamond model of design• Question problem,
expand scope, discover fundamental issues
• Converge on problem
• Expand possible solutions
• Converge on solution
Fail fast• “Fail frequently, fail fast” David Kelley, founder of Ideo
• Failure is learning experience
• Crucial to understand correct problem to solve & ensure solution is appropriate
• Abstract requirements are invariably wrong
• Requirements produced by asking people what they want are wrong
7
Iterative model of design
8
(re) Define the problem
Needfinding
Brainstormideate
understand the users
Prototype
build
Test
Iteration• Repeated study and testing
• Use tests to determine what is working or not working
• Determine what the problem might be, redefining the problem
• Collect more data
• Generate new alternatives9
Flexibility-usability tradeoff
11
Flexibility-usability tradeoff• Jack of all trades, master of none
• Better understanding needs enables specialization and optimization for common cases
• System evolution over time:
• flexibility —> specialization12
Navigating a design space
13
• What are key decisions in interaction design
• What alternatives are possible
• What are tradeoffs between these alternatives
Hierarchy of design decisions• What are you (re)designing?
• the width of the text input
• the maximum length of a valid username
• when in the signup process users enter their username
• if the user must create a username when signing up
• whether users are anonymous or have a login
• if users can interact with other users in your application
14
Picking the right level of redesign• Where are the user’s pain points
• What are the underlying causes
• What would be the value to the user of addressing issue
• What do you have time to build (or change)15
Activities and tasks• Activity - set of tasks performed together for a common goal
• Go shopping
• Task - component of an activity, organized cohesive set of operations towards a single low-level goal
• Drive to market
• Find shopping basket
• Find item in store
• Pay for items16
Activities and tasks• Activities are hierarchical
• High-level activities spawn other activities, spawn tasks
• Software supports tasks and activities
• Important to design for activities, not just tasks
• Support whole activity seamlessly
• Ensure interactions between tasks do not interfere17
Example - iPod• Supports entire activity of listening to music
• discovering music
• purchasing music
• getting it into music player
• developing playlists
• sharing playlists
• listening to music
• ecosystem of external speakers and accessories
18
Example of a design process• How do you get from let's make listening to music better to designing
an iPod??
• Iterative design...
• But what does that actually look like more concretely?
• What insights into activity help inspire design?
• How does watching users help lead to these insights?
• How do insights translate into an actual real design?
• How do know the new design is actually better?19
Example
20
Domain: Debugging• Design goal: how do we better support activity of debugging in large,
complex codebases?
• Build a better debugging tool (?)
• What should it do? How would it help?
• Design a better watch window?? Support new types of breakpoints?
• What's really the key steps in debugging that lead users to struggle the most?
21
Observations of developers in the field
22
Par$cipants Tasks~90minutes
pickedoneoftheirowncodingtasksinvolvingunfamiliarcode17professionaldevelopers
Transcripts
(386pages)
InteresAng.Thislookslike,thislookslikethecodeisapproximatelythesamebutit’srefactored.Buttheothercodeis.
Changedwhatflagsit’s???
HeaddedanewflagthatIdon’tcareabout.Hejustrenamedacouplethings.
Well.
Sothechangeseemedtohavechangedsomeofthewaythesethingsareregistered,
butIdidn’tseeanythingthattalkedatallaboutwhethertheappisrunningorwhethertheappisbooted.Soitseemslike,thiswasuselesstome.
(annotatedwithobservernotesaboutgoalsandacAons)
Ac$vi$es
Coding activities working with unfamiliar code
23
Circlesize:%ofAme Edgethickness:%oftransiAonsobserved
ReproduceDebug
Investigate
Test6%
33%
28%
4%
11%
16% 5%Edit
Reuse
Compile50%
50%28%
40%
12%
20%
11%86%
3%
22%
67%11%
14%
22%
20%
18%
29%
11%86%3%
55%32%
5%6%
Longest activities related to control flow questions
24
4outofthe5longestinves$ga$onac$vi$es
5outofthe5longestdebuggingac$vi$es
Primary question Time (mins) Related control flow question
How is this data structure being mutated in this code? 83 Search downstream for writes to data
structure“Where [is] the code assuming that the tables are already there?” 53 Compare behaviors when tables are or are
not loadedHow [does] application state change when m is called denoting startup completion? 50 Find field writes caused by m
“Is [there] another reason why status could be non-zero?” 11
Find statements through which values flow into status
Where is method m generating an error? 66 Search downstream from m for error textWhat resources are being acquired to cause this deadlock? 51 Search downstream for acquire method
calls“When they have this attribute, they must use it somewhere to generate the content, so where is it?”
35 Search downstream for reads of attribute
“What [is] the test doing which is different from what my app is doing?” 30 Compare test traces to app traces
How are these thread pools interacting? 19 Search downstream for calls into thread pools
Longest debugging activity
25
RapidlyfoundmethodmimplemenAngcommand Unsurewhereitgeneratederror
StaAcallytraversedcallslookingforsomethingthatwouldgenerateerror
Trieddebugger
Didstringsearchforerror,foundit,butmanycallers
Steppedindebuggertofindsomethingrelevant
StaAcallytraversedcallstoexplore
WentbacktosteppingdebuggertoinspectvaluesFoundtheanswer
(66minutes)
Whereismethodmgenera$nganerror?
debugger
staAccalltraversal
grep
debugger
staAccalltraversal
debugger
3Sowe’llgothereandwe’lljustcrawlthroughthiscodeandwe’lltrytounderstandthat.Sothiscodehassomeotheroptionsinit.So,I’mjustscanningthroughtojustunderstandwhatthisisdoing.Typicallythesefunctionslookforsubcommandsforthemaincommand.Souhas[lookingcasestatementlookingatcharacterenteredbytheusertodispatchonwhatcommandtoexecute]onefunctionality.Andubhasanother,andufhasanother.Sothat’swhatthiscodeisactuallydoing,henceparse.4Andtheguythatwrotemostofthiscodewasprettyconsistentwithhiscodepatternsforhowhedoesstuff.So,againthefunctionnamesareidicativeofwhatisgoingon.AndhemakesheavyuseofElagspassingaround.Somoreprecisely,whatI’mlookingfor,I’mlookingforwhoisactuallyreturningthismemoryaccessstring.SoIdon’tseeanythingjustscrollingthroughthisfunction,clearlyit’snotthisfunction,butthisfunctioncallsabunchofotherfunctions,soIcouldwalkthroughallofthecallstotrytoisolatethat,butI’mgoingtoseeEirstifIcangetluckyandnarrowitdownfromtheotherendandlookatwheretheoutputiscomingfrom.5Searchingtheentireprojectandwe’rejustgoingtodoastringsearchforalloftheprojectandseeifthatcomesupwithany,withbasicallywherethatoutputcomesfrom.[doingsourceinsightsearch]So,luckilythisdoesn’tseemtobeapieceofoutputthatgetsspewedeverywherewhichisnice,butitseemsprettysparse.Andinfactsearchingtheprojectdidn’tactuallyEindthatatall.SoI’mreallynotgoingtobeabletoworkbackwardsfromtheerrorstring.6Sowe’llgobacktothesourceEileitself,sowe’llgobacktotheoriginalparseEile,sowehavenooptions.Nomodes,sowe’reinthedefaultmode,sowe’regoingintothisinstruction,thisisjustElags.I’mjusttryingtogetafeelfortheparametersthatwearedraggingalonghere.Idon’tknowifyouwanteditorialcommentoncodeorcodetoolswhileI’mwalkingthroughthis.“Ah,whatever,it’smostlyjustwhateveryou’rethinkingaboutwhileyourworkingonthetask,ifthat’swhatyou’rethinkingaboutthat’sEine.”7[laughing]Yeah,itwouldbeniceiflookingatthisfunctionontheparametersthemselveswereoverlaidwiththetype.Sothishasamouseoverthatissomething,butitdoesn’tactuallytellmewhatthetypeis.Soagainlookingatthefunctiondeclarationagain???SowhatIreallywanttoknowiswhichoneistheaddressthatI’mactuallygoingtodisassembleandbeonthat.SoSorightnowI’mmostlyjustreadingthecodeandtryingtounderstandstuff.ButafewthingsIdonoticieisalotofthelackoftheinitialization.SosomeofthatImightchangeisIultimatelydoeditthisfunction.[lookingatthemethodthatiscalledfromdispatchandthatdoestheactualworkforthesubcommandofinterest]8Justbecauseit’sannoying.Soscanningthorughhere,I’mjustlookingforthecallsareandwherewegonext,orwheretheoutputis,becauseagainI’minterestedinwhoisputtingthatoutputinthere.Sohereisthisfunctioncall,machinedisassemble.Here’saplacewheresourceinsightfallsshort,it’sshowingmethewrongpreviewforthedissasmeblefunction.[littlepreviewwindowinbottomofsourceinsightwindowforcallee]Iknowthatbecausethisisamemberfunctionandthisoneisnot.Thisisthewrongnumberofparameters,blah,blah,blah.Soagainwehavetogobacktobrowsetheprojectsymbolsforthedisassemblefunction.9Soagain,lotsofdifferentones,butIhavewhichobjectthisis.SoifIgobackhere,wearelookingatthemachineisourobjecthere,anditisamachineinfostruct,sowecangobackhereto[Eiguringouttypeofobjecttoreasonaboutdynamicdispatchformanuallygoingtocallee]gotothemachineinfoversionofdisassemble.Sothisisinterestingbecause,nowwe’reoutsideofwhatthedebuggeritselfisdoingandwe’renowinthedebuggerAPIs.Sothatmakestheriskofachangehigher.There’smoreofaregressionrisk,becauseit’snotjustthedebuggerthat’susingit,10it’sallthedebuggerincludingonesthatarenotours.SoI’mjustscanningthrough,typicallylookingatstufflikethisI’mjustinterestedinhowbigthefunctionis,howmanydifferentbranchesitcouldtake,howcomplexitisgoingtobetoEindoutwherewearegoing,justfromreadingthecode.OfcourseIcanattachthedebuggertothedebuggerandwalkthroughthatwhichisprobablywhatI’mgoingtodohereinaminute.11Yeah,sothiswillbealittleeasiertounderstandifIactuallywalkthroughthecode.SoI’lljustopenanotherdebuggersessionandattachittothisEirstone.[startsasecondwindbg]Sowestartedwithparseunassemble,whichisgoingtobeindbgeng.12Mighthavetolineupsymbolsforthis.Sowe’llwaitonthat,we’llgobackhere.[tosourceinsight]Decode.SojustscrollingthroughthefunctionandlookingattheEirstactualfunctioncallthatwewillmakegoestothisdecodefunction.AndagainI’mjustlookingforwherethatoutputcomesfromorifwewouldsetadifferentsetofbrnaches.Thecommentsare,thisisnicelycommentedcodewhichisraretosaytheleast,soit’sactuallyalittleeasiertotryandthrowoutpiecesofcodethatareprobablynotrelatedtowhatI’mlookingfor.BecauseIhavesomeinnateknowledgeastowhatI’mlookingfor,andthiserror[againthinkingaboutreadingsourcecodeasaEiltering/searchtask]isactuallymostlikelycomingfrom,we’rereadinganaddressthatisnotinthedumpEile.SoI’mlookingspeciEicallyforreadmemoryorreadpointerorstufflikethat.[he’sright–itdoesendupbeingfromoneofthesecalls]Butsincethisentirecodebasecalls,soIknowthatthatisgoingtobesomethinglikereadptrorreadvirtual,butIalsoknowthatthere’sabazillioncallstothatfunction,andit’snotveryeasytonarrowitdownthatway,soIcan’tgoaboutitthatway.[wantsdostringsearchofcalleetreeidentiEiers]14Sothedebuggeroverherecameback,sonowIcangogetsymbolsforthisversion.SoIcanpickthesymbolpathinthedebuggersothatIcanwalkthroughthecode,andagainwewaitalittlebitsowe’llgobackoverhere.[tosourceinsight]15Thispartofthecodeisactuallytakingaparttheinstructions,sobythispointwealreadyhavethedata,sothereaddatawouldhavealreadyoccurred,andwewouldhavefailedbythepointthatwegottothiscode.Sowe’llgobackhere,weneedtogobacktothewritedisassemble,Ibelievethisistherightone.[sourceinsightsymbolbrowserforit]16Sowe’llassumethatdecodefailed,butifwedothat,ifitfailstotallythenwewouldjustexit,whichdoesn’tseemtobewhat’shappening.Becauseotherwisethisfunctionwouldn’thavethistextoutputthatwe’reinterestedin.17SotheotherthingsthatInoticedwhenIwaslookingatthedeEintionsforunsassemblewhenwecalleddisassemble,there’sabunchofmachinespeciEicimplementationsofdisassemble.Soitcouldbethatwe’renotactuallycallingthemachineinfo,therecouldbeanx86onethatweareactuallycallingsincethisisdebuggingx86code.Somyia64version,whichapparentlyIdon’thavecodeforormaybeitwasremovedfromtheproject,samething,sothere’sclearly,sotheremightbesomethingwrongwithmyprojectwhichiswhythere’ssomanydeEinitionsfforthis.18Ok,sothedebuggeroverherecamebacksoIcanjustsetabreakpointonparseunassembleandthenwalkbackthroughthecode,ohweactuallydon’tthatonebececausethatoneisgoingtosucceed,wewantthefailurecasewhichisthisone.[demonstratedsomebehaviorandgotacallintoittwice]Ok,soweareatparseunassembled,sowe’llmakethedebuggerlookatthesamesourcecodethatwearelookingatinsourceinsightoverhere.Andthedebuggershould,ifitcanEindthecode,maybeitdoesn’tlikethiscodepath.ThatwilldeEinitelymakeithardertowalkthroughthecode.19Sowe’llgobackintodisassemblehere,sincethere’snotreallyabetterimplementationthatisabletodoit,we’llgobacktothemachineinfoone.Whatwouldreallybehelpfulhereistoknowwhatcodepathsaremostcommon,likethemetadatathatpreEixprovides,orsometracingtools.Ifthatwassomehowoverlaidwiththesourcecode,thenyoucouldseewhatcode20wasdeadeffectively,orwhatcodegetsrunincertainenvironments,we’lljustputthatinthepipedreampile.“Soyoujustwanttoseewhat’salwaysexecuted?”Soitwouldbenicetosee,solikepreEixonlydoesasetnumberofpaths,butlikeBenLiblithasaproject,you’refamiliarwithhim?“He’sfromwisonsin”Yeah,he’saresearcherfromWisconsin,hisstatisticaldebuggingishisthing,andhehasallthistracingstuffthatcomesupandbackandforth.Sothat,thethingaboutlookingatfailuredata,becausewehavefailuredatatoo,wecanseewhatcodepathexecuteswhenthingsfault,whatcodeexecutescommonlywhenstuffworks,soifwehadsomewaytosayinthesourcecode,becauseIcandoitfromthedebugger,21butIhadsomewaytosayinthesourcecode,ok,ifIgiveyouthesevalues,whatpathswillexecute.WhichIguessiseffectivelydebuggingthecode.“Soyou’dwanttospecifythosevaluesatfunctionentryratherthanjustrandomlyendupwiththevaluesfromplayingwiththeUI?”IthinkwhatI’msayingisthat,givenafunctiondeEintion,IEillinasetofvalues,sowhathappensifthisguyisnull,andthisguyisalsonull,[writingassertsonparams]“Makeabunchofassertsessentially”Yeah,itwouldbasicallyhighlightinthecodewhichpathsaregoingtoexecute,somethinglikethat.“Whatwouldyouusethatinformationfor,howwouldthatchangehowyouarelookingatthismethod,itwouldhelpyouruleoutpieces?”22Yeah,itwouldhelpmeruleoutwhichpathsweregoingtoexecute,socommonlywhenI’mlookingatcode,eithercodethatI’mfamiliarwithintheprojectthatIworkedoncommonlyorbecausemyjobispartiallytodebugeverybodyelse’scode,soalotofthetimeIhavecrashdumpsthatsaywhatthestateatthetimeofthefailurewas,andIhavethesourcecode,butIhavetodoalotofeitherqualiEicationofvaluesinthedebuggeritselforabunchofguessingwhateverinmyheadtotryandEigureoutwhichpaths,becausewe’relookingatastaticpointoftimeinthedebguggerandastaticpieceofcode.Andthe2won’t,youcanoverlythetwo,butyouwon’tnecessarilyknowwhichpathsexecuted,soyouhavetokindofwalkthroughbackwards.So,butIdohave,ingeneralIdohavetheparameters,thisisnull,thisisnotnull,thisisthisstaticvalue,thisisstaticvalue.23SoifIcouldoverlaywiththesource,sothatmight,forsomeofthesesigniEicantlylongerfunctions,itwouldhelpmeunderstandwhat’sgoingonthere.TheotherthingthatIdoalotwhenIlookatcodethatIown,I’mtypicallylookingforplacesthat,thisisforstuffthatImuchmorefamiliarwith,I’malwaysinterestedinwhatsortofthingscouldberefactored.WhereIcouldImakeafunctionsmaller,wherecouldIreducethenumberofparameters.Sohavingarefactoringmodeinthesourceeditorwouldbehelpful.Slickedithassomeinterestingthingswhereyoucanhighlightasectionofcodeandslickeditwill,ifyou’regoingtorefactorthis,thenyoualsoneedtodragalongtheselocalsandtheseparameters,andtheyhavetobepassed,anditmakesyourfunctiondeEinitionforyou.24Sothat’sveryinteresting.So,anyway,we’rebacktothis.Itdoesn’tlikemysourcepath,ohbecauseI’mgivingitthewrongone.[stilltryingtoloadsymbolsindebugger]25[waitingonittotrytoloadsymbolsagain,backtosourceinsight]Ok,againthecommentsarehelpful,becauseIcanbasicallyignorethisbranchbecauseI’mprettysurethatthedecoderdidn’tfailandIdon’tseethisoutput.[reasoningaboutwhatbranchesweretakenbasedonoutputbehavior]Butthisisinterestingtoseethisoutputinthecontextofthat,Iwaslookingforapieceofoutput,becausethisoutputissplitacross2sourcelinesasifsomeonehadasigniEicantlymorenarrowsourceeditorview.SothatmightmeanthatoneofthereasonsthatIcouldn’tEindthestringIwaslookingforbeforewasbecauseitwaswrapped.SomaybeifIgobacktomysearch,Iwassearchingfortheentirestring“memory26spaceaccessspaceerror”somaybeifIjustmakeitmemoryaccessandletitsearchalong,andthatEindssigniEicantlymoreentires,includingoneinutilsdotcpp,inatableoferrorstrings.[goestothatreference]Sowhereverthatguywas,thereyougo.Sothisislikeaninterestingsearchproblemingeneral.Actually,Idon’tthinkgoogleorlivesearchdothis,butifyougiveasetof4individualsearchterms,usuallyyougetallornothingfromasearchengine.Soyougetthesetofresultsthatgetall4terms,orinthiscaseall3terms,ornoterms.27Butyoudon’ttypicallygetatreedsetofterms,herearethesetofresultsthathaveallofyourterms,hereisthesetofresultsthathaveallminus1,allminus2,allthewaydownto0.Butinthatcase,thiswouldhavebeenveryhelpful,thiswouldhavepotentiallysavedmeagoodbitoftime.SoI’mlookingforacalltoerrorstringwiththeerrorvaluememory.[wantsthecallertothismethodwithaparticularparameter–theenumthatforcesthecasewhereitprintsthestring]Sowecanseehowmanycallerstherearehere,ok,sothere’saprettylargenumberofcallersofthis.Maybewecanlookatwherethosecallersareandnarrowthatdownbasedonwhatweknow.Sotherearealotintypeddata,alotinsystem.28SpeciEicallywe’relookingforcallingerrorstringwiththeEirstparameterofmemory,butthisisanothercasewheresearchgenerallyfailsingeneralbecauseofspacing.Sothisiserrorstringopenparen,andthenthewordinallcapsmemory[(MEMORY)].Butthere’sallsortsofpermutationsofhowthatcouldbespacedandstillbelegitamitecompilablecode,sowe’llstartwiththisoneandseeifwegetanything.Whichwedon’t.Sowe’llgobackhere29alotofspaces,errorspace,openparenspace,andthewordmemory,andwe’llsearchforthat.[stillnothing]SoI’mdonetryingtodothat.Solet’slookatcallersoferrorstring.[backtootherstrategyoflookingthroughcallers]Somaybeifwejustparsethroughhere,orstepthroughhere,wecanseewhichonesarecallingwiththeparameterofmemory.Butunfortunately,manyofthesearecallingwiththeEirstparameterasavariable.Sothatwouldmeanthatwhatwewerelookingatbeforeisnotasearchproblem,it’savariableinterpretationproblem.30So,I’mjustkindofsteppingthroughthesevalues,andinmyhead,I’mjusttryingtorememberwhichonesarelegitimateandwhichonesmightnotbe.SoitwouldbeniceifIcouldjusttakethiswholelistofresultvaluesandselectthemalloutofthiscombobox,andthenpastethemintonotepad,soIcouldthenremovethemfrommylist.SoIwouldn’thavetojustworryaboutrememberingthem.31Ithinkthat’ssomethingthatItendtodoalotwhendebuggingaswellasreadingcode,isthatIendupwithlotsofclipboarditems,butnotclipboardinthesenseofyou’resharingtextbetweenapplications,butclipboardinthesenseofthesearelittlehintsonwhichpathsIwentdownandwhichpathsIdidn’t.“Soyouwanttomakesureyou’renotrepeatingpaths,andthatyou’repursuingallthepathsthatyoumightwanttoreasonablypursue?”AndmorewhatIwasthinkingatthetimewhenIomittedapathorconsideredapath.SosometimeswhenIamactuallyeditingthecode,Iwillgothroughanoutputnotlikelytobethepathbecauseofthis,andthenalotofthosecommentsIwouldthenclipoutbeforethecodegetssubmittedbecausetheyaremostlyjustcodereviewercomments.Andtypically,that’ssomethingthatweseeincollaborativeworddocs.32It’sprettytypicalthatyou’llcollaborateonaworddoc,andpeoplewillputcommentsinlinewithstuff,butit’salittlelesstypicalforsourcecode,sourcecodecommentstendtobemissingintotalorthecommentbytheactualdeveloperorthemaintainer.There’snotreallyaplaceforcommentsforreaders.Thismayhavebeenperfectlyclearforthedeveloperwhowroteit,thesourcecodemaintainermightunderstandit,butthethousandsofotherpeoplewhoaregoingtoreaditfordebugging,forcustomers,forthedevelopersthemselves,there’sreallynoplaceforthemtoputcomments,andmaybethereshouldbe.“Whatstopspeoplelikefromjustcheckingthecommentintothesourcedepot?There’sjusttoomuchoverheadandyoudon’t’havetheauthoritytodothat,oryoudon’townthatcode?”33Ithinkit’snotnecessarilyauthority,it’srespectforone.Becausethisissomebodyelse’scode,sounlessyouaregoingtomakeanetpositivechange,Iwouldn’teffectapieceofcode.AndIwouldn’tconsidercommentstobeanetpositivechange,althoughmaybeIshould.Usuallyit’snotpermissions,itsusuallythischangedoesn’tneedtobepersisted.Orinmyopinion,itdoesn’tneedtobepersisted.Youcouldendupwithaveryinterestingsourcetreeifyoujustopenedituptoadhoccommentsbyanyreviewer.34Firstyou’dhavetoassumethatpeopledoit.Butthenyougetintoatoolsuchasacompilerthatwilldecideifyouarecorrectornot.Soifyouputacommentintherethatsaysthatthisisgoingtodothisinthiscaseandyou’rewrong,there’snocheckingforthat.“Youneedsomesortofauthorityaboutwhothispersoniswhoismakingthecomment”Maybethatwouldhelp.Ormaybe,justlikeyoudoforchanges,thatchangeiseffectivelyasuggestionthatyousendforreviewtothecodeowner.Maybewedothatwithcommentsto.Butthenthatputsanadditionalconstraintonthedeveloperswhichisalreadyanoverusedresource,orabusyresourceifwe’reexpectingdeveloperstodothis.So,35noneoftheseareclearlywhat’snotgoingonhere.Alotoftheseareclearlyunrelated,butverygeneric.Sothiserrorhereisbeingusedbyalotofverygenericfunctions.Which,Iguessisprettytypicalforc++code.Iguessanyobjectbasedcode,becauseyouseeitinC#andseeitinjaveaswell,soyouhavelotsofclassesandmembersthatoverriding???,soyouendupwithlotsofwrappers,aroundwrappers,aroundotherwrappers.Andit’sapuzzleinitselftoEigureoutwhichoneisactuallyofinterest.36Sowe’llgobacktothedebuggertoseeifIcanactuallywalkthroughsomeofthiscode.SoitlookslikeI’mnotactuallygoingtogetsourceinhere,whichisunfortunate.Sowe’lljusthaveto,we’llhavetokindofguesswhatisgoingonhere.[lookingthroughassemblytoseefunctioncallsamidstlotsofotherassembly]SosomeofthesefunctionsIdon’trememberseeingintheparsefunctionwherewestarted,soI’mgoingtogolookforthose.37Sothere’sgetrange.SotheotherthingthatIknowaboutthisisthatI’mrelativelycertainthattheoutputwillcomefromoneoftwofunctions.Sowe’llsetabreakpointonthose,andwe’llseewhothecallerofthoseis,butclearlyI’mwrong,becausewedidn’tcalleitherofthosefunctions,sowe’regoingthroughsomeotheroutputroutine.38Soagain,backtotheoriginalfunction,let’swalkthroughthisalittlemorecarefully.[backtosourceinsight]Sothat’sgoingtobetrue,thatisfalse,thatisfalse,soisthat,aswellasthat.[pickingguardsandpathstofollow]That’swherethatannoyingerrorcomesfrom.39Thatoneisfalseandsoisthisone.Sowe’rejustgoingtogoinhere,sotheEirstonethatwe’lldoiscallgetrange,andset???tofalse,sonow.Somaybewe’renotmakingitasfarastheunassembleditself,maybewe’restuckinthisgetrangefunction.Sohere’sanothercasewheretheeditorisprovidingmewhatitthinkcouldbetheimplementationthatI’mcalling,butI’msecondguessingthatbasedonexperience.Sowe’lllookforotherones.[browsinglistofsourceinsightsymbolsforthatmethod]Butthat’sprobablythecorrectone,solet’sgointogetrangeandwalk.40Ok,sothisonegoes,skipsthatnextcharacter,andthat’snotgoingtodoanythinginthere[reasoningaboutcall–won’thavethiseffectinit–basedonidentiferanddomainknowledge]andthenweregoingtocallget???addressmostlikely.No,we’regoingtocallevaluateaddressandherethesourcecodeeditordoesn’tgivemeapreviewofwhichoneitis.Sowe’llseeifweevenhavethisatall,sonowwehavetoguesswhichoftheseimplementationsweareactuallygoingtocall.41Thec++oneorthemasmone,ortheseareallprototypes.Butitmaybethatweonlywantthisone.Sowe’llgointothatoneandseeifwesee.Anotherfucnitoncallingafunctionwherewedon’tseethisatall.Nowthesourcecodeeditoristellingmethatthisisamacro,butitdoesn’tseemtobeabletoEindthemacrodeEinition.42So,we’lljustignorethatforthetimebeing.Here’sanotheronethatlookslikeamacro,butitdoesn’tseemtolikethat,sowe’llgooverhereandsearchforitinadifferentway,becausethatEinallyEinished.SonowI’minarazzlewindowtryingtosearchforthesamething,sincemysourcecodeeditorisfailing,I‘llgolook,I’llgolower.[Eixedatypoinsearch]43Wellthat’siterating,it’snotEindinganything.IalwaysEindmacrosverydifEiculttodealwithandI’munsureifthatisjustgeneralunfamiliarityorthelackofgoodtoolsorsomethingelseIdon’tknow.Butitalwaysseemthatwhenlookingatcodeinasourcecodeeditor,itwouldbehelpfultohaveaninstancemacroexpanderinline,soitwouldjustshowyouwhat44isactuallygoingtohappen.Becausemacrosarethereforthecodewriter,andmakesitdifEicultforeverythingelse.Sowhilethisissearchingandwe’respendingawholelotoftimewalkingthroughthiscodewithminimalresults.45Sowe’llgobackheretothedebugger.Andwhatarewelookingfor,we’relookingforthatfunction,whateverthatfunctionwas.Yeah,probablygetaddressexpression.So,we’llsetabreakpointthere,andsureenoughwecallthatguy.SoinsomewaysthisischeatingbecauseIcanbasicallyreprowhatI’mdoing.[canreproandstepthroughcoderatherthanjustgettingstaticsnapshot]46Butdebuggingpostmortemfailures,Idon’tgetachancetodothat,sothat’swhyIhavethisdichotomyofwhatthetoolsshoulddo,becauseontheonehandI’mreadingthecodeandmaybeeditingit,andontheotherhandIspendasmuchtimeifnotmorereadingcodeandtryingtounderstandwhatit’sdoingwithoutnecessarilyhavingtointeractwithitatall.Everytingveryslowtoday.[stillwaitingonEilesystemsearchanddebugger]SoI’llswitchoverhereanddoatotallyunrelatedtaskwhileI’mwaiting.47Ok,sothedebuggercamebacksoIcanwalkthorughhere.Well,it’sgottobethiscallhere.Yep.Sohere’swhatIwaslokingforinthesourcecodebefore,andIdidn’thaveanywaytotellotherthanwalkingthrough???,buthere,walkingthroughthedebugger,48Icanclearlyseethattheimplementation,thisparticularevalholderlocalvariable,whattypeitis.SoIcango,basicallygrabthisandgobacktothecodeandEindthemasmevalexpressionandlookforevaladdressinthere.But,again,myhandydandycodeeditorisnotveryhelpfultoday.Wait,doIknow?Ok.Thesymbolssaythisshouldbeineemasm,andthere’seemasm,butitsnotthere.Sothat’sodd,maybethere’ssomethingwrongwiththeproject,so.50[tryingtoaddEiletorpoject,butnotinEilesystem]So,noeemasm.So,clearlythatEileisnothere.Well,that’sbecauseit’snotthere.Myclientviewismessedup.51[editssdclientconEigEile]Oh,becausewhenIchangedthatbeforeIbrokeit.SoindoingsomethingelsetotallydifferenttodayIbasicallydeletedabunchofsourceEilesfrommymachine.So,we’llhavetoEixthat.[addingseveralextradirectorylinestohissdclientconEigEiletorestoreittonormalstate]52AndmagicallyalltheEileswillreappear,whichprobablyexplainswhymysourcedebuggingwasn’tworking.[doessdsynctogetalltheEileshedidn’thavebefore][wasn’tgettingsymbolsindebugger]53Ormaybenot,butalas,maybeIhavetheEileI’mlookingfornow,andlowandbeholdthereitis.[sourceinsightscansEilesystemandupdatesindex,andnowcannavigatetothatEile]Sowhatarewelookingfor,evaladdress.54SoifIwouldhavetoeditthisEile,IwouldprobablyEixsomeofthespacingissues,thesetabsnotreplacedwithspacesetc.ButI’malsomissingsomesourceEile,let’sseeifsearchingforeitherofthosethingsIcouldn’tEindbeforeactuallyyieldsanythingnow.[repeatssourceinsightsearch,stillnothing][illustratesreturningtoanoldpathwheninformationaboutpathschanges]Sowherearewe,we’reinevaladdress.55[skimmingthorughablockofseveralmethodsinsourceinsight]Sothisfunctioncallsthisfunction.Pushcontext,popcontext,compute???address,andwhatdoesitdo?56[goestocallee]Solet’sjusttestthis.Soweactuallymakeitbacktogetaddressexpression.Andwe’reingetaddressexpression.Ok,soitseemslikewearegoingtoreturnfromthisguywithoutdoinganything.Sonowwe’rebacktogetrange.57Andthat’swherewecallpeekchar,wedon’t.Andacasewheretheparameters,there’snocommentshere,buttheparametersarenamedwell,soIcanmakesomeassumptionsbasedonthem.Soifnothas???addressexpression,addressok,sowegodownthere.[goingbackandforthbetweendebuggerandsourceinsight–inspectingcodeinsourceinsight,lokingatvaluesindebugger]butwedocallgetaddressexpressionagain.Solet’s58we’llgo,butwedidn’tactuallycall,sowe’rebacktothatreset.Soweonlycallgetaddressexpressiononce,andthatcallisfromgetrange,solet’ssee.Sothere’s1,2.Sowe’renotgoingtodoanyofthisif.[stillreasoningaboutwhatpathstofollowinsourceinsight]59We’rejustgoingtoreturn?Soparseandassemble.Sowedoreturnfromgetlength,ohhaslength,sowearegoingtogointothatunassembledinstruction.We’llcheckthat,andtherewego,wegointotheunassembledinstruction.1:00[checkingindubggerwhichmethodgetssteppedinto,thenbacktosourceinsighttoreadit]Sohere’swherewecreatethat,???1:01Address,ok,calls,ok.Sowe’llassumethatit’srelatedtothiscall,yep.Sowecallmachinecodeassemble??ok1:02SoIwasrightintermsofreadingthecode.It’snicetohavethistoconEirm.Soitseemsthatweactuallymakeittothisdecodecall.“You’rebasicallyusingthedebuggertostep”Morelikejumparound.I’mnotreallysteppingthroughthecodebecausealotofthesefunctionsareprettylong.SoI’mbasicallyjustpickingapoint,readingcode,pickingapoint,andrunningtothatpointtomakesureweactuallygotthere,sojusttestingatheory.Solikethisone,1:03IamlookingforwherewecalldecodebecauseIthinkwe’llmakeitatleastthatfar.Sosincethesourcecameback[canEinallylinktosourceindebugger]Icanactuallyjustruntothispoint.Sincethesourcecodeofthedebuggerwasn’tworking,Iwasactuallyswitchingbackandforthbetweensourcecodedebugginandassemblydebugging.Therearedifferentthingsthatareeasiertodo.Ok,sothat’swherewecalldecode.[wantedtoknowcallerofdecode]1:04SonowI’mjustgoingtostepalittlebithere.Ah,Ibetthat’sit.Sowherearewe–we’reinmachinedotcpp.[sawacall–readaddressmemory=thathethinkstriggerserrorstring]Andwe’reindecode.[methodofmachinedotcpp]1:05AndIwentahabecauseIsawthefunctionreadaddressmemory.AndI’mpositingthatthatiswhereourerroriscomingfrom,andthisiswhat’scallingreadvirtual.SoIbetreadvirtualisfailing,sowe’llcontinuetostepthroughhere.Andthat’sactuallywhatIwanttochange.Iwanttochangethebehaviorwhenreadvirtual[Einallylocatedprogrampointwherehewantstomakeachange!–beinterestingtocomparetohowlongittakesdevelopersinothercases]failsso.SoIthinkIfoundwherethecodechangewouldneedtogo,butIneedtoconEirmthatthat’stheplacewherethecodechangeneedstogo,andthenIneedtoreadthroughthecodetoseewhatmightbethesafestwaytomakethischange.1:06Sowe’llputthecursorhereandwe’llruntothispoint,andwe’llstepintoreadinstructionmemory.AndIthinkallofthisislargely,wait,that’snotright.Here’swherewecallreadvirtual,sowe’llwalkthroughthisjusttomakesurewedon’t’,butIthinkwe’rejustgoingtohit,yeah,wehitthatbranch,andthenwegointothatbranch,yep.[veriEiedthatthatisthecallthatfailsbyjustseeinghowthereturnfromfailureiscausingittostepintootherbranchesofthemethodonthefailurepath]Readphysicalisgoingtobefalse,sowe’retocallreadvirtual,yepthat’souroffset,ouroffsetisourorigainlparameter,there’souroutparameter,let’ssee.Sowherearewe,we’reindump.cpp1:07Andwe’reatline8958.[stepping,inspectingsomeimmediates]Oh,that’snotright.Cpp.Sowe’lljustwalk,andthereisourreadvirtualfailure.Sowhenreadvirtualfailswithsomethingotherthansok,wegotodone.[stepping]Yep,andthen1:08Isthatinstructionmemory.Yeah,sothat’snull.Whatisthison,memorybites.Yeah,sowereadnothingandthere’snothinginourbuffer.Soclearthat.1:09What’sourstatusatthispoint,ourstatusisthehresult,ok.[checkinginimmediate]Ok,so,oksothere’swherewereturnthestatus.And,yepwefailed,sowereturnthatback,decodereturnstodisassemble,Sothat’sinteresting,wedon’tactuallycheckthestatushere,sowhatfunctionareweinhere,machinedotcpp,andwe’reindisassemble[functionname]
Why was this question so hard to answer?
26
HardtopickthecontrolflowpaththatleadsfromstarAngpointtotargetGuessandcheck:whichpathleadstothetarget?
m
error
Why are control flow questions frequent?
27
causality
ordering
choice
HelpsanswerquesAonsabout
Whatdoesthisdo?Whatcausesthistohappen?
DoesAhappenbeforeB?
Doesxalwaysoccur?InwhichsituaAonsdoesxoccur?
Whensca\eredacrossacodebase,findingstatementstoanswerthesequesAonscanbehard.
28
Defect-relatedfalseassumpAons&incorrectlyansweredquesAonsrelatedtocontrolflow
(commoncharacterisAcsofevidencesought)ReachabilityQues$ons
labobservaAons fieldobservaAons
28
PrimaryquesAonsfromlongestinvesAgaAon&debuggingacAviAesrelatedtocontrolflow
29
feasiblepaths
statementsmatchingsearchcriteria∩
Defect-relatedfalseassumpAons&incorrectlyansweredquesAonsrelatedtocontrolflow
(commoncharacterisAcsofevidencesought)
.
downstream upstream
searchcriteriaidenAfierstatementtype(fieldwrite/read,librarycall)
feasiblepathsfilter compare
ReachabilityQues$ons
Asearchalongfeasiblepathsdownstreamorupstreamfromastatementfortargetstatementsmatchingsearchcriteria
labobservaAons fieldobservaAons
29
PrimaryquesAonsfromlongestinvesAgaAon&debuggingacAviAesrelatedtocontrolflow
Reachability question: example
30
feasiblepaths
statementsmatchingsearchcriteria∩
Whereismethodmgenera$nganerror?
m
e
Asearchalongfeasiblepathsdownstreamorupstreamfromastatement(m)fortargetstatementsmatchingsearchcriteria(callsto methode)
Longest activities related to reachability questions
31
4outofthe5longestinves$ga$onac$vi$es
5outofthe5longestdebuggingac$vi$es
Primary question Time (mins) Related reachability question
How is this data structure being mutated in this code? 83 Search downstream for writes to data
structure“Where [is] the code assuming that the tables are already there?” 53 Compare behaviors when tables are or are
not loadedHow [does] application state change when m is called denoting startup completion? 50 Find field writes caused by m
“Is [there] another reason why status could be non-zero?” 11
Find statements through which values flow into status
Where is method m generating an error? 66 Search downstream from m for error textWhat resources are being acquired to cause this deadlock? 51 Search downstream for acquire method
calls“When they have this attribute, they must use it somewhere to generate the content, so where is it?”
35 Search downstream for reads of attribute
“What [is] the test doing which is different from what my app is doing?” 30 Compare test traces to app traces
How are these thread pools interacting? 19 Search downstream for calls into thread pools
Insights
32
‣Developerscanconstructincorrectmentalmodelsofcontrolflow,leadingthemtoinsertdefects
‣ThelongestinvesAgaAon&debuggingacAviAesinvolvedasingleprimaryquesAonaboutcontrolflow
‣FoundevidenceforanunderlyingcauseofthesedifficulAesChallengesansweringreachabilityques$ons
Paper prototype study• Built mockups of interface for task from lab study
• Asked 1 participant to complete lab study task with Eclipse & mockup of Reacher
• Paper overlay of Reacher commands on monitor
• Experimenter opened appropriate view
• Asked to think aloud, screen capture + audio recording34
Study results• Used Reacher to explore code, unable to complete task
• Barriers discovered
• Wanted to see methods before or after, not on path to origin or destination
• Switching between downstream and upstream confusing, particularly search cursor
• Found horizontal orientation confusing, as unlike debugger call stacks
• Wanted to know when a path might execute35
Examples of observed reachability questions Reacher supports Steps to use Reacher
What resources are being acquired to cause this deadlock? Search downstream for each method which might acquire a resource, pinning results to keep them visible
When they have this attribute, they must use it somewhere to generate the content, so where is it?
Search downstream for a field read of the attribute
How are these thread pools interacting? Search downstream for the thread pool classHow is data structure struct being mutated in this code (between o and d)?
Search downstream for struct class, scoping search to matching type names and searching for field writes.
How [does] application state change when m is called denoting startup completion?
Search downstream from m for all field writes
Step 2: Find statements matching search criteria
36
Step 3: Help developers understand paths and stay oriented
37
Goal:helpdevelopersreasonaboutcontrolflowbysummarizingstatementsalongpathsincompactvisualizaAon
Challenges:controlflowpathscanbe
complex
long
repeAAve
developersgetlostanddisorientednavigaAngcode
Approach:
visuallyencodeproperAesofpath
hidepathsbydefault
coalescesimilarpaths
usevisualizaAontosupportnavigaAon
Example
38
Evaluation
39
Method 12developers15minutestoanswerreachabilityquesAonx6Eclipseonlyon3tasksEclipsew/REACHERon3tasks
Tasks
BasedondeveloperquesAonsinlabstudy.
Example:
WhenanewviewiscreatedinjEdit.newView(View),whatmessages,inwhatorder,maybesentontheEditBus(EditBus.send())?
DoesREACHERenabledeveloperstoanswerreachabilityquesAonsfasterormoresuccessfully?
(ordercounterbalanced)
DeveloperswithREACHERwere5.6AmesmoresuccessfulthanthoseworkingwithEclipseonly.
Results
40
TaskAmeincludesonlyparAcipantsthatsucceeded.
(notenoughsuccessfultocompareAme)
REACHER helped developers stay oriented
41
WhennotusingREACHER,parAcipantsogenreportedbeinglostandconfused.
ParAcipantswithREACHERusedittojumpbetweenmethods.
“WhereamI?I’msolost.”“Thesecallstacksarehorrible.”“Therewasa call to ithere somewhere,butIdon’trememberthepath.”“I’mjusttoolost.”
“Itseemsprettycool ifyoucannavigateyourwayaroundacomplexgraph.”
“Ilikeitalot.Itseemslikeaneasywaytonavigatethecode.AndtheviewmapstomoreofhowIthinkofthecallhierarchy.”“Reacherwasmyhero.…It’salotmorefuntouseandlookat.”“Youdon’thavetothinkasmuch.”
ParAcipantsreportedthattheylikedworkingwithREACHER.
Reflection on design process• Started with a goal: make debugging in large, complex
codebases better
• Observed users to build insight into what key challenge was
• Rather than address usability challenges of existing debugging tools, designed new way to debug
• Gathered evidence that it worked better42
Needfinding
43
Needfinding (a.k.a. design research)• Goal: understand user’s needs
• Use of methods to gather qualitative data
• behaviors, attitudes, aptitudes of potential and existing users
• technical, business, and environmental contexts - domain
• vocabulary and social aspects of domain
• how existing products used
• Empowers team w/ credibility and authority, helping inform decisions44
Needfinding vs. market researchNeedfinding
• What users really need
• How they will really use product
• Qualitative methods to study in depth
• Small numbers of participants45
Market research
• Who might purchase item
• What factors influence purchasing
• Quantitative studies w/ focus groups, surveys
• Large numbers of participants
Example• Cooper conducted a user study for entry-level video editing
product
• Company built professional software, looking to move into consumer software
• Help connect those w/ computers and video cameras
• Found strongest desire for video editing was parents
• Found 1/12 had successfully connected camera, using work IT guy46
Solving the correct problem
• Practices may sometimes mask deeper problems
• Goal: uncover layers of practices to understand how problems emerge
47
Interviews• May include bother current users and potential users w/ related needs
• Questions
• context of how product fits into lives or work
• when, why, how is or will product be used
• what do users need to know to do jobs?
• current tasks and activities, including those not currently supported
• goals and motivations of using product
• problems and frustrations with current products or systems48
Observations• Most incapable of accurately assessing own behaviors
• May avoid talking about problems to avoid feeling dumb
• Observing yields more accurate data
• Capture behaviors: notes, pictures, video (if possible)
49
Contextual inquiry
• Method that includes both interviews and observations
• Next time
50
Ideation
51
Ideation• Process of generating, developing, communicating new ideas
• Guidelines and best practices
• Generate numerous ideas
• Number ideas
• Avoid premature dismissal of ideas
• Sharpen the focus - pose the right problem
• Build and jump - build to keep momentum on ideas, jump when theme tapers out
52