'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer...
-
Upload
ed-chi -
Category
Technology
-
view
1.130 -
download
3
description
Transcript of 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer...
![Page 1: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/1.jpg)
EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter
2009HCICWorkshopatYMCACamp,Colorado
![Page 2: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/2.jpg)
Asafield,earlyfundamentalcontributionsfrom:– Computerscientistsinterestedinchangesinwaysweinteract
withinformationsystems– Psychologistsinterestedintheimplicationsofthesechanges
Combustible,because:– Computerscientistswanttocreategreattools,butdidn’tknow
howtomeasureimpact– Psychologistswanttogobeyondclassicalresearchofthebrain
andhumancognition
Example:– anenduringinterestin‘augmentinghumanintellect’:V.Bush,
Licklider,Engelbart,inturninspiringStuCard,AlanNewell,AlanKay,andmanyothers.
2/13/09 HCIC "Living Lab" 2
![Page 3: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/3.jpg)
TheneedtoestablishHCIasascience– Adoptmethodsfrompsychology– Convenientandfitswellwithproblemsathand
Issuesaroundpersonalcomputing(WIMP)– Dualpurpose:understandnatureofhuman
behaviorandbuildupascienceofHCItechniques.
– GoodExamples:Fitts’Law,ModelsofHumanMemory,CognitiveandBehavioralModeling,InformationForaging
– StuartK.Card,WilliamK.English,andBettyJ.Burr(1978).Evaluationofmouse,rate‐controlledisometricjoystick,stepkeys,andtextkeysfortextselectiononaCRT.Ergonomics,21(8):601–613,1978.
2/13/09 3 HCIC "Living Lab"
![Page 4: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/4.jpg)
Beyondauserinfrontofcomputer– Yetevaluationmethodsmostlystayedthesame– PerceivedCHIpapertemplateforacceptance
Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore– Yesterday’sdiscussionaboutLargeDataandHCIwaslargelydrivenby
howHCIevaluationmethodsneedtochangetofitthewild
Thebestexamples:TrendsinSocialComputingandUbiCompforceustothinkaboutnewcontextofuse
2/13/09 HCIC "Living Lab" 4
![Page 5: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/5.jpg)
Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with
no time boundries Controllable experimental conditions Uncontrollable experimental conditions
2/13/09 5 HCIC "Living Lab"
![Page 6: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/6.jpg)
ThinkaboutresearchthathasbeendoneonUIanimationsorflashingicons.
Fromvisualperception,weknowmotionintheperipheryismorenoticeablethaninthefovealregion[DaVinci].
2/13/09 HCIC "Living Lab" 6
![Page 7: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/7.jpg)
EvaluationssurroundingmanyHCIsystemsforknowledgeworkfocusonproductivityincrease,butwhataboutfactorsforadoption?– Argument:ifnoproductivityincrease,thenadoptionis
irrelevant– Buttheoppositeargumentisjustasright:ifnoadoption,no
amountofproductivityincreaseshownisrelevant!
Academicresearchoftenfocusonproductivityimprovements,increasingtheperceivedgulfbetweentheivorytowerandthetrenches
AnExample:Colorcopierstudies
2/13/09 HCIC "Living Lab" 7
![Page 8: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/8.jpg)
Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations– Ecologicalconsiderations– Hardtogeneralizetonewtaskcontexts(withinterruptions,other
tasks,othergoals,unfocusedattention,moredisplays)– Hardtogeneralizetoothertools,apps– Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups
Exampleproblems:– Adoptionofmobiletechnology
iPhonesinJapan,single‐handedinput[PARC] BestsellingphonesinIndonesiacomeswithacompass[Bell]
– AggregatebehaviorofWikipediaorDelicioususers Bigdataanalysisofeditlogs
2/13/09 8 HCIC "Living Lab"
![Page 9: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/9.jpg)
Wasacomputationalmolecularbiologist Analogy:Justasbiologistsworkonmodelplantsand
genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.
Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.
2/13/09 HCIC "Living Lab" 9
![Page 10: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/10.jpg)
ObservationalStudies– Ethnography– SocialTechnicalDesign– IterativeDesign– DiaryStudies– LongitudinalStudieswithsingleoutcome
Problems:– Samplingfromnon‐normaldistributions– Treatvariationsinsocialcontextsas‘noise’– Whataboutadoption?
MixedMethodsisnowpopulartopartiallyameliorate– Convergentmeasurestellsuswe’regettingclosertothetruth
2/13/09 10 HCIC "Living Lab"
![Page 11: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/11.jpg)
– similartoVentureBusinessmentionedbyDavidMillen
Conductresearchonrealplatformsandservices– Nottoreplacecontrolledlabstudies– Expandourarsenaltocovernewsituations
Someprinciples:– Embeddedintherealworld– Ecologicallyvalidsituations– Embracethecomplexity– Relyonbig‐data‐sciencetoextractpatterns
Notfirsttosuggestthis:– S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On
ecologicalvalidityandubiquitouscomputing.HCIJournal,2008– EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital
FamilyPortrait[Rowan,Mynatt]
2/13/09 11 HCIC "Living Lab"
![Page 12: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/12.jpg)
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
2/13/09 12 HCIC "Living Lab"
![Page 13: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/13.jpg)
TraditionalApproach;Numerousexamples FavoredbyCHIreviewers Typicalsituationisthestudyofsomeinteractiontechnique
– Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput
Typicalmeasuresarequantitativeinnature– performanceintime,performanceinaccuracy,eyetracking,learning
measures,userpreferences
Issues:– Notalwaysecologicallyvalid– Hardtotakeallinteractionsintoaccount– Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.
2/13/09 13 HCIC "Living Lab"
![Page 14: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/14.jpg)
Hardertofindintheliterature Oftencomparingagainstanoldersystemasbaseline Typicalcaseiscomparisonoftwosystems
– (onewebsitewithanother,onewordprocessorvs.another)– Whichhighlightingfeatureworksbetter– Twotextinputtechniqueonacellphone
Typicalmeasuresaresimilarto(1) Issues:
– Somesimilarissuesto(1)becauseit’sinlab– Systemfeaturenotincontrol,sonotabletocomparefairly,or
isolatethefeature
2/13/09 HCIC "Living Lab" 14
![Page 15: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/15.jpg)
2/13/09 HCIC "Living Lab" 15
http://www.flickr.com/photos/crush_images/3245220458/
![Page 16: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/16.jpg)
Realapplicationsinecologicalvalidsituations Realfindingscanbeappliedtoarunningsystem Impactofresearchismoreimmediate,sincesystemisalready
running Typicalcaseisloganalyticswithlargesubjectpools
– logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.
Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis
Issues:– Factorsnotincontrol,findingsnotcomparable– Factorscannotbeisolated– Reasonsforfailureisoftenjustguesswork
2/13/09 HCIC "Living Lab" 16
![Page 17: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/17.jpg)
Hypothesis:ConflictiswhatdrivesWikipediaforward. Howtostudythis?
– Tukeyparadigm– Getalargepaper,andplotthedamndata!
– DownloadedallofWikipediaandalloftherevisions– Hadoop/MapReduce,MySQL,etc.
2/13/09 HCIC "Living Lab" 17
![Page 18: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/18.jpg)
2/13/09 18
60%
65%
70%
75%
80%
85%
90%
95%
100%
2001 2002 2003 2004 2005 2006
Perc
enta
ge o
f tot
al e
dits
Article
User
Article Talk
User Talk
Other
Maintenance
HCIC "Living Lab"
![Page 19: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/19.jpg)
Group A
Group B Group C
Group D
Number of users in user group A B C Total
Users with Korean point of view 10 6 0 16
Users with Japanese point of view 1 8 7 16
Neutral or Unidentified 7 3 6 17 2/13/09 19 HCIC "Living Lab"
![Page 20: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/20.jpg)
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
2/13/09 20 HCIC "Living Lab"
![Page 21: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/21.jpg)
2/13/09 HCIC "Living Lab" 21
![Page 22: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/22.jpg)
2/13/09 HCIC "Living Lab" 22
![Page 23: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/23.jpg)
2/13/09 23 HCIC "Living Lab"
![Page 24: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/24.jpg)
2/13/09 24 HCIC "Living Lab"
![Page 25: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/25.jpg)
2/13/09 25 HCIC "Living Lab"
![Page 26: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/26.jpg)
Hypothesis:SocialTaggingdoesn’tscaleovertime. Howtostudythis?
– Crawlasmuchtaggingdataaswecan.– Studythenoiseinthesystem.
– 40machinesfor3months
2/13/09 HCIC "Living Lab" 26
![Page 27: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/27.jpg)
Topics
Users Documents
Tags
T1…TnEncodingDecoding
Noise
2/13/09 27 HCIC "Living Lab"
Concepts
![Page 28: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/28.jpg)
2/13/09 HCIC "Living Lab" 28
Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)
![Page 29: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/29.jpg)
29 2/13/09 HCIC "Living Lab"
![Page 30: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/30.jpg)
Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.– Goodforcasesinwhicheconomicsmakessense[Google]– Changestosystemispossible;Factorscanbecontrolled.
TypicalcasemightbeA/Btesting,largesubjectpools Typicalmeasuresarebeingdeveloped
– Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?
– Usabilitymeasuresvs.Usefulnessmeasures
Issues:– Effortandresourcerequirementisdroppingbutstillsignificant– Hardforaresearchlabtotakeon
2/13/09 HCIC "Living Lab" 30
![Page 31: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/31.jpg)
2/13/09 HCIC "Living Lab" 31
CollaborativeFiltering
TapestrysystematPARC
GroupLens 6kusers Ecosystemwith
passionateusers
![Page 32: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/32.jpg)
2/13/09 HCIC "Living Lab" 32
![Page 33: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/33.jpg)
2/13/09 HCIC "Living Lab" 33
![Page 34: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/34.jpg)
2/13/09 34
HCIC "Living Lab"
![Page 35: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/35.jpg)
2/13/09 35
HCIC "Living
Lab"
![Page 36: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/36.jpg)
2/13/09 36
HCIC "Living
Lab"
![Page 37: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/37.jpg)
2/13/09 37
Guide
Web
Howto
Tips Help
Tools
Tip
Tricks
Tutorial
Tutorials
Reference
Semantic Similarity Graph
HCIC "Living Lab"
![Page 38: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/38.jpg)
2/13/09 HCIC "Living Lab" 38
![Page 39: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/39.jpg)
2/13/09 HCIC "Living Lab" 39
![Page 40: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/40.jpg)
Interactioncostsdeterminenumberofpeoplewhoparticipate
Surplusofattention&motivationatsmalltransactioncosts
Therefore… Importanttokeep
interactioncostslow
Cost of participation #
Peop
le w
illin
g to
pro
duce
for “
free
”
40 2/13/09 HCIC "Living Lab"
![Page 41: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/41.jpg)
In situ tagging while reading – No new window – Clicking vs typing
Tagging + highlighting
41 2/13/09 HCIC "Living Lab"
![Page 42: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/42.jpg)
![Page 43: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/43.jpg)
Exitingthecleanroom[Carteretal.]– SparseData– CriticalMass– Dealingwithrealisticenvironments– Unobtrusivenessofmonitoring– Ambiguityanderrorofinferencing
2/13/09 HCIC "Living Lab" 43
![Page 44: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/44.jpg)
Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.
Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.
2/13/09 44 HCIC "Living Lab"
Characteriza7on Models
PrototypesEvalua7ons
![Page 45: 'Living Laboratories': Rethinking Ecological Designs and Experimentation in Human-Computer Interaction](https://reader034.fdocuments.net/reader034/viewer/2022042602/557d66d1d8b42ae1438b571a/html5/thumbnails/45.jpg)
ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.
LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.
http://asc‐parc.blogspot.comhttp://[email protected]
WikiDashboard MrTaggy SparTag.us