Crowd Agents: Interactive Crowd-Powered Systems in the Real World

39
University of Rochester Human-Computer Interaction Jeffrey P. Bigham Crowd Agents Interactive Crowd-Powered Systems in the Real World Jeffrey P. Bigham University of Rochester

description

In this talk, I discuss several interactive crowd-powered systemsthat help people address real-world problems. For instance, VizWizsends questions blind people have about their visual environment tothe crowd, Legion allows outsourcing of desktop tasks to the crowd,and Scribe allows the crowd to caption audio in real-time. Thethousands of people have engaged with these systems, providing aninteresting look at how end users want to interact with crowd work.Collectively, these systems illustrate a new approach to humancomputation in which the dynamic crowd is provided the computationalsupport needed to act as a single, high-quality agent. The classicadvantage of the crowd has been its wisdom, but our systems arebeginning to show how crowd agents can surpass even expert individualson motor and cognitive performance tasks.

Transcript of Crowd Agents: Interactive Crowd-Powered Systems in the Real World

  • 1. Crowd AgentsInteractive Crowd-Powered Systems in the Real World Jeffrey P. Bigham University of RochesterUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham

2. Crowd AgentsInteractive Crowd-Powered Systems in the Real World Jeffrey P. Bigham University of RochesterUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham 3. IntroductionVizWizCrowd AgentsScribeUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 4. Introduction VizWizCrowd AgentsScribeHuman Assistance in HistoryWhat the Disability CommunityCan Teach Us About InteractiveCrowdsourcing. Jeffrey P.Bigham and Richard Ladner.Iteractions magazine. July 2011.University of Rochester Human-Computer InteractionJeffrey P. Bigham 5. IntroductionVizWizCrowd AgentsScribeConnectivityUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 6. IntroductionVizWizCrowd Agents ScribeUniversity of Rochester Human-Computer InteractionCourtesy Jeffrey P. Brabyn of John Bigham 7. IntroductionVizWizCrowd AgentsScribeRemote AssistanceVideo Relay ServicesReal-timeCaptioningUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 8. IntroductionVizWiz Crowd AgentsScribeConnectivity -> Crowd Mechanical Turk Friends and Family on Social NetworksUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham 9. IntroductionVizWiz Crowd Agents ScribeVizWizBigham et al. Nearly Real-Time Answers to Visual Questions. UIST 2010.University of Rochester Human-Computer InteractionJeffrey P. Bigham 10. IntroductionVizWizCrowd AgentsScribeAccess Technology Optical Character Recognition Color Recognizers Talking GPS Problems 1. Limited Scope 2. Unacceptable Error Rate 3. $$$ 4. Not Exactly What Users WantUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 11. IntroductionVizWizCrowd AgentsScribe Releasing VizWiz Released on May 31, 2011 5000 users asked more than 50,000 questions answers in less than a minuteUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 12. IntroductionVizWiz Crowd Agents Scribe Recruiting Crowd Quickly How many workers do we need? - number of current workers - likelihood of needing more workers Post jobs or remove jobs Turkers answer multiple questionsTurkit For $4/hr goes down to under 30s from start to finish. quikturkit.googlecode.comBigham et al. Nearly Real-Time Answers to Visual Questions. UIST 2010.University of Rochester Human-Computer InteractionJeffrey P. Bigham 13. IntroductionVizWizCrowd AgentsScribe Characterization of the Crowd - Workers Come and Go - Some May Do the Wrong ThingUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 14. IntroductionVizWizCrowd AgentsScribeSupporting a Continuous Interaction? Wheres the coffee? Walk to end of this hall, turn right. Turn right into the kitchen. Wheres the Soda on left, coffee on the right How do I use this machine? coffee?University of Rochester Human-Computer InteractionJeffrey P. Bigham 15. IntroductionVizWizCrowd AgentsScribeModel for Crowd AgentsUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 16. IntroductionVizWizCrowd Agents ScribeModel for Crowd Agents Input MediationLearningUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham 17. IntroductionVizWizCrowd AgentsScribeModel for Crowd Agents What interface is being controlled? How is input mediation done? Role of automated agents?University of Rochester Human-Computer InteractionJeffrey P. Bigham 18. IntroductionVizWizCrowd AgentsScribeChorusUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 19. IntroductionVizWizCrowd AgentsScribeUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 20. IntroductionVizWiz Crowd Agents Scribe Legion: Control of Any InterfaceInput Media onLegion Server- video stream Flash Media Server -- task descriptionInput Mediators -- crowd agreement/payment info- video streamquikTurkit -- task description - worker input (key presses, mouse clicks)Worker Interface - mediated inputLegion Client 250 8/108/10 20010/10 Explanation ofTime (sec)controls, and feedback 150regarding currentbonus level (tied tocrowd agreement). 10/10 100 4/10500Feedback reflecting workers Solo MobVote Active Leaderlast key press, and whether the interface last followedmultiple workers the crowd or the worker. W. Lasecki, S. White, K. Murray, R. Miller, and J.P. Bigham Real-Time Control of Existing Interfaces. UIST 2011.University of Rochester Human-Computer InteractionJeffrey P. Bigham 21. IntroductionVizWizCrowd AgentsScribeUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 22. Introduction VizWiz Crowd Agents Scribe Crowd MemoryW.S. Lasecki, S.C. White, K.I. Murray and J.P. Bigham. Crowd Memory: Learning inthe Collective. Collective Intelligence 2012.University of Rochester Human-Computer InteractionJeffrey P. Bigham 23. IntroductionVizWizCrowd AgentsScribeCrowd MemoryUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 24. IntroductionVizWizCrowd AgentsScribe Deployable Activity Recognition W.S. Lasecki, Y. Song, H. Kautz, and J.P. Bigham. Real-Time Activity Labeling for Deployable Activity Recognition. Submitted to CSCW 2012. Pervasive 2012 (poster)University of Rochester Human-Computer InteractionJeffrey P. Bigham 25. Legion:ScribeReal-Time Captions by Groups of Non-ExpertsUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 26. Introduction VizWiz Crowd Agents Scribe Real-Time Captioning Problem: produce text transcript of speech with less than 5-second latencyStenographersASR expensive cheapdifficult to schedule available on demand lack domain expertise Can Ican be trained for new vocab help?pretty accurate does not work*NO,you are worse than ASR. * in real settings from an unknown mic with speaker who hasnt trained the ASRUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 27. Introduction VizWiz Crowd Agents ScribeReal-Time Captioning W. Lasecki, C. Miller, A. Sadilek, A. Abumoussa, D. Borrello, R. Kushalnagar, J.P. Bigham. Real-Time Captioning by Groups of Non-Experts. UIST 2012.University of Rochester Human-Computer Interaction Jeffrey P. Bigham 28. Introduction VizWizCrowd AgentsScribeInput Mediator Multiple Sequence Alignment Online Version Stage 1 the Stage 2theStage 3thenowand Graphopen open openfileTime java javajava upWorker 1 open thefilenowWorker 2the java fielWorker 3openjava file up andBaseline open the javafilenowand W.S. Lasecki, C.D. Miller, D. Borrello and J.P. Bigham. Online Sequence Alignment for Real-Time Audio Transcription by Non-Experts. AAAI 2012 (poster).University of Rochester Human-Computer InteractionJeffrey P. Bigham 29. IntroductionVizWizCrowd AgentsScribeScribe Interface Encourages: - real-time input - global coverage - short sequences Co-evolution of Interface and AlgorithmUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 30. IntroductionVizWizCrowd AgentsScribeCoverage GraphUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 31. IntroductionVizWizCrowd AgentsScribeTradeoff Failures: n-factorial in pectoralUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 32. IntroductionVizWiz Crowd AgentsScribeInteresting Qualities Captionists can be experts not at captioning but in the subject Low cost $30/hour on Mturk (did not optimize) or free (impossible before) Recruited on demand for only as long as neededUniversity of Rochester Human-Computer Interaction Jeffrey P. Bigham 33. IntroductionVizWizCrowd AgentsScribe Scribe ASR Web prefetching is 1 technique thatA lactate fencing is one thinking that ressearchers rely on history based to andetc. rely on to improve network. the non history based technique thePhillipe pitching. Anything survived downloaded pages will be scanned and all incident techniques hyperlinks will beUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 34. Introduction VizWiz Crowd AgentsScribe Incorporating ASRCoverage Increase: 28% to 55%(single worker case)University of Rochester Human-Computer InteractionJeffrey P. Bigham 35. Conclusions General Lessons, Science, and the FutureUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham 36. Introduction VizWizCrowd Agents ScribeWhat would it take for meto be proud of my daughterbeing a crowd worker?- Niki Kittur @ CrowdCampCurrence Bigham after her first running race.University of Rochester Human-Computer Interaction Jeffrey P. Bigham 37. IntroductionVizWiz Crowd AgentsScribe Do GoodConnect to help and support.Do BetterDo better work than anyone could alone.University of Rochester Human-Computer Interaction Jeffrey P. Bigham 38. IntroductionVizWiz Crowd Agents Scribe hci.cs.rochester.edu @jeffbighamThanks!Funded by: National Science Foundation Grants (#IIS-1149709, #IIS-1116051, #IIS-1049080 ), and Google.University of Rochester Human-Computer InteractionJeffrey P. Bigham 39. IntroductionVizWizCrowd AgentsScribeUniversity of Rochester Human-Computer InteractionJeffrey P. Bigham