MakingBetterUseoftheCrowd

JennWortmanVaughanMicrosoftResearch

Afewdisclaimers…

Aretherebetterwaystomakeuseofthecrowd?

Whatotherproblemscanthecrowdsolve?

1. DirectApplicationstoNLPandMachineLearning

2. HybridIntelligenceSystems

3. LargeScaleStudiesofHumanBehavior

Part1:ThePotentialofCrowdsourcing

“Crowd”guitar

man

Part2:TheCrowdisMadeofPeople

• Whatmotivatesworkers?• Areworkersindependent?• Areworkershonest?

Whatdoesthisteachusabouthowtoeffectivelyinteractwithcrowd?

Hint:Berespectful.Beresponsive.Beclear.

Extensivenotes,slides,andeventuallyvideoat

http://www.jennwv.com/projects/crowdtutorial.html

Part1:ThePotentialofCrowdsourcing

1. DirectApplicationstoNLPandMachineLearning

2. HybridIntelligenceSystems

3. LargeScaleStudiesofHumanBehavior

ThePotentialofCrowdsourcing

GeneratingLabeledData

Learner

Learner“dog”

“cat”

“dog” “cat”

“cat” “cat”

Aggregationofnoisylabels

Learner“dog”

“cat”

TrainedModel

Aggregationofnoisylabels

“dog” “cat”

“cat” “cat”

“cat”

Learner“dog”

“cat”

TrainedModel

Aggregationofnoisylabels

“dog” “cat”

“cat” “cat”

Usedtoannotatemedicalimages,labeltext,extractandlabelfeatures ofscenes.

Inspiredhugeamountsofalgorithmicworkonaggregation.

AggregatingLabelswithEM• Input:Worker-generatedlabelsforeachinstance

• Calculateaninitialestimateofeachinstance’slabelbasedonasimplemajorityvote

• Repeatuntilconvergence:– Treatingthecurrentlabelestimatesastruth,estimateeachworker’squality

– Treatingthequalityestimatesastruth,calculatethemostlikelylabelforeachinstance

• Output:Oneaggregatedlabelforeachinstance[Dawid andSkene,1979]

AggregatingLabelswithEM

• Noguaranteesonoptimality,buttendstoworkprettywellinpractice

• Manyrecentvariantshavebeenproposedtoincorporatethevaryingdifficultylevelsofinstances,workerexpertise,theexistenceof“gold”tasks,etc.

[Dawid andSkene,1979]

BeyondSimpleLabels:CrowdTranslation

• Crowdworkers areaskedto– Translatesentencesfromonelanguagetoanother– Editotherworkers’translationstomakethemmorefluentandgrammatical

– Rankthequalityoftheresultingtranslations

• Canusemachinelearningtopredictthehighestqualitytranslationbasedonsentence-levelfeatures,worker-levelfeatures,andranks

[Zaidan andCallison-Burch,2011]

GeneratingSimilarityMeasures

[Gomesetal.,2011]

GeneratingSimilarityMeasures

flags noflags[Gomesetal.,2011]

GeneratingSimilarityMeasures

Democrats Republicans[Gomesetal.,2011]

CrowdClustering

[Gomesetal.,2011]

Bayesianmodel

CrowdsourcingforEvaluation

cheesekalebreadsteak

mushroompizza...

electionsenatebill

delegatepresidentproposal

...

EvaluatingTopicModels

Tobeusefulfordataexplorationorsummarization,topicsmustbehuman-interpretable!

[Changetal.,2009]

EvaluatingTopicModels

mushroom,kale,cheese,bread,election,steak

workeraccuracy

human-interpretability

Previousmeasuresofsuccess(e.g.,loglikelihoodofheld-outdata)donotimplyinterpretability!

[Changetal.,2009]

Word intrusiontask:

HumanDebugging

HumanDebugging

• Semanticsegmentation:partitionanimageintosemanticallymeaningfulparts,labeleachpart

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

“cat”

HumanDebugging

• Semanticsegmentation:partitionanimageintosemanticallymeaningfulparts,labeleachpart

Whichcomponentistheweakestlink?

segmentclassifier

supersegmentclassifier

sceneclassifier

shapeprior

objectdetector

CRFmodel

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

HumanDebugging

segmentclassifier

supersegmentclassifier

sceneclassifier

shapeprior

objectdetector

CRFmodel

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

HumanDebugging

segmentclassifier

supersegmentclassifier

sceneclassifier

shapeprior

objectdetector

CRFmodel

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

HumanDebugging

segmentclassifier

supersegmentclassifier

sceneclassifier

shapeprior

objectdetector

CRFmodel

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

HumanDebugging

segmentclassifier

supersegmentclassifier

sceneclassifier

shapeprior

objectdetector

CRFmodel

[Parikh&Zitnick,2011;Mottaghi etal.,2013]

Humanslessaccurateattask,butsystemperformancestillimproved

1. DirectApplicationstoNLPandMachineLearning

2. HybridIntelligenceSystems

3. LargeScaleStudiesofHumanBehavior

ThePotentialofCrowdsourcing

HybridIntelligenceforSpeechRecognition

Crowd-BasedClosedCaptioning

Isitpossibletoprovidereal-timeclosedcaptioningoflectures,meetings,orotherday-to-dayconversations?

[Lasecki etal.,2012]

Thesystemmergesreal-timepartialinputs fromdynamic,untrainedcrowdstooutperformindividuals

Crowd-BasedClosedCaptioning

[Lasecki etal.,2012]

HybridIntelligenceforScheduling

Cobi:Communitysourced Scheduling

[projectcobi.com]

Abigconstrainedoptimizationproblemwithnoaccesstotheconstraints!

1. Committeesourcing 2. Authorsourcing

3. Scheduling 4. Attendeesourcing

[projectcobi.com]

Authorsourcing

crowdsourced clustering!

[projectcobi.com]

87%responserate!

Scheduling

[projectcobi.com]

Thesystemsolvesanoptimizationproblemtoproposeaschedule,butchairsretaincontrol.

HybridIntelligenceforWriting

TheSelfsourcing Process

1. Collectcontent

2. Organizecontent

3. Turncontentintowriting

[Teevan etal.,2016]

CollectContent

The MicroWriter breaks writing into microtasks.

Collaborative writing typically requires coordination.

Microtasks can be done while mobile.

Structure turns big tasks into small microtasks.

Microtasks can be shared with collaborators.

Collaborators can be known or crowd workers.

People have spare time when mobile.

Microtasks make it easy to get started.

[Teevan etal.,2016]

OrganizeContent

collaboration

microtask

mobile

The MicroWriter breaks writing into microtasks.

Collaborative writing requires coordination.

Microtasks can be done while mobile.

Structure turns big tasks into small microtasks.

Microtasks can be shared with collaborators.

Collaborators can be known or crowd workers.

People have spare time when mobile.

Microtasks make it easy to get started.

[Teevan etal.,2016]

TurnContentintoWriting

Collaborative writing typically requires coordination, but microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues or paid crowd workers.

[Teevan etal.,2016]

Collaborative writing requires coordination.

Microtasks can be shared with collaborators.

Collaborators can be known or crowd workers.

collaboration

TurnContentintoWriting

Collaborative writing typically requires coordination, but microtasks are easy to share with collaborators without the need for coordination. The collaborators can be known colleagues or paid crowd workers.

Structure makes it possible to turn big tasks into a series of smaller microtasks. For example, the MicroWriterbreaks writing into microtasks. These microtasks make the larger task easier to start.

People have spare time when mobile, and these micromoments are ideal for doing microtasks.

[Teevan etal.,2016]

TheSelfsourcing Process

1. Collectcontent

2. Organizecontent

3. Turncontentintowriting

• Steps2&3couldbedownbycrowdworkers,traditionalML/AIapproaches,oracombination

• Authortakesfinalpass,noneedforperfection

Crowdsourcing

[Teevan etal.,2016]

HybridIntelligenceinIndustry

1. DirectApplicationstoNLPandMachineLearning

2. HybridIntelligenceSystems

3. LargeScaleStudiesofHumanBehavior

ThePotentialofCrowdsourcing

UserStudiesforSecurityResearch

HowwelldoInternetusersunderstandsecurityrisks?

Whotriestoguesspasswords?

Only14%mentionedbothstrangersand familiarpeopleasthreats

p@ssw0rd pAsswOrdvs.

[Uretal.,2016]

UserStudiestoImprovetheCommunicationofNumbers

[Barrioetal.,2016]

Perspectives

• IsaonehundredbilliondollarcuttotheUSfederalbudgetbigorsmall?

• Onehundredbilliondollarsisabout...– 3%ofthe2015USfederalbudget– 1/6ofannualUSspendingonmilitary– 30%ofthenetworthofBeyoncé– $5foreverypersoninNewYorkstate

[Barrioetal.,2016]

SixmonthsofNewYorkTimesfrontpagearticles

Workersratedotherworkers’perspectivesforhelpfulness

Chosethehighest-ratedperspectives

64quoteswithmeasurements

370crowd-generatedperspectiveswithincentivesforquality

[Barrioetal.,2016]

Step1:PerspectiveGeneration

PerspectiveExamples

• TheOhioNationalGuardbrought33,000gallons ofdrinkingwatertotheregion.

• Toputthisintoperspective,33,000gallonsofwaterisaboutequaltotheamountofwaterittakestofill2averageswimmingpools.

[Barrioetal.,2016]

PerspectiveExamples

• Theyalsorecommendedsafetyprogramsforthenation’sgunowners;Americansownalmost300millionfirearms.

• Toputthisintoperspective,300millionfirearmsisabout1firearmforeverypersonintheUnitedStates.

[Barrioetal.,2016]

Step2:PerspectiveExperiments

• Randomizedexperimentsrunon3200+subjectsonAMTtotestthreeproxiesofcomprehension– Recall– Estimation– Errordetection

• Supportfoundforthebenefitsofperspectivesacrossallexperiments– Example:55%rememberednumberoffirearmsinUSwithperspective,only40%without

[Barrioetal.,2016]

UserStudiesforOnlineAdvertising

TheCostofAnnoyingAds

[Goldsteinetal.,2013]

Advertiserspaypublisherstodisplayads,butannoyingadscostpublisherspageviews.

Howmuchdoannoyingadscostpublishersindollars?

vs.

TheCostofAnnoyingAds

[Goldsteinetal.,2013]

Step1:Usethecrowdtoidentifyannoyingads.

vs.

GoodAds

[Goldsteinetal.,2013]

BadAds

[Goldsteinetal.,2013]

Step2:EstimatetheCost•Workersaskedtolabelemailasspamornot• Showngood,bad,ornoads;paidvaryingamountsperemail• Howmuchmoremustaworkerbepaidtodothesametaskswhenshownbadads?

[Goldsteinetal.,2013]

Step2:EstimatetheCost

• Goodadsleadtoaboutthesamenumberofviews(emailsclassified)asnoads

• Costsmorethan$1extratogenerate1000viewsofbadadsinsteadofnoadsorgoodads

• Takeaway:Publisherslosemoneybyshowingbadadsunlesstheyarepaidsignificantlymoretoshowthem

[Goldsteinetal.,2013]

1. DirectApplicationstoNLPandMachineLearning

2. HybridIntelligenceSystems

3. LargeScaleStudiesofHumanBehavior

SummaryofPart1

Part2:TheCrowdisMadeofPeople

Traditionalcomputersciencetoolsletusreasonaboutprogramsrunonmachines(runtime,scalability,correctness,...)

Whathappenswhentherearehumansintheloop?

Needamodelofhumanbehavior.(Aretheyaccurate?Honest?Dotheyrespondrationallytoincentives?)

Wrongassumptionsleadtosuboptimalsystems!

“ButIonlywanttousecrowdsourcing togeneratetrainingdataorevaluatemymodel.”

Understandingthecrowdcanteachyou– Howmuchtopayforyourtasksandwhatpaymentstructuretouse

– Howmuchyoureallyneedtoworryaboutspam– Howandwhytocommunicatewithworkers–Whetheryourlabels/evaluationsareindependent– Howtoavoidcommonpitfalls

TheCrowdisMadeofPeople

• Crowdworker demographics• Honestyofcrowdworkers• Monetaryincentives• Intrinsicmotivation• Thenetworkwithinthecrowd

Bestpractices!Tipsandtricks!

CrowdsourcingPlatforms

AmazonMechanicalTurk

Workers Requesters

AlternativePlatforms

• GermanplatformwithmanyEuropeanworkersofferingsupportfortranslationandwebresearchplusmobilecrowdsourcing

• OffersenterprisesolutionsforbusinesseswithAI/dataneeds(searchrelevanceevaluation,sentimentanalysis,dataclassification)

AlternativePlatforms

• Marketplaceforfreelancerswithlargerjobslikewritingarticlesordesigningwebsites

• UK-basedplatformfocusedonconnectingresearcherswithsubjectsforexperiments

Crowdworker Demographics

DemographicsofMechanicalTurk

[mturk-tracker.com]

DemographicsofMechanicalTurk

[mturk-tracker.com]

• 70-80%US,10-20%India

• Roughlyequalgendersplit

• Median(reported)householdincome:– $40K-$60KforUSworkers– Lessthan$15KforIndianworkers

• Canbebigchangesdependingontimeofday

Areworkersdishonest?

ExperimentalParadigm

• Askparticipantsaboutdemographics– Sex,Age,Location,Income,Education

• Askparticipantstoprivately rolladie(orsimulateitonanexternalwebsite)andreporttheoutcome

payment=$0.25+($0.25*roll)

• Ifworkershonest,meanreportedrollshouldbeabout3.5...Whatdoyouthinkthemeanwas?

[Suri etal.,2011]

Baseline

• Averagereportedrollhigherthanexpectation– M=3.91,p <0.0005

• Playersunder-reportedonesandtwosandover-reportedfives

• Butmanyworkerswerehonest!

• SimilartoFischbacher &Huesi labstudy

Roll

Proportion

0.00

0.05

0.10

0.15

0.20

0.25

1 2 3 4 5 6

[Suri etal.,2011]

Thirtyrolls

• Overall,muchlessdishonesty

• Averagereportedrollmuchclosertoexpectation– M=3.57,p <0.0005

• Only3of232reportedsignificantlyunlikelyoutcomes

• Only1wasfullyincomemaximizing(allsixes)

• Whyisthisthecase?

Roll

Proportion

0.00

0.05

0.10

0.15

1 2 3 4 5 6

[Suri etal.,2011]

DishonestyCanAddUp

• “Areyoutheparentorguardianofachildwithautism?”– 4.3%ofparticipantssaidyesincontrol– 7.8%ofparticipantssaidyeswhentoldthatthiswasaprescreeningtest forafurtherstudy

• Seemslikeasmalldifference,butwouldleadto(atleast)45%impostersinthesubsequentstudy!

[ChandlerandPaolacci,2011]

Takeaways&RelatedBestPractices

• Mostworkersarehonestmostofthetime.

• Butsomearenot.Youshouldstillusecaretoavoidattacks.

• Workersmaydeceiverequesterstogainaccesstowork.Prescreeningshouldbedonewithcare,ideallyaspartofaseparatetask.

MonetaryIncentives

Howmuchshouldyoupay?

Ausefultrick:• Pilotyourtaskonstudents,colleagues,orafewworkerstoseehowlongitgenerallytakes.

• UsethattomakesureyourpaymentsworkouttoatleasttheUSminimumwage.

Benefits:• It’sthedecentthingtodo!• Ithelpsmaintaingoodrelationshipswithworkers.

Canperformance-basedpaymentsimprovethequalityofcrowdwork?

Proofreadthistext,earn$0.50

Earnanextra$0.10foreverytypofound

[Hoetal.,2015]

PriorWorkonCrowdPayments

–Payingmoreincreasesthequantityofwork,butnotthequality[MW09,RK+11,BKG11,LRR14]–PBPsimprovequality[H11,YCS14]–PBPsdonotimprovequality[SHC11]–Bonussizesdon’tmatter[YCS13]

[Hoetal.,2015]

Performance-BasedPayments

Weexplorewhen,where,andwhy performace-basedpaymentsimprovethequalityofcrowdworkonAmazonMechanicalTurk.

[Hoetal.,2015]

CanPBPs work?

• Warm-uptoverifythatPBPs canleadtohigherqualitycrowdwork onsometask.

• TestwhetherthereexistsanimplicitPBPeffect:workershavesubjectivebeliefsonthequalityofworktheymustproducetoreceivethebasepayment,andsoalreadybehaveasifpaymentsare(implicitly)performance-based.

[Hoetal.,2015]

CanPBPs work?• Task:Proofreadanarticleandfindspellingerrors. • Werandomlyinsert20typos

• sufficiently->sufficently• existence->existance• …

• Usefulproperties:• Qualityismeasurable• Exertingmoreeffort->betterresults

[Hoetal.,2015]

CanPBPs work?Basepayment:$0.50;Bonuspayment:$1.00

ThreeBonusTreatments:• NoBonus: nobonusormentionofabonus• BonusforAll: getthebonusunconditionally• PBP: getthebonusifyoufind75%of

thetyposfoundbyothers

TwoBaseTreatments:– Guaranteed: guaranteedtogetpaid– Non-Guaranteed: nomentionofaguarantee

[Hoetal.,2015]

CanPBPs work?

• Resultsfrom1000uniqueworkers

• Guaranteedpaymentshurt(implicitPBP)

• PBPs improvequality

• Unlikeinpriorwork,payingmorealsoimprovesquality

[Hoetal.,2015]

UnderwhatconditionsdoPBPs work?Bonusthreshold(585uniqueworkers)• $0.50base+$1.00bonusforfindingXtypos

Ctrl 5T 25% 75% All

• PBPs workforawiderangeofthresholds

• Subjectivebeliefs(5typosvs.25%oftypos)canimprovequality

[Hoetal.,2015]

Bonusamounts(451uniqueworkers)• $0.50base+$X bonusforfinding75%of typos• PBPs workaslongasthebonusislargeenough

●

●

●

●

11

12

13

14

0.00 0.25 0.50 0.75 1.00

Bonus Amount

Typo

s Fo

und

couldexplainShawetal.,2011couldexplainYinetal.,2013

[Hoetal.,2015]

UnderwhatconditionsdoPBPs work?

WhichtasksdoPBPs workon?

• Whatpropertiesofataskleadtoqualityimprovementsfromperformance-basedpay?

• Somepilotexperimentsonaudiotranscriptionsuggestedthat– PBPs improvequalityforeffort-responsivetasks– Itisnotalwaysstraight-forwardtoguesswhichtasksareeffort-responsive

[Hoetal.,2015]

WhichtasksdoPBPs workon?

[Hoetal.,2015]

Takeaways&RelatedBestPractices

• AimtopayatleastUSminimumwage.Pilotyourtasktofindouthowlongittakes.

• Performance-basedpaymentscanimprovequalityforeffort-responsivetasks.Pilottochecktherelationshipbetweentimeandquality.

• Bonuspaymentsshouldbelargerelativetothebase.Thepreciseamountandprecisecriteriaforreceivingthebonusdon’tmattertoomuch.

IntrinsicMotivation

WorkThatMatters• Threetreatments:– control:nocontextgiven– meaningful: toldtheywerelabelingtumorcellstoassistmedicalresearchers

– shredded: nocontext,toldworkwouldbediscarded

• Meaningful->quantity up,butquality similar• Shredded->quality down,butquantity similar

[ChandlerandKapelner,2013]

Gamification

[vonAhn andDabbish,2004]

Gamification

[vonAhn,Kedia,andBlum,2006]

Takeaways&RelatedBestPractices

• Workersproducemoreworkwhentheyknowtheyareperformingameaningfultask,butthequalityoftheirworkmightnotimprove.

• Gamificationcanalsoincreaseproductivity.Wellcalibratedtimedresponsesandscorekeeping(withorwithouthighscorelists)canbothincreaseenjoyment.

TheCommunicationNetworkWithintheCrowd

Implicitassumption:Crowdworkersareindependent

[Yinetal.,2016]

Inrealityworkerstalkandcollaborate

Recreatesocialconnectionsand

supportM.L.Gray,S.Suri,S.S.AliandD.Kulkarni.TheCrowdisaCollaborativeNetwork.CSCW 2016

N.Gupta,D.Martin,B.V.Hanrahan andJ.O’Neil.Turk-lifeinIndia.Group 2014

Helpeachotherwithadministrativeoverhead

Ming’stasksaregreat!

Sharetasksandreputableemployers

Ethnographicfieldstudiesshowthat crowdworkers...

[Yinetal.,2016]

ACommunicationNetwork

Whatisthescale? Whatisthestructure? Howisitused?[Yinetal.,2016]

Ourgoal:Opentheblackboxofcrowdsourcing tomapthecommunicationnetworkofcrowdworkers

[Yinetal.,2016]

Whyisitchallenging?

ThenetworkisnotaccessiblefromtheAPIsowecan’tsimplydownload,crawl,orscrapeit!

Wanttomapthenetworkinawaythat#1Elicitsonly“true”edges#2Elicitsasmanytrueedgesaspossible#3Preservesworkers’privacy

[Yinetal.,2016]

AWebApp

• Workers self-report their connections

• Providessomevaluebacktotheworkerssothatit’sintheirbestinteresttoreportasmanytrueconnectionsaspossible

[Yinetal.,2016]

5268connections

10,354 workers(roughlyacensusofMechanical Turk[Stewartetal.2015])

[Yinetal.,2016]

1,389(13%)connectedworkers

Onaverage,workerscommunicatewith7.6 others

Maxdegreeis321

[Yinetal.,2016]

Largestcomponentincludes994(72%) workers

[Yinetal.,2016]

ANetworkEnabledByForums

• 59% ofallworkersand83% ofconnectedworkersreportedusingatleastoneforum.

• 90%ofalledgesarebetweenpairsofworkerswhocommunicateviaforums,and86% arebetweenpairswhocommunicateexclusivelythroughforums.

[Yinetal.,2016]

ForumsCreateSubcommunities

Reddit HWTF MTurkGrind TurkerNation

Facebook MTurkForum[Yinetal.,2016]

Subcommunities AreDifferent

TopologicalStructure: Howtightlyconnectediseachsubcommunity?

TemporalDynamics: Dorelationshipsendureovertime?

CommunicationContent: Iscommunicationsocialorstrictlybusiness?

[Yinetal.,2016]

MeasuresofSuccess

Property Connected UnconnectedBeactive>1year 55% 46%

Useforums 83% 56%Master 11% 7%

Approvalrate 98.6% 97.4%

[Yinetal.,2016]

Connectedworkerswerealsomorelikelythanunconnectedworkerstofindourtaskearly.

TakeawaysandRelatedBestPractices

• Forumusageiswidespread.Forumsarethevirtual“watercoolers”ofcrowdworkers.

• Engagewithworkersonforums.Introduceyourself.Introduceyourtasks.

• Activelymonitorforumdiscussionaboutyourtask.Whenappropriate,requestthatworkersdonotdiscussyourtask.Monitoranyway.

• Becarefulaboutassumingindependence!

AdditionalBestPractices

MaintainGoodRelationshipswithWorkers

• Setasidetimetoactivelymonitoryourrequesteremailaccountandrespondtoquestions.

• Approveworkquickly.

• Avoidrejectingworkexceptinthemostextremeofcircumstances.

• Strivetobeanethicalrequester.

http://guidelines.wearedynamo.org

TipstoMakeYourProjectRunSmoothly

• Pilot,pilot,pilot!Testyourtaskonyourcollaborators,othercolleagues,andeventuallysmallbatchesofworkers.

• Iterateasmanytimesasneeded.

Ifyourememberoneslidefromthistalk,rememberthis!

TipstoMakeYourProjectRunSmoothly

• Createclearinstructions.Includequizquestionsifneeded.Pilotthemandcollectfeedback.

• Createanattractiveandeasy-to-useinterface.Pilotthistoo!

• Askworkersforfeedback.Askthemtoreportbugs.Conductexitsurveyswhenappropriate.Workersgenerallywanttohelp!

Thanks...

ToChien-Ju Ho,AndrewMao,JoellePineau,SidSuri,HannaWallach,andespeciallyMingYinforextensivediscussionsandfeedback

ToDanGoldstein,Chien-Ju Ho,JakeHofman,Roozbeh Mottaghi,SidSuri,JaimeTeevan,MingYin,Haoqi Zhang,andalloftheircollaboratorsfortheuseofmaterialfromtheirslides

Andtoallthepeoplewhosentmepointerstocoolresearch...thistutorialwasacrowdsourced effort!

Extensivenotes,slides,andeventuallyvideoat

http://www.jennwv.com/projects/crowdtutorial.html

jenn@microsoft.comhttp://jennwv.com@jennwvaughan