Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

Easy (and cheap) data

Iperotis 2010b

Median hit - $006

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

gt500000 ldquoWorkersrdquo

~35-40 US 30-50 India+ ~100 countries

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

Easy (and cheap) data

Iperotis 2010b

Median hit - $006

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

gt500000 ldquoWorkersrdquo

~35-40 US 30-50 India+ ~100 countries

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Easy (and cheap) data

Iperotis 2010b

Median hit - $006

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

gt500000 ldquoWorkersrdquo

~35-40 US 30-50 India+ ~100 countries

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

gt500000 ldquoWorkersrdquo

~35-40 US 30-50 India+ ~100 countries

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

gt500000 ldquoWorkersrdquo

~35-40 US 30-50 India+ ~100 countries

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Heinrich Heine Norenzayan 2010

Western

Educated

Rich

Industrialized

Democracies

ldquoa randomly selected American undergraduate is more than 4000 times more likely to be a research participant than is a randomly selected person from outside of the Westrdquo (p 63)

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Easy (and cheap) data

Diverse

Intuition pump

(My) reasons for using crowdsourcing

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

N=126(135 tested)$270 + 10

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Dale amp Lupyan 2011

How acceptable is it to sayHe speeded down the roadHe lighted the candlesThey sneaked aroundetc

r = 2

3 p

= 0

25Lo

git r

eg o

dds 1048576

ratio

= 7

8 p

= 0

28

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

Self reported demographic information from 2896 workers over 3 years (MW lsquo09 MW lsquo11 SW rsquo10)

55 Female 45 Malendash Similar to other internet panels (eg

Goldstein)

Age ndash Mean 30 yrs ndash Median 32 yrs

Mean Income $30000 yr

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Whoworks

ldquoMTurk money is always necessary to make ends meetrdquondash 5 US 13 India

ldquoMTurk money is irrelevantrdquondash 12 US 10 India

ldquoMTurk is a fruitful way to spend free time and get some cashrdquondash 69 US 59 India

Ross et al rsquo10 Ipeirotis rsquo10

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Why

Common TasksImage labelingAudio transcriptionClassification images websitesProduct evaluation

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Uncommon TasksWorkflow optimizationCopy editingProduct descriptionTechnical writing

Wha

t

Companies crowdsourcing part of their businessSearch companies relevanceOnline stores similar products from different stores (identifying competition)Online directories accuracy freshness of listingsResearchers

IntermediariesCrowdFlowerSmartsheetcom

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

WhorequestsCrowdFlower is the leader in enterprise crowdsourcing CrowdFlowerrsquos technology platform offers quality-ensured crowdsourcing at massive scale The company solves problems ranging from product categorization to business lead verification to content creation Clients from startups to the Fortune 500 enjoy increased flexibility faster turnaround time and cost savings

hellipCrowdFlower takes large data-heavy projects and breaks them into small tasks that are distributed to more than 15 million on-demand contributors globally

Whorequests

Iperotis 2010b

Wha

t

Anatomy of a HIT

Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)

HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT

Not everyone is eligible to do every HIT

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Anatomy of a HIT

Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)

HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT

Not everyone is eligible to do every HIT

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Anatomy of a HITHITs with the same title description pay rate etc are the same HIT type(for us itrsquos mostly 1 HIT Hit Type)

HITs are broken up into AssignmentsA worker cannot do more than 1 assignment of a HIT

Not everyone is eligible to do every HIT

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Some HIT groups have many HITs

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Which is the better translation for Taacutey o Blacko Night

HIT 1

Which is the better translation for Nedj o Cleano White

HIT 2

HIT GROUP

Assignment 1 ldquoBlackrdquo

bullbullbull

Assignment 2 ldquoNightrdquo

Assignment 3 ldquoBlackrdquo

Alice

Bob

Charlie

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Which is the better translation for Taacutey o Blacko Night

HIT 1

Which is the better translation for Nedj o Cleano White

HIT 2

HIT GROUP

Assignment 1 ldquoWhiterdquo

bullbullbull

Assignment 2 ldquoWhiterdquo

Assignment 3 ldquoWhiterdquo

Alice

Bob

DavidSlide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Build HIT

Test HIT

Post HIT

Reject or Approve HIT

Search for HITs

Accept HIT

Do work

Submit HIT

Requester Worker

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Pay rate can affect quantity of workPay rate does not have a big impact on quality

Pay per Task Pay per Task

Num

ber o

f Tas

ks C

ompl

eted

Accu

racy

How much $

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Completion Time

3 6-question multiple choice surveys

Launched same time of day day of week$001 $003 $005

Past a threshold pay rate does not increase speed

Start with low pay rate work up

Slide adapted from Winter Masonrsquos presentation httpwwwslidesharenetwinteram

Turker Community

Reputation of Workers is given by approval rating

Requesters can reject workRequesters can refuse workers with low approval ratesWorkers can also rate requesters (stay tuned)

Validity

How

Qualtrics and Mturk

-As good as point and click gets

Number of choices

Attr

ibut

e

FriendlySafe

SmoothRounded

RoundShortFat

SmallWideLargeTallThin

SharpNarrowSpikey

DangerousPointy

UnfriendlyAngular

0 5 10 15 20

Labelcrelchfoove

Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Validity

How

Qualtrics and Mturk

-As good as point and click gets

Number of choices

Attr

ibut

e

FriendlySafe

SmoothRounded

RoundShortFat

SmallWideLargeTallThin

SharpNarrowSpikey

DangerousPointy

UnfriendlyAngular

0 5 10 15 20

Labelcrelchfoove

Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

How

Qualtrics and Mturk

-As good as point and click gets

Number of choices

Attr

ibut

e

FriendlySafe

SmoothRounded

RoundShortFat

SmallWideLargeTallThin

SharpNarrowSpikey

DangerousPointy

UnfriendlyAngular

0 5 10 15 20

Labelcrelchfoove

Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Qualtrics and Mturk

-As good as point and click gets

Number of choices

Attr

ibut

e

FriendlySafe

SmoothRounded

RoundShortFat

SmallWideLargeTallThin

SharpNarrowSpikey

DangerousPointy

UnfriendlyAngular

0 5 10 15 20

Labelcrelchfoove

Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Number of choices

Attr

ibut

e

FriendlySafe

SmoothRounded

RoundShortFat

SmallWideLargeTallThin

SharpNarrowSpikey

DangerousPointy

UnfriendlyAngular

0 5 10 15 20

Labelcrelchfoove

Imagine a shape called a fooveldquo (ldquocrelchrdquo) Would such a shape be

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Imagine that the word lsquocrelchrsquo refers to a shape Draw what you think the shape looks like

Imagine that the word lsquofooversquo refers to a shape Draw what you think the shape looks like

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

If you can code it you can run it

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Test run 1

bull Recruited through social network contactsbull Overall about ~60 players produced 1377

squiggles and ldquolistenedrdquo 4136 timesbull Collected in 1 daybull One user played for 1 hour straight

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Test Run 2

bull 53 participants from Amazonrsquos crowdsourcing service Mechanical Turk

bull 1034 squiggles gt 3000 listensbull Collected in under 1 daybull Some users played for over 30 minutesbull Effective hourly rate

ndash $015 hour

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

TipsBe creative Donrsquot try to force a lab study into a Turk format

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

Thibodeau amp Boroditsky 2011

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

TipsValidate the work when necessary

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

TipsTake the workers perspective

Are you describing your HIT wellAre you paying people fairlyAre you rejecting work fairlyPosting a HIT at 3AM and expecting results

Ask workers for feedback

Be aware of your reputationTurkopticon Turker Nation

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

IRB

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56

(f) Human subject means a living individual about whom an investigator (whether professional or student) conducting research obtains

(1) Data through intervention or interaction with the individual or(2) Identifiable private information

Intervention includes both physical procedures by which data are gathered (for example venipuncture) and manipulations of the subject or the subjects environment that are performed for research purposes

Interaction includes communication or interpersonal contact between investigator and subject

Private information includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place and information which has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public (for example a medical record) Private information must be individually identifiable (ie the identity of the subject is or may readily be ascertained by the investigator or associated with the information) in order for obtaining the information to constitute research involving human subjects

IRB

• Slide 1
• Slide 2
• Slide 3
• Slide 4
• Slide 5
• Slide 6
• Slide 7
• Slide 8
• Slide 9
• Slide 10
• Slide 11
• Slide 12
• Slide 13
• Slide 14
• Slide 15
• Slide 16
• Slide 17
• Common Tasks
• Slide 19
• Slide 20
• Slide 21
• Slide 22
• Slide 23
• Slide 24
• Slide 25
• Slide 26
• Slide 28
• Completion Time
• Turker Community
• Slide 31
• Slide 32
• Slide 34
• Slide 35
• Slide 36
• Slide 37
• Slide 38
• Slide 39
• Slide 40
• Slide 41
• Slide 42
• Slide 43
• Slide 44
• Slide 45
• Test run 1
• Test Run 2
• Tips
• Slide 49
• Slide 50
• Tips (2)
• Slide 52
• Tips (3)
• Slide 54
• Slide 55
• Slide 56