TakingSurveystoPeople’sTechnology:Implica:onsforFederalSta:s:cs

andSocialScienceResearch

Frederick G. Conrad Michael F. Schober

Collaborators: Christopher Antoun, David Carroll, Patrick Ehlen, Stefanie Fail, Andrew L. Hupp, Michael Johnston, Courtney Kellner, Kelly F. Nichols, Leif Percifield, Lucas Vickers, H. Yanna Yan, & Chan Zhang

Support:NSFgrantSES-1025645(Methodology,Measurement,andSta?s?csprogram)

Mobilemul?modalphones(smartphones)

•  Peopleincreasinglycommunicateviasmartphones–  >68%ofUSadultsasof2015–  Con?nuinggrowth

•  Peopleincreasinglyuseandswitchbetweenmul?plemodes(manyna?vetosmartphone)forinterac?ng–  Voice–  Text(SMS)–  Email–  Videochat– Webaccessforcommunica?on(e.g.,blogposts)–  Specializedappsforcommunica?on(e.g.,Facebook)

Peopleincreasinglyexpectcapabilityto:•  communicatewhilemobileand/ormul?tasking•  chooseamodethatfitstheircurrentseWngandneeds

–  e.g.,urgentvs.canwait,publicvs.private,noisyvs.quiet,brightvs.dim•  switchmodeswhilecommunica?ng•  respondinadifferentmodethancontacted

–  e.g.,respondtovoicemailwithatext

Newpossibili?esandchallengesforsurveydatacollec?on

•  AsR’sexpectmul?plemodesonthesamedevice,mayexpectthatsurveysaremul?modal–  Poten?altointeractviaSMStextwhenconvenient–  Poten?altorespondinmodethatisappropriatetocurrentseWng

(e.g.,textinnoisyenvironment,voicewhenthereisglare,etc.)–  Moregenerally,tobeabletorespondinanymode,any?me,

anywhere

•  Dopeoplerespondtoconven?onalsurveymodes(e.g.,telephoneinterviews)inthesamewayonsmartphonesasonlandlines?

•  Howdopeoplerespondtolessconven?onalsurveymodesthatusesmartphonecapabili?es?

Newop?onsforsurveymodechoice•  Nowpossibletochooseamodeonasingledevice,immediatelyandconveniently

•  Quitedifferentfrompriorimplementa?onsofsurveymodechoice– WhenRinvitedbymailcompleteseitheronpaperorweb,thisrequiresextrastepoftypingURLintobrowser

–  Canreduceresponserates(e.g.,Fulton&Medway,2012)

•  Choiceonsingledevicemayleadtodifferentoutcomes

Modecomparisonstudy(Schoberetal.,2015,PLOSONE)

•  examines–  dataquality(sa?sficing,disclosure)–  comple?onrates–  respondentsa?sfac?on

•  fourexis?ngorplausiblesurveymodesthatworkthroughna?veappsontheiPhone– Asopposedtospeciallydesignedsurveyapps– Asopposedtowebsurveyinphone’sbrowser– UniforminterfaceforallRs

•  Asopposedtomixofplacorms(Android,Windows,etc.)

Experiment:4modesoniPhone

Medium

Voice SMSText

Interviewing

Agent

Human

Humanvoice

(RspeakswithI)

Humantext

(RtextswithI)

Automated

SpeechIVR

(Rspeakswithsystem)

AutomatedText

(Rtextswithsystem)

Surveysviatextmessaging?•  Moreandmorepeopleareembracingtextmessagingforpersonalandprofessionalcommunica?on–  ontheirmobilephones(smartphonesornot)–  onotherdevices(e.g.,tablets,desktops)

•  Tex?ngisbecomingapoten?allyimportantwaytoreachrespondents–  somemayaeendtotextmorethantoemailsorvoicemails

–  respondentsmayexpecttobeabletopar?cipateinasurveyviatext

•  Someorganiza?onsarenowincludingSMStextintheirsuiteofmodesformobilesurveys–  e.g.,GeoPoll,PollEverywhere,iVisionMobile,etc.

Textasamodeofinterac?on•  Turn-by-turn

–  Threaded(onasmartphone)

•  Responsesdon’tneedtobeimmediate–  Allowsmul?tasking

•  Worksevenwithintermieentnetwork/cellservice–  unlikevoice

•  Doesnotrequirewebcapacityondevice–  unlikemobilewebsurvey

Property Voice Text Synchrony Fully synchronous Less or asynchronous Medium Auditory Visual Language Spoken/heard Written/read Conversational structure

Turn-by-turn, with potential for simultaneous speech

Turn-by-turn, rarely but possibly out-of-sequence

Persistence of turn No Yes Persistence of entire conversation No Yes, threaded

Social presence of partner

Continuous (auditory) presence

Intermittent evidence (when texts arrive)

Character of multitasking

Simultaneous, especially when hands free, unless other task involves talking

Switching required between texting and other tasks

Impact of environmental conditions

Potential interference from ambient noise

Potential interference from visual glare

Impact of nearby others

Others may hear answers; potential audio interference from others’ talk

Others unlikely to see text and answers on screen, though possible

t

Measuresofdataquality•  Conscien/ousresponding(lesssa?sficing)

–  Rsareknowntotakeshortcuts—to“sa?sfice”—differentlyindifferentsurveymodes

•  e.g.,Chang&Krosnick(2009),Heerwegh&Loosveldt(2008)–  Weexamine

•  roundednumericalresponses(e.g.,mul?plesof10)–  Unroundedanswersaremorelikelytoresultfromdeliberate,memory-basedthoughtprocessesthan

es?ma?on(Brown,1995;Conrad,Brown,&Dashen,2003)–  morelikelytobeaccurateinanswerstoobjec?vefactualques?ons(Holbrooketal.,2014)

•  straightlining(nondifferen?a?on)–  givingsameanswertobaeeryofQs

•  Disclosure(moreisbeeer)–  Rsopendisclosemoresensi?veinforma?onwhentheyself-administeraques?onnaire(websurveys,ACASI)

•  e.g.,Kreuteretal.(2009),Tourangeau&Smith(1996)

•  Par/cipa/onandcomple/on

Possibleoutcomes:Conscien?ousresponding

•  TEXTVS.VOICE–  Rsmightbelessconscien>ousintextbecausetheyimport“leasteffortstrategy”fromhowtheyusuallytext

– ORRsmightmoreconscien>ousintextbecausetheyfeelless?mepressuretorespondthaninspokeninterviews

–  andcananswerwhentheyareready•  HUMANVS.AUTOMATED

–  Rsmightbelessconscien?ouswithautomatedinterviewer(selfadministra?on)becausethereisnohumantomo?vatethemtobeconscien?ous

Possibleoutcomes:Disclosure•  TEXTVS.VOICE

–  Rsmightdisclosemoreintextbecauseoffewersocialcuesintheinterac?on

•  lessevidenceofreac?ontoanswers?•  more?metobecomfortablewithanswers?•  nooneelsecanheartheques?onsoranswers?

–  ORRsmightdiscloselessintextbecause•  theyworrythatothersmightseevisuallypersistentanswers?•  theyworrythatanswersarepermanentlystored?•  theycantake?metoanswerinwaysthatgivethebestimpression?

•  HUMANVS.AUTOMATED–  Rsmightdisclosemorewithautomatedinterviewer,asinACASIorwebsurvey

Items•  First,safe-to-talkorsafe-to-textques?on•  32QstakenfrommajorUSsocialsurveysandmethodological

studies–  E.g.,BRFSS,NSDUH,GSS,PewInternet&AmericanLifeProject–  Formost,knowntohaveproduceddifferencesinsa?sficingor

disclosurebetweenconven?onalmodes•  Yes/no,numerical,categorical,baeeryitems(seriesofQs

withsameresponseop?ons)•  Ra?onaleforinclusion

–  Qswithmoreandlesssociallydesirableanswers•  e.g.,sexualhistory,druguse,newspaperreading

–  Qsforwhichfrequencyreportscouldbepreciseores?mated(rounded)

•  e.g.,numberofmoviesseenlastmonth,numberofappsoniPhone–  BaeeryQ’sthatcouldproducestraightlining(non-

differen?a?on)

Implementa?on:Humanvoice•  8interviewers(Is)fromUMichsurveyresearchcenter

•  customdesignedCATIinterfacethatsupportsvoiceandtextinterviews(PAMSS)

Implementa?on:Humantext•  Same8IsfromUMichsurveyresearchcenter•  SamecustomdesignedCATIinterface

–  Iselects,edits,ortypesques?ons/prompts,andclickstosend•  Textmessagessentthroughthirdparty(Aerialink)•  Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),ornumber

Implementa?on:SpeechIVR

•  Custombuiltspeechdialoguesystem•  UsesATT’sWatsonspeechrecognizer,Asterisktelephonygateway

•  Recordedhumaninterviewer,speechresponses(nottouchtone)

Implementa?on:Auto-text•  Custombuilttextdialoguesystem•  Textmessagessentthroughthirdparty(Aerialink)•  Rscananswerwithsinglecharacter:Y/N,leeer(a/b/c),or

number

Respondents:634iPhoneusers•  n=157to160randomlyassignedtoeachmode•  RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk– Webscreenerverifiedage(>21years)andUSareacode–  iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse

•  $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire

•  Age,gender,ethnicity,income,educa?onnotreliablydifferentinfourmodes

•  SomewhatyoungerandlessaffluentthanUSna?onaliPhoneusers

TextRespondent

22

HumanTextInterviewerInterface

23

Datacollec?on

•  InterviewscarriedoutMarch-May2012

•  Resultsbasedonspeech-IVRsystemrecogni?on– 95.6%correctrecogni?onaccuracybasedontranscripts

– Samepaeernofresultsifweusehumanannota?ons(Johnston,etal.,2013)

Percent respondents reporting rounded numbers of… Human Auto Human Auto Estimate SE

Odds ratio Estimate SE

Odds ratio Estimate SE

Odds ratio

Movies seen in theaters in past 12 months 24.4% 18.2% 17.1% 12.1% -0.463* 0.211 0.630 -0.383† 0.210 0.682 -0.035 0.425 0.965

Songs on iPhone 66.9% 61.8% 45.2% 51.6% -0.655*** 0.163 0.520 0.026 0.163 1.026 0.478 0.326 1.612

Apps on iPhone 80.6% 78.6% 47.5% 54.1% -1.332*** 0.179 0.264 0.112 0.175 1.119 0.391 0.358 1.479

Text messages sent and received on iPhone in current billing cycle 91.1% 90.1% 73.2% 70.5% -1.331*** 0.233 0.264 -0.125 0.214 0.882 -0.019 0.466 0.981

Times they ate spicy food in last month 46.5% 41.2% 38.6% 52.2% -0.325 0.228 0.640 -0.218 0.229 0.804 0.771* 0.323 2.162

Movies watched in any medium in last month 30.6% 40.9% 32.9% 30.6% -0.179 0.168 0.836 0.179 0.168 1.196 -0.557† 0.338 0.573

Times they shopped in a grocery store in last month 33.8% 41.0% 29.1% 35.0% -0.236 0.168 0.790 0.293† 0.168 1.340 -0.039 0.336 0.961

Times they ate in restaurants in last month 39.4% 36.7% 35.4% 36.9% -0.080 0.165 0.923 -0.025 0.165 0.975 0.178 0.329 1.195

†=p

Conscien?ousresponding:Straightlining

•  Q:supportforvariousdietaryprac?ces(ea?ngredmeat,limi?ngfastfood,etc.)

»  stronglyfavor»  somewhatfavor»  neitherfavornoroppose»  somewhatoppose»  stronglyoppose

•  Wedefineanswersinbaeeryas“straightlining”whenatleast6of7responsesarethesame

•  Significantlylessstraightliningintextthanvoice

37

Table 5. Disclosure effects for each question.

Percent people reporting… Human Auto Human Auto Estimate SEOdds ratio Estimate SE

Odds ratio Estimate SE

Odds ratio

Having smoked at least 100 cigarettes in their entire life 39.2% 34.0% 42.4% 50.3% 0.404* 0.162 1.497 0.054 0.162 1.055 0.547† 0.326 1.727

Exercising less than 1 time per week in a typical week 13.1% 12.6% 21.5% 29.3% 0.838*** 0.212 2.312 0.239 0.206 1.270 0.462 0.425 1.587

Having had 3 or more sexpartners in the last 12 months 7.6% 10.1% 13.6% 14.3% 0.520* 0.257 1.681 0.160 0.254 1.174 -0.261 0.518 0.770

Personally watching television for five or more hours on the average day 10.7% 9.5% 15.9% 15.3% 0.499* 0.243 1.647 -0.083 0.239 0.921 0.084 0.486 1.087

Having had one or more drinks of analcoholic beverage on morethan 15 days of the past 30

10.6% 11.4% 8.2% 19.1% 0.247 0.244 1.280 0.546* 0.248 1.727 0.890† 0.504 2.436

Never attending religious services 32.7% 44.7% 37.6% 44.0% 0.088 0.163 1.092 0.385* 0.164 1.469 -0.243 0.327 0.785

Never reading the newspaper 16.9% 29.6% 14.6% 27.4% -0.134 0.194 0.875 0.759*** 0.198 2.136 0.069 0.397 1.071

Smoking every day 13.8% 13.2% 9.5% 16.6% -0.040 0.235 0.960 0.283 0.236 1.327 0.684 0.477 1.892

Having ever, even once,used marijuana or hashish 58.8% 54.7% 65.0% 61.9% 0.281† 0.163 1.324 -0.148 0.163 0.862 0.034 0.326 1.034

Having had 5 or more drinks on the same occasion on more

than 3 days of the past 3010.6% 12.0% 8.9% 11.5% -0.115 0.257 0.892 0.202 0.258 1.224 0.147 0.517 1.159

Having had more than 30 female partners since their 18th birthday (among straight men

and homosexual or bisexual women)16.1% 11.0% 10.3% 9.3% -0.366 0.346 0.694 -0.299 0.344 0.741 0.334 0.698 1.396

Having had more than 25 male partners since their 18th birthday (among straight women

and homosexual or bisexual and men)9.7% 9.1% 10.5% 12.0% 0.203 0.381 1.224 0.046 0.380 1.047 0.222 0.763 1.248

Having had sex 4 or more times a week during the last 12 months 3.9% 9.7% 9.7% 9.7% 0.391 0.297 1.479 0.391 0.297 1.479 -0.999 0.628 0.368

Describing themselves as homosexual,gay, lesbian, or bisexual 9.5% 10.8% 7.1% 10.9% -0.134 0.272 0.875 0.297 0.274 1.345 0.338 0.551 1.403

†=p

Whataccountsfortextvs.voicedifferencesinprecisionanddisclosure?•  Couldbeanyorallofthemanydifferencesin/mingandbehaviorbetweentextandvoiceinterviews–  aloneorincombina?on

•  Plausiblecontribu?ngfactorsinclude:–  Textreducesimmediate?mepressuretorespond,soRhasmore?metothinkorlookupanswersàCouldexplaingreaterprecision(lessrounding)intext

–  Textreduces“socialpresence”•  ReducedsalienceofI’sabilitytoevaluateorbejudgmental?•  NoimmediateevidenceofI’sreac?on?àCouldexplainmoredisclosureintext

Experimentaldesignhelpsruleinorruleoutaccounts

•  e.g.,maybeR’sroundlessintextbecausetextI’sneverlaugh(noLOL’sorhaha’s)– Maybelaughterinvoiceinterviewssuggeststhatcasualresponsesaresufficient

–  Butthatcan’tbeitbecauseR’sroundjustasmuchinHumanandAutoVoiceinterviews,andautomated“interviewer”neverlaughed

0"

0.5"

1"

1.5"

2"

2.5"

3"

3.5"

Text" Voice"

Human"

Automated"

Examples:Textvs.voiceinterac?ons

HUMANTEXT HUMANVOICE

1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?

1 I: Duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.

2 R: 3 2 R: OH,GOD.U:hman.That’salot.HowmanymoviesIseen?Like30.

3 I: 30.

Totalelapsed>meun>lnextQ:1:21 0:12

Examples:Textvs.voiceinterac?onsHUMANTEXT

1 I: Duringthelastmonthhowmanymoviesdidyouwatchinanymedium?

2 R: Medium?

3 I: Here’smoreinforma?on.Pleasecountmoviesyouwatchedintheatersoranydeviceincludingcomputers,tabletssuchasaniPad,smartphonessuchasaniPhone,handheldssuchasiPods,aswellasonTVthroughbroadcast,cable,DVD,orpay-per-view.

4 R: 3

Totalelapsed>meun>lnextQ:2:00

HUMANVOICE

1 I: *Duringthelast*

2 R: Huh?

3 I: Oh,sorry.Um,duringthelastmonth,howmanymoviesdidyouwatchinANYmedium.

4 R: Oh!Let’ssee,whatdidIwatch.Um,shouldIsayhowmanymoviesIwatchedorhowmanymovieswatchedme?[laughs]Allrightlet’s-letmethinkaboutthat.IthinkyesterdayIwatchedu:m,notinitsen?retybutyouknow,comingandgoing.Mykidsarewatchingin.Um,Idon’tknowmaybe2or3?mesaweekmaybe?

Examples:Textvs.voiceinterac?onsHUMANVOICE

5 I: Uh,sowhatwouldbeyourbestes?mateonhowmany,um,yousawinthewholemonth.

6 R: [pause]Um,Idon’tknowI’dsaymaybe3moviesifthatmany.

7 I: 3?

8 R: Isthatgoingtothemoviesorwatchingthemoviesontv.Likeyousaid*anymedium*right?

9 I: That’s*anymovies.*Yep.

10 R: Maybe1or2amonthI’dsay.

11 I: 1or2amonth?[breath]Uh,sowhatwouldbe*closer*

Examples:Textvs.voiceinterac?onsHUMANVOICE

12 R: *Yeah,because*Iuh,um,occasionallyItakethekidsonaTuesdaytoseeamovie,dependingonwhat’splaying.SoI’dmaybeonceortwiceamonth

13 I: Whichwouldbecloser,onceortwice.

14 R: Iwouldsaytwice.

15 I: Twice?

16 R: R:Mhm.Becauseitruns4Tuesdayswhichischeapertogo

17 I: Right

18 R: R:soI’dsaytwice,yah.BecauseIdotakethemtwice.Notlastmonthbutthemonthbefore

Totalelapsed>meun>lnextQ:ß

1:36

ExamplesfromAutomatedVoiceI:Howopendoyoureadthenewspaper?'Everyday','afew?mesaweek','onceaweek','lessthanonceaweek',or'never'?R:EverydayI:Gotit.I:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:NoneI:Ithinkyousaid'9'.Isthatright?YesorNo.R:NoI:Thinkingaboutthe?mesinceyoureighteenthbirthday(includingtherecentpastthatyou'vealreadytoldusabout),howmanymalepartnershaveyouhadsexwith?R:ZeroI:Thanks

Numerical

FirstHypothesis:“Nine”

LastHypothesis:“Zero”

LastAnnota?on:“Zero”

Categorical

ExplicitConfirma>on

Median?meperques?onandanswer

Mul?tasking(self-reported)

Cannonresponseornoncomple?onexplainthepaeerns?

•  Werepeoplewhoweremoreconscien?ous(lesslikelytoroundtheiranswers)ormorewillingtodisclosesensi?veinforma?onactuallylesslikelytostartorfinishinterviewsinvoicemodesthanintextmodes?

•  Couldourmodeeffectsresultnotfromthecontribu?onofrespondersandcompleters,butinsteadfromthenon-contribu?onofnon-respondersandnon-completers?

Studydesignallowslookingatthisinafocusedway

•  alloursamplemembershadalreadyindicated,byscreeningintothestudy,interestinandatleastsomecommitmenttopar?cipa?nginaninterviewontheiriPhone(inanunspecifiedinterviewmode).

•  Thefactthatourpar?cipantswererandomlyassignedtoaninterviewingmodemeansthattheirini?a?vewasunlikelytohavedifferedacrossthemodes.

Nonresponse?

•  noevidencethatdifferentkindsofpeople(age,gender,ethnicity,race,educa?on,income)fromoursamplewereanymoreorlesslikelytostarttheinterviewsinthedifferentmodes

•  Implausiblethatanotherfactorcouldexplainpaeern:– wouldrequirethattendencyofRstogiveimpreciseanswersandreluctancetoengageinatextinterview(butwillingnesstoengageinavoiceinterview)wouldhavethesameorigin

Noncomple?on?•  Comple?ongreaterinhumanthanautomatedinterviews

•  Nodifferencebetweentextandvoice•  àUnlikelytoaccountforvoicevs.textdifferences•  Fornoncomple?ontoaccountfordisclosure,wouldrequireasystema?creversalofthepaeernofdisclosureobservedforthosewhocompletedandthosewhobrokeoff–  thosewhobrokeoffwithautomatedinterviewerswouldhavetobethosewhohadlesstodisclose

–  Butonewouldthinkthatpeoplewhobreakoffwouldbethosewithmoretodisclose

Prefertext(vs.voice)forfutureiPhoneinterview?

0

10

20

30

40

50

60

70

80

90

100

Text Voice

Percen

t Human

Automated

Othersa?sfac?onmeasures

•  MostRsfoundinterviewveryorsomewhateasy– Morefoundspeech-IVRsomewhathard

•  Futureinterviews:– TextRsoverwhelminglypreferredfutureinterviewintextvs.voice

– VoiceRspreferredvoice,butlesssoifspeech-IVR

Summary:Voicevs.Text•  Textinterviewsproducehigherdataquality:greaterdisclosure,lesssa?sficing,highsa?sfac?on

•  Eventhough(orbecause?)theytakelonger•  Eventhoughdataarelesssecure(morepersistentandtraceable)thanvoice–  Perhapsbecauseofdifferent?mepressurethanvoice?–  PerhapsbecauseofconvenienceofansweringwhenandhowRwants?

–  Perhapsbecauseofgreatersocialdistancewithinterviewer?

•  Caveat:weimplementedtextinterviewsinonepar?cularway,withsingle-characterresponses

Summary:Humanvs.AutomatedInterviewer

•  Automatedinterviewsonasmartphone(inthesemodes)canleadtodataatleastashighinqualityasdatafromhumaninterviewsinsamemodes–  Nomoresa?sficingthanwithhumaninterviewers!–  Moredisclosure

•  Tradeoffs–  Fieldperiodcanbeshorter–  interviewscantakelonger–  Higherbreak-off–  requireaddi?onaldevelopmenteffort,especiallyspeech-IVR

•  Caveat:weimplementedonepar?cularversionofspeech-IVR;otherscoulddiffer

ModeChoiceStudyConradetal.(underreview)

•  Ismessage–  Urgentorcanitwait?–  Sensi?veornot?–  Shortvs.long?

•  WillIbemul?tasking?Ifso,whatelsewillIbedoing?•  Whatmodewillbeeasiestorleastdisrup?veforpartner?•  IsseWngpublicvs.private,noisyvs.quiet,brightvs.dim?•  Whatismygenerally(chronically)preferredwayofcommunica?ng?

–  e.g.,talkingvs.tex?ng

•  Sopeoplecanusethesamedevice,forexample,torespondto–  avoicecallwithatextmessage–  atextmessagewithaFacebookpost–  emailwithavoicecall

48

Onsmartphones,peoplechooseandswitchmodestofitneeds

Implica?onsforSurveyPrac?ce

•  Nowpossibleformembersofpublictochooseoneofmanysurveymodesonasingledevice–  immediatelyandconveniently

•  Notofferingachoicecoulddeterpar?cipa?onbysmartphoneusers– orreducemo?va?onwhenansweringques?ons

49

Inothertasks,choiceseemstohelpandhurt

•  Choiceenhancesintrinsicmo?va?on(byincreasingautonomy)andperformance–  Patalletal.(2008)meta-analysis:78of91effectsofchoiceonintrinsicmo?va?onareposi?ve

•  Toomanyop?ons(overload)leadstonochoice(paralysis)andreducedsa?sfac?onwithchoices–  IyengarandLepper(2000):par?cipantsmorelikelytopurchasegourmetjams/chocolatesortocompleteop?onalassignmentswhenoffered6vs.24or30choices

•  Howdoeschoiceaffectsurveypar?cipa?on?

50

SurveyModeChoice•  Toincreasepar?cipa?on,researchersofferpoten?alrespondentsachoiceofmodes–  e.g.,mailpaperques?onnaireandgiverandomhalfchoiceofcomple?ngonline;requiresextrastepoftypingURLintobrowser

•  Butthiskindofchoiceseemstoreducepar?cipa?on:–  Fulton&Medway(2012)meta-analysisof19mail/webchoicestudiesfindsthat,comparedtonochoice,modechoicereliablyreducespar?cipa?onby3.8%

–  suggestcouldbeduetoParadoxofChoice(Schwartz,2009)orcostsofswitchingfrominvita?ontointerviewmode

•  Choiceonsingledevicesimplifieschoiceimplementa?on

51

Currentstudy

•  Examineshowmodechoiceonasingledeviceaffects– Par?cipa?on– Dataquality(rounding,straightlininganddisclosure)

– Rsa?sfac?on

•  Same4modes,same32items

52

Possibleoutcomes:Par?cipa?on

•  IfRscanChoose– Mightreducepar?cipa?onbecause

•  Increasedcomplexity(Schwartz,2004;FultonandMedway,2012)•  Breakinresponseprocess(Fulton&Medway,2012)

– Mightincreasepar?cipa?onbecause•  Canchooseamodethatissuitablegiventheircurrentenvironmentandotherdemands(e.g.,whethertheycantalknow)

Possibleoutcomes:Conscien?ousresponding

•  IfRscanChoose– mightprovidefewerconscien?ousanswersbecausetheychooseamodeinwhichit’seasiertotakeshortcuts

•  e.g.,anautomatedmodebecausenohumaninterviewertopressthemtoworkhard

– mightprovidemoreconscien?ousanswersbecausebeingabletochoosemayincreasetheircommitmenttothetask

•  Mayincreasemo?va?on

Possibleoutcomes:Disclosure

•  IfRscanChoose– mightdisclosemorebecausechoosemoreprivatemodewithfewersocialcues

•  e.g.,Automatedtext– mightdiscloselessbecausechoosemoreconvenient,fastermodewithmoresocialcues

•  E.g.,humanvoice

Possibleoutcomes:Sa?sfac?on

•  IfRscanChoose– Mightreducesa?sfac?onbecause

•  Addingop?onsincreasesR’sexpecta?ons(Schwartz,2004)•  Leadstoregretovernotchoosingimaginedalterna?ve(Schwartz,2004)

– Mightincreasesa?sfac?onbecausepeopleperceivethechosenalterna?veasmoreaerac?ve(Fes?nger,1948;Cooper,2007)

– Orjustbecausemoreconvenientandeasier!

ExperimentalCondi?ons

1.  AssignedMode(NoChoice)•  Rsrandomlyassignedtoamode•  Contactedandinterviewedinsamemode

2.  Choice•  Rsrandomlyassignedtoacontactmode•  Requiredtochooseinterviewmode

–  Couldchoosecontactmodeoranyofotherthree–  Makesexplicittheirmodechoiceinten?on

57

ModeComparisonExperiment

ModeChoiceDesignandImplementa?on(2)

•  ModeChoiceintroduc?on:“Togetstarted,weneedyoutochoosehowyouwanttobeinterviewed--whateverworksbestforyou.Therearefourchoicesandanychoiceisfinewithus.Doyouwantto‘talkwithaperson’,‘talkwithanautomatedinterviewer’,‘textwithaperson’,or‘textwithanautomatedinterviewer’?

•  Withineachcontactmode,orderofinterviewmodeop?onsrotatedacrossRs(16orders)

58

Respondents:1260iPhoneusers•  AssignedMode(NoChoice):n=634

–  n=157to160permode–  InterviewedMarch–May,2012

•  Choice(AbletoChooseInterviewMode):n=626–  n=149to170permodeofcontact–  InterviewedJuly–September,2012

•  RecruitedfromCraigslist,Facebook,GoogleAds,andAmazonMechanicalTurk–  Webscreenerverifiedage(>21years)andUSareacode–  iPhoneusageverifiedviatextmessagetodeviceanduseragentstringinresponse

•  $20iTunesgipcodeincen?ve,providedaperpost-interviewwebques?onnaire

•  Age,gender,ethnicity,income,educa?onnotreliablydifferentbetweenAssignedModeandModeChoicegroups

•  13Umich/SRCinterviewers:–  5onlyinAssignedModecondi?on,3onlyinModeChoice,5inbothcondi?ons

59

Par?cipa?on•  Doessimplybeingpresentedwithachoicereducepar?cipa?on?

–  SlightlyfewerRsinModeChoicecondi?on(52.1%)choseamodethanansweredthefirstques?oninAssignedModecondi?on(55.9%).

•  Doeschoosingamodereducecomple?on?–  Overall,46.4%ofRsinChoicecondi?onvs.50.5%inAssignedMode

condi?oncompletedques?onnaire(RR1)

•  Inallcases?–  WhenRschosetostayincontactmode48.3%completedinterview,

notdifferentfrom50.5%AssignedMode–  Modechoiceinautomatedmodeshasnoimpactoncomple?on

(43.4%vs.44.0%)

•  Howdoesmodechoiceaffectbreakoffs?–  MoreRswhochoseaninterviewmodecompletedtheinterview

(94.9%)thanthosewhowereassignedamode(90.4%)

YES

NO

ITREDUCESBREAKOFFS

YES

Par?cipa?on

55.9%

90.4%

48.9%

94.9%

0102030405060708090

100

Startinterview(answerQ1) Completeinterviewoncestart

Percen

t

AssignedMode

ModeChoice

•  Overallcomple?onhigherwithout(50.5%)thanwithchoice(46.4%)•  Noimpactonkindsofpeoplewhopar?cipatesochoiceprobablydoesnot

introducenonresponsebias 61

Breakoffaperchoicebutbeforeinterview

0.7%

11.1%

0

2

4

6

8

10

12

StayinMode(n=301) SwitchMode(n=388)

%don

’tan

swerQ1aW

er

choo

singm

ode

62

Bothgroupsmakechoicesoincreasedbreakoffswhenchoicerequiresswitchingmodesduetotoswitchingcosts,notParadoxofChoice

HumanVoice HumanText

Whatmodeswerechosen?

0

50

100

150

200

250

300

SwitchintoMode

StayinMode

AutomatedText

n=170

n=150 n=157 n=149

OriginalSampleSize(beforemodechoice)

AutomatedVoice

Num

bero

fRs

63

DataQuality:Rounding

•  Wedefineroundinghereasnumericalanswersdivisibleby10– HowmanysongsdoyoucurrentlyhaveonyouriPhone?

•  Exampleroundedanswer:1100•  Exampleunroundedanswer:1126

64

AverageNumberofRoundedNumericalAnswers

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

Effectofchoicenotduetopar?cularchoiceofmode:lessroundingwithchoicethanwithoutapercontrollingformode,p=0.008

2.58

p

Rounding:“Numberof?mesea?nginrestaurants”

0

5

10

15

20

25

AssignedMode ModeChoice

Percen

tRsrep

or>n

groun

ded

answ

er

*During the last month, how many times did you eat in restaurants?

p<0.01

66

Rounding:“NumberofsongsonyouriPhone”

40

42

44

46

48

50

52

AssignedMode ModeChoice

Percen

tRsrep

or>n

groun

ded

answ

er

*How many songs do you currently have on your iPhone?

p=0.02

67

PercentofRsstraightlining

0%

2%

4%

6%

8%

10%

12%

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

Effectofchoicenotduetopar?cularchoiceofmode:marginallylessstraightliningwithchoicethanwithoutapercontrollingformode,p=0.085

6.78%

p=0.029

3.99%

68

AveragenumberofSociallyDesirableAnswers

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

AssignedMode ModeChoice

Voice

Text

Human HumanAutomated Automated

2.71 2.83

Num

berofAnswers

1

Figure4.Disclosure:Averagenumberofresponsesdeemedmostsociallyundesirable:abovethetopdecileforconKnuousnumericalresponsesorthemostextremecategoricalresponseopKoninthesKgmaKzeddirecKon.

Responsesdeemedmostsociallyundesirable:abovetopdecileforcon?nuousnumericalresponsesormostextremecategoricalresponseop?onins?gma?zeddirec?on

Sa?sfac?onhigherwithmodechoice

0

10

20

30

40

50

60

70

80

AssignedMode ModeChoice

%Verysa>sfie

d

p<0.001

Overall,howsa?sfiedwereyouwiththeinterview?

70

Reasonsforchoosingmodes

*Why did you choose this interviewing method?

Mostcommoncategories %ProvidingReason

Ease/simplicity 33.8%Convenience/flexibility 22.8%Quickness(shortestinterview?me) 10.3%Privacy 9.8%Liketex?ng 9.0%Environment--loca?on 8.8%

Threecoders;Agreement=98.1%

•  Codedopen-endedanswersinto29categories

71

Reasonsforchoosingmodes*(examples)

•  Humanvoice:–  “Morecomfortablespeakingwitharealperson”

•  Humantext:–  “IchosetotextbecauseIhadasmallchildwithmeinmyhomeduringthe

interviewandcouldnothaveconcentratedontheques?onsifitwasonthephone.”

–  “Toavoidbackgroundnoiseandtoclearlyunderstandtheques?onandtakemy?metoanswerit.”

•  Automatedtext:–  “Iamatworkandwouldn'talwaysbeabletoanswerques?onsifIspoketo

someoneonthephone.”–  “BecauseIdidn'twanttotalkonthephoenordidIwanttotextaperson

simplybecausIknewsomeofmyresponseswouldhavebeenalielelate”•  AutomatedVoice:

–  “ididn'twanttotalktoanyonebut,IwasdrivingsoIcouldn'tlookatascreen”

–  “Talkingtoanautomatedpersonwaslesspersonal”

*Why did you choose this interviewing method? 72

Summary•  Modechoiceproduced:

–  lessrounding–  lessstraightlining–  fewerbreakoffs–  higherRsa?sfac?on

•  Choicedidnotaffectpaeernsoftextvs.voiceforrounding,straightlining,disclosure

•  Par?cipa?on–  Lowerstartandcomple?onrateswithchoicethannot– Mostlyduetowhetherchoiceinvolvesmodeswitch–  Rswhostartaperchoosingmodemorelikelytocomplete

73

Manyques?onsremain(overall)•  Dodifferentdemographicsubgroups(e.g.,age,income,educa?on)varyindisclosure,effort,preferences?

•  Generalizabilitytoothermobileplacorms?Tolesssmartmobilephones?

•  Generalizabilitytoanon-convenienceornon-incen?vizedsample?

•  Dorespondentswanttobeabletoswitchmodesmid-interviewwhencircumstanceschange(mobile,noisy,private,etc.)?

•  HowmanyQscanbeaskedviatextinterviews?

Implica?ons

•  Tex?ngisworthexploringfurtherasamodeofsurveydatacollec?onforFederalsta?s?csandsocialscienceresearch

•  Asynchronous,less-?me-pressuredrespondingmayreallybebeeerthanusualmodes–  Raisesques?onofwhetherFTFortelephonemodesshoulds?llbeconsideredthegoldstandardinasmartphoneera

– Andwhether“best”modevariesfordifferentresearchques?onsorpar?cipants

Implica?ons(2)

•  Mul?taskingwhileansweringsurveyques?onsdoesnotnecessarilyleadtopoorerdataquality

•  Maywellenhancerespondents’sa?sfac?onandwell-beingbyallowingthemtorespondwhereandwhentheyfinditconvenient

Implica?ons(3)

•  Poten?albenefitsofautoma?onforsocialmeasurementextendtotheuseofapersonalportabledevicesdespitethevaryingcontexts(publicandprivate)inwhichthedeviceisused

Implica?ons(4)

•  Offeringrespondentsamodechoiceonasingledevicemayhaveimportantbenefits

•  Butnotallmodetransi?onsarethesame•  Differentdesignsolu?onswillbeneededfordifferentmodetransi?ons

78

Specula?on:ANewTakeonStandardiza?on

•  Shouldourtradi?onalone-size-fits-allapproachtocollec?ngself-reportdataberethought?

•  Maybedifferentmodesfordifferentpeopleondifferentoccasionscanincreasecomparabilityoftheirresponses

•  Maybewhatisneededisstandardizingpar?cipants’experience–  enhancingeveryone’sabilitytofocusonthetaskinawaythatsuitstheirpreferencesandcircumstances

•  Smartphones–mul?modal,mobiledevices–maybeforcingustothinkthisway

Thankyou!

80