The Future of Survey Research March 2017

CLICK TO EDIT MASTER TITLE STYLE

THE PAST, PRESENT & FUTUREOF TRADITIONAL SURVEY RESEARCH

MARCH 2017

[email protected] | 484-840-4406 | @ddutwin

David Dutwin, Ph.D.SSRS EVP & Chief Methodologist

CLICK TO EDIT MASTER TITLE STYLETHINGS THAT MAKE YOU GO…

• Metrics in survey research are no different than other fields: cost and quality…cost we

know but what about quality?

• Response rates have dropped substantially: canary in a coal mine?

• Are non-probability panels the future?

• Can telephone last?

• What the hell just happened (re: election)?!?!?

• Where is traditional survey research in 5, 10 years?

© S S R S | A L L R I G H T S R E S E R V E D 2

CLICK TO EDIT MASTER TITLE STYLEA MATTER OF PAPERS

• RQ #1: What exactly has happened to telephonic survey response in the past decade?

• “Trends in Telephone Outcomes, 2008 - 2015.” Survey Practice, (2016) (D. Dutwin, P.

Lavrakas).

• RQ #2: Has the answer to RQ #1 done anything to data quality?

• “Telephone Sample Surveys: Dearly Beloved or Nearly Departed? Trends in Survey Errors

in the Age of Declining Response Rates.” (under peer review, 2017) (D. Dutwin, T.

Buskirk).

• RQ #3: Where do we stand with regard to low response rate probability versus

nonprobability?

• “Apples to Oranges or Gala versus Golden Delicious? Comparing Data Quality of Non-

Probability Internet Samples to Low Response Rate Probability Samples.” Public Opinion

Quarterly, (Special Issue on the Future of Survey Research, in press 2017) (D. Dutwin, T.

Buskirk).



FIRST, WHAT IS THE STORY OF TELEPHONE

RESPONSE IN THE LAST DECADE?(PLEASE DO NOT GET TOO DEPRESSED…)

CLICK TO EDIT MASTER TITLE STYLEDECLINING RESPONSE RATES

0%

5%

10%

15%

20%

25%

30%

35%

40%

1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015

Response Rates, 1997 - 2015

ABC Pew CBS


CLICK TO EDIT MASTER TITLE STYLETRENDS IN TELEPHONE DISPOSITIONS I N T H E A G E O F C E L L P H O N E S

Data Study Scrub LLScrub

Cell

First

Year

Last

Year

LL

Sample

Cell

Sample

ABC ABC Polls Biz Purge 2010-2015 None 2008 2015 259,677 188,177

Gallup Gallup Daily Tracking Surveys None None 2009 2015 18,490,017 14,465,292

GfK AP Polls Aug 2012 to present None 2009 2014 434,405 100,586

NBC NBC Polls 2012 2015 125,382 140,384

PSRAI Pew Omnibus Biz Purge None 2010 2015 285,708 165,711

PewPew Internet & American Life

PollsBiz Purge None 2007 2015 369,301 185,385

RTI Survey of Consumer Attitudes None None 2010 2013 197,878 432,149

SRBI Confidential Biz PurgeInactive

2014+2007 2014 280,880 85,329

SSRS SSRS Omnibus MSG ID+ None 2009 2015 696,688 622,684


Initial ask was of ABC, CBS, Gallup, GfK, ICF, Ipsos, Nielsen, NORC, ORC, Pew, RAND, RTI, SRBI, TNS, Westat


50% 51% 51% 51% 52% 52%59%

54%

0%

20%

40%

60%

80%

2008 2009 2010 2011 2012 2013 2014 2015

Refusal Rate: Landlines4% Increase

48%52%

55% 53% 53%48% 46% 46%

0%

20%

40%

60%

80%

2008 2009 2010 2011 2012 2013 2014 2015

Refusal Rate: Cellphones2% Decrease

TRENDS IN DISPOSITIONSR E F U S A L S & C A L L B A C K S


CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONSN O A N S W E R / A N S W E R I N G M A C H I N E S


26%

32% 32% 31% 33% 34% 35% 36%

0%

20%

40%

60%

80%

2008 2009 2010 2011 2012 2013 2014 2015

NA/AM Rate: Landlines10% increase (4% since 2009)

21%

33% 31%34%

37%42%

45% 45%

0%

20%

40%

60%

80%

2008 2009 2010 2011 2012 2013 2014 2015

NA/AM Rate: Cellphones24% increase (14% since 2010)

CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONSN O N - W O R K I N G


28% 28%30%

33% 35%37%

41% 40%

0%

20%

40%

60%

2008 2009 2010 2011 2012 2013 2014 2015

NW Rate: Landlines12% increase

39%

34% 34%30% 29%

25% 24% 24%

0%

20%

40%

60%

2008 2009 2010 2011 2012 2013 2014 2015

NW Rate: Cellphones15% decrease (10% since 2009)

CLICK TO EDIT MASTER TITLE STYLETRENDS IN DISPOSITIONSY I E L D


.08

.07 .07 .07.06

.05

.03 .03

.00

.02

.04

.06

.08

.10

.12

2008 2009 2010 2011 2012 2013 2014 2015

Yield: Landlines

.06.05

.05 .05

.04 .04 .04.04

.00

.02

.04

.06

.08

.10

.12

2008 2009 2010 2011 2012 2013 2014 2015

Yield: Cellphones

Decreased yield by a factor of 2.4 (2.0 since 2009)

Have gone from 14 records per complete to 46

▲ Decreased yield by a factor of 1.3 (1.15 since 2009)

▲ Have gone from 19 records per complete to 25

CLICK TO EDIT MASTER TITLE STYLELANDLINE OWNERSHIP


0%

20%

40%

60%

80%

100%

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025

Percent of HH Cell Phone Only

CLICK TO EDIT MASTER TITLE STYLETELEPHONIC RESPONSE SINCE 2008

• Landline has no more than 10 years left in its lifetime, barring a change in trend

• Refusal rates have potentially “hit the ceiling”…

• …Possibly because people, more and more, just don’t pick up

• Non-working rates increasing precipitously on the landline, declining on cells

• Overall yield on landlines has increased dramatically, but only modestly for cell phones

• The good news, however: costs and response rates have flattened.



SO DOES THIS MEAN TELEPHONE SURVEY

QUALITY HAS SUBSTANTIALLY DECLINED?

CLICK TO EDIT MASTER TITLE STYLEDATA

Sample Name Mode Sample SizeResponse Rate

Formula

The BRFSS Telephone 6,118,156 CASRO/RR4

CBS Polls Telephone 168,826 RR1

ABC Polls Telephone 179,939 RR3

Pew Polls Telephone 213,191 RR3

The GSS In-Person 27,219 RR3

The NHIS In-Person 1,232,179 RR3


• All Pew, CBS, and ABC Polls (455 total polls), all BRFSS

• Studies span 1996 - 2015

CLICK TO EDIT MASTER TITLE STYLECOMPUTING THE PRIMARY METRICS

Consider the demographic cross tabulation of Race and Region producing a 4-by-4 table.Taking the absolute value of the difference between the row percentages and thecorresponding benchmarks from CPS produces a total of 16 absolute bias measures.(Distribution of Region within Race)


Race Midwest South West Northeast

White

Black

Other

Hispanic

4 absolute bias measures




Row Percentages

The average of these 16 bias

measures represents the

Mean Absolute Bias (MAB) of

Region within Race.

Repeating the calculations for each of the column percentages (Distribution of

Race within Region) yields the MAB for Race within Region.

CLICK TO EDIT MASTER TITLE STYLETRENDS IN UNWEIGHTED MEAN ABSOLUTE BIASES


0%

1%

2%

3%

4%

5%

6%

7%

8%

Unweighted Overall Mean Absolute Biases

ABC CBS Pew Brfss GSS NHIS

CLICK TO EDIT MASTER TITLE STYLETRENDS IN UNWEIGHTED MEAN ABSOLUTE BIASES


3.9%

3.8%

3.9%

3.8%

4.0%

4.0%

4.3%

4.4%

4.4% 5.0%

5.3%

5.8%

5.7%

5.8%

5.6%

5.5%

5.6%

5.3%

5.2%

4.8%

0%

1%

2%

3%

4%

5%

6%

7%

Unweighted Overall Mean Absolute Biases

CLICK TO EDIT MASTER TITLE STYLEWEIGHTED TRENDS IN MEAN ABSOLUTE BIAS


CLICK TO EDIT MASTER TITLE STYLEPOTENTIAL IMPACT OF CELL PHONES I N T H E R D D S A M P L E S


0%

10%

20%

30%

40%

50%

60%

70%

Percent of Interviews Attained by Cell Phone

ABC CBS Pew Brfss

4.0%

4.5%

5.0%

5.5%

6.0%

6.5%

0%

10%

20%

30%

40%

50%

60%

70%

Overlay of Cell Phone Share and MAB

Share of Cell Phones MAB


SO CELL PHONES ARE THE FUTURE…BUT

KIDS DON’T ANSWER THEM RIGHT?THEY CANNOT POSSIBLY BE REPRESENTATIVE….

CLICK TO EDIT MASTER TITLE STYLECHANGES IN TELEPHONE SURVEY DEMOGRAPHICS


34% 33%

65%

36% 32%

26%31% 34%

7%

Benchmark Cell Sample LandlineSample

Age Distribution By Sample

18-34

35-54

55+

Missing

8% 8% 6%

15% 12%8%

12% 12%

8%

66% 66%79%

Benchmark CellSample

LandlineSample

Race Distribution By Sample

White Non-Hisp

Black Non-Hisp

Hispanic

Other/MixedNon-Hisp

Missing

Kennedy (Pew Research Data), 2014


© SSRS | ALL RIGHTS RESERVED 22

MEAN ABSOLUTE BIASES FOR CELL PHONE OWNERS COMPARED TO U.S.

ADULT POPULATION ON 12 DEMOGRAPHIC CROSS-TABULATIONSM

ean

Abs

olut

e B

ias

(fro

m to

tal U

.S. A

dult

Pop

ulat

ion)


HOW DOES LOW RESPONSE RATE

TELEPHONE COMPARE TO

NONPROBABILITY RESEARCH IN TERMS

OF DATA QUALITY?

CLICK TO EDIT MASTER TITLE STYLEPROBABILITY VS. NONPROBABILITY

0% 2% 4% 6% 8% 10% 12% 14% 16% 18%

Web 1 Unweighted

Web 2 Unweighted

SSRS Omnibus Cell Phones Unweighted

SSRS Telephone Omnibus Unweighted

SSRS Tracker Telephone Unweighted

NHIS Unweighted

Web 1 Raked

Web 2 Raked

Web 2 Propensity Weighted

Web 2 Propensity Weighted and Raked

Web 2 Matched

Web 2 Matched and Raked

Web 1 Matched

Web 1 Matched and Raked

Cell Phones Raked

SSRS Omnibus Telephone Raked

SSRS Tracker Raked

NHIS Weighted

Mean Bias of Interactive Marginals by Sample/Weighting Type

Race within Education

Age within Education

Region within Education

Race within Age

Education within Age

Region within Age

Age within Race

Region within Race

Education within Race

Race within Region

Age within Region

Education within Region


CLICK TO EDIT MASTER TITLE STYLEOVERALL AVERAGE BIASES COMPARISON


CLICK TO EDIT MASTER TITLE STYLEOVERALL VARIANCE OF THE BIASES COMPARISON


CLICK TO EDIT MASTER TITLE STYLEPROBABILITY VS. NONPROBABILITY


Malhotra & Krosnick (2007), Chang & Krosnick (2009), Dutwin and Buskirk (2015), Tourangeau,

Conrad, and Couper (2013), Walker et al. (2009), Yeager et al. (2011)…


The Advertising Research Foundation

(ARF) set up the Online Research

Quality Council (ORQC) in August 2007.

17 US online panel providers (all using

nonprobability samples) a telephone

sample panel, and a mail sample panel.

Factual and behavioral questions were

asked with the same question wording

as the benchmarks they would be

compared against; data weighted.

Wide variation across panels in the

survey estimates.

Sample tenure effected estimates

derived from panels.

ARF STUDY


1819

2020

2123

242626

2727

2828

3030

3132

33

Benchmark

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

Comparison of Smoking Prevalence; Benchmark vs. 17 Opt-In Panels

Walker, R., Pettit, R., & Rubinson, J. (2009). A special report from the

Advertising Research Foundation: The foundations of quality

initiative: A five-part immersion into the quality of online research.

Journal of Advertising Research, 49, 464–485.


SEGUE…SO WHAT ABOUT THE

ELECTION ANYWAY?

CLICK TO EDIT MASTER TITLE STYLETHE STORY OF THE 2016 ELECTION AS WE KNOW IT

0%

1%

2%

3%

4%

5%

6%

7%

8%

2000 2002 2004 2006 2008 2010 2012

Accuracy in U.S. Polls, 2000 - 2012

Gubernatorial

Senatorial

House

Presidential


CLICK TO EDIT MASTER TITLE STYLETHE STORY OF THE 2016 ELECTION AS WE KNOW IT

The Macro Problem:

• Polls always challenging: a survey of a population that does not yet exist.

• Polls ask if election were held today.

• Polls try to predict the popular vote…so what is the problem here?

Still There Were Problems: State polls in aggregate, and “the aggregators,” can predict the

Electoral College and did not:

• State polls typically of lower cost: RBS sample; fewer call attempts; smaller sample size; less

sophisticated likely voter models employed

• State polls done less frequently

Possible Causes of Poor Election Results:

• Momentum effect

• People lying about intention

• Undecideds/3rd party intenders going Trump/Last minute decision-making

• Nonresponse error

• Likely voter model wrong

• Challenges to state polling


CLICK TO EDIT MASTER TITLE STYLETHE STORY SO FAR

The Past

• Survey response down…telephonic survey costs up…but

Mostly in landlines

Cellphone response and costs flat since 2014

Data bias up starting in ‘05 but on a downward since

‘09, little net effect

The Present

• The age of fragmentation and fit for purpose…telephone,

abs, nonprobability, probability panel, omnibus survey…is

already upon us.

• There remains a strong correlation between cost and

data quality…period.

What will the future hold?


CLICK TO EDIT MASTER TITLE STYLEFUTURE 1

This presentation has established:

• Landline future is limited

• Cellphone data has very low bias, only marginally worse than in-person

• Cellphone costs per interview have flattened, as have response rates

The next few years are critical, but the data suggest we all may be surprised at the future of

telephone research


CLICK TO EDIT MASTER TITLE STYLEFUTURE 2: BETTER OPT-IN DATA

Many people have been working hard to find a way to make opt-in data quality on its

own better. How:

• Webographics and other propensity adjustments.

• Sample matching: a potential way to stack the deck before data collection.

• Advanced calibration: in short, rake to everything.

• Balanced samples: pay the extra money to get samples pre-balanced on key

demographics.

…In three years of many people trying these out, so far the results have been inconsistent.


CLICK TO EDIT MASTER TITLE STYLEFUTURE 3: HYBRIDS

Fielding studies that are a blend of probability and non-probability samples.

• The goal: cost savings without significant risk of bias.

• Can use propensity or calibration techniques to combine.

• Two approaches: 1) true hybrid 2) limited questions in probability for calibration purposes

only.

• Must be mindful of mode effects.

…In three years of many people trying these out, so far the results have been somewhat

promising, but inconsistent.


CLICK TO EDIT MASTER TITLE STYLECASE STUDY: C A L I B R A T I O N T O P R O B A B I L I T Y E S T I M A T E S


Benchmark Unweighted Raked Propensity Calibration1 Calibration 2

Moved last 6 months 7% 14% 10% 9% 9% 9%

One person household 23% 18% 18% 18% 17% 19%

Do not own a computer 16% 6% 7% 7% 10% 16%

Do not own a smartphone 32% 28% 28% 28% 29% 31%

Do not own a tablet 62% 53% 42% 59% 58% 60%

Own home 67% 63% 57% 58% 67% 67%

Single 28% 30% 33% 32% 28% 28%

Not employed 42% 42% 46% 45% 45% 45%

Have children 37% 39% 35% 35% 36% 36%

38%

47%

27%

36%42%

27%27%31%

15%17%22%

15%7% 9% 11%

0%

10%

20%

30%

40%

50%

Overall Bias w/o OH & S w/o OH, S & C

Bias in 8 Estimates Unweighted Raked Propensity Calibration1 Calibration 2

CLICK TO EDIT MASTER TITLE STYLESOMETIMES HYBRIDS WORK…


Hybrid: anything derived from heterogeneous sources, or composed of elements of

different or incongruous kinds.

CLICK TO EDIT MASTER TITLE STYLE…AND SOMETIMES THEY DON’T


CLICK TO EDIT MASTER TITLE STYLEFUTURE 3: PROBABILITY PANELS?

Mimicking Opt-In Panels, but with a Probability Recruitment Method

• The goal: cost savings without significant risk of bias.

• About half the cost of custom telephone.

• However: typically small panel sizes (10k – 50k) limit consistent tracking or low incidence

studies.

• Suffer from the same non-response patterns of opt-in samples.


CLICK TO EDIT MASTER TITLE STYLEDATA QUALITY OF PROBABILITY PANELS


CLICK TO EDIT MASTER TITLE STYLEFUTURE 4: ADDRESS BASED DESIGNS?

High quality policy researchers do not have an “in-person” budget, but are searching

for telephone alternatives

• Tendency to systematic bias typical of telephones, if not worse.

• High response rates often require multiple modes

• Multiple languages more challenging

• Need field time to do well


CLICK TO EDIT MASTER TITLE STYLETHE FUTURE? ITS COMPLICATED

• One size does not fit all; we are truly in the “fit for purpose” age.

• The “space race” to “fix” nonprobability will continue. Every study will need its own model

• Telephone will continue to serve about as well as they have for the past 10 years, if, with a

higher level of distrust and elevated costs

• ABS will serve where applicable; in-person will continue to be the method of choice for

those with deep pockets

• Hybrid surveys will find a greater role in industries where future telephone costs are

unpalatable



D A VI D D U T WI N4 8 4 . 8 4 0 . 4 4 0 6

[email protected]

@ddutwin

@ssrs_solutions | 484.840.4300 | [email protected]

CONTACT MEW I T H A N Y Q U E S T I O N S

The Future of Survey Research March 2017

Data & Analytics

Transcript of The Future of Survey Research March 2017