© GfK 2012 | Title of presentation | DD. Month 2012 1 Mansour Fahimi, Frances Barlas, Randall...

© GfK 2012 | Title of presentation | DD. Month 2012 1

SCIENTIFIC SURVEYS BASED ONINCOMPLETE SAMPLING FRAMES AND HIGH RATES OF

NONRESPONSE

Mansour Fahimi, Frances Barlas, Randall Thomas, and Nicole Buttermore

DC-AAPOR – WSS SUMMER CONFERENCE PREVIEW/REVIEW 2015


Overview

Current statistical machinery for sample surveys requires: Probability-based samples Complete sampling frames High response rates

Ground realities: Many sampling frames are incomplete Most surveys suffer from high rates of nonresponse Often budget cannot afford fully probability-based samples

Where do we go from here? Develop a "new math" for survey research Consider more robust weighting/calibration alternatives

Interest/acceptance for non-probability samples is growing2


The Big Picture

3

Interpretation

Total Survey Error

Errors of

Non-observation

Errors of

Observations

Errors of

Processing

Errors of

Dissemination

Coverage

Response

Sampling

Instrument

Interviewer

Mode

Coding/Editing

Imputation

Weighting

Data Compilation

Analysis


The Magic of Survey Sampling (Theory)

4

Target Universe(N)

Sample(n)

Random

Selection

Weighting

n

N


The Magic of Survey Sampling (Practice)

5

Target Universe(N*)

Sample(n)

Quasi-Random

Selecti

on

Weig

hting

Inferred Universe

)ˆ(N

*

*

* N

N

n

N

n

nRespondents

(n*)

Qu

asi-Ran

dom

Non

respon

se


Sample Types

Probability-Based Samples:

All units have known and nonzero selection probabilities

The sampling frame contains all eligible units

Samples from KnowledgePanel (KP)

Non-probability Samples:

Units do not have known selection probabilities

The sampling frame is very incomplete

Samples from opt-in (OP) online panels

Many probability-based samples are not far from OP samples6


Demo-WeightedKP

24%

34%

36%

22%

45%

Demo-Weighted

OP1

44%

52%

55%

38%

60%

Demo-Weighted

OP2

41%

54%

59%

40%

62%

To what extent do you agree with the following statements?

How are Opt-in Samples Different?

EA1: I usually try new products before others do

EA2: I often try new brands because I like variety

EA3: When I shop I look for what is new

EA4: I like to be the first to try something new

EA5: I like to tell others about new things


Demographic Weighting is Not Enough

8

EA1 EA2 EA3 EA4 EA5 SV1 SV2 SV3 SV4 SV5 SV6 SV7 SV8 SV90%

30%

60% Weighted Estimates for 5 EA and 9 Outcome Variables

KP WeightedOP Weighted


Weighted Results With Calibration

9

EA1 EA2 EA3 EA4 EA5 SV1 SV2 SV3 SV4 SV5 SV6 SV7 SV8 SV90%

30%

60% Weighted Estimates for 5 EA and 9 Outcome Variables

KP Weighted

OP Weighted

KP+OP Calibrated


In What Other Ways are KP and OP Different?

10

Social Engagement: Take vacation with others

Exercise/play sports with others

Have a meal in someone’s home

Self-Importance: Importance of sharing opinions

My opinions are hard to change

Feel confident in social settings

Shopping Habits: Use of coupons when shopping

Enjoying shopping online

Importance of brand compared to price



11

Happiness and Security: Happiness with life

Feeling insecure and lonely

Concerned about others collecting information

Politics: Having influence on national politics

Views on government effectiveness

Closely following the news

Community: Feeling part of a larger community

Number of moves in past 5 years

Religiosity



12

Altruism: Donating blood

Donating items

Volunteering without pay

Survey Participations: Experience with online surveys

Important of taking opinion surveys

Frequency of online surveys in a month

Internet and Social Media Usage: Frequency of accessing personal email

Frequency of accessing Internet

Time spent watching TV per day


Searching for a Parsimonious Subset of Differentiators?

13

Number of online surveys taken in a month

Hours spent on the internet in a week for personal needs

Trying new products before other people do

Time spent watching television in a day

Using coupons when shopping

Number of relocations in the past 5 years


Measuring the Efficacy of the New Calibration(Against Government Statistics)

14

Behavioral Risk Factor Surveillance System (BRFSS): Smoked 100 cigarettes in lifetime Physical check-up past year History of depressive disorder One or more days per month physical health not good Seven or more hours sleep per night

National Survey on Drug Use and Health (NSDUH): Always wear seatbelt as front passenger Great risk when smoking one or more packs a day Great risk when trying heroin once or twice

Current Population Survey (CPS): Householders receiving social security Marital status Homeownership status Household income

American Community Survey (ACS): Number of bedrooms in house Number of automobiles


Results of Evaluation(Against Government Statistics)

15

OPCalibrated (1.0)

OPDemo Weighted

OPCalibrated (2.0)

KP+OPCalibrated (1.0)


KPDemo Weighted

7370

54

62

48

53

69 68

62

55

43

38

Average Mean Square Error Estimating BRFSS, NSDUH, CPS, and ACS Statistics

Georgia Illinois


Measuring the Efficacy of the New Calibration(Against Election Statistics)

Georgia: Percent Registered Voters Percent Republican Percent Conservative Senate Race Governor’s Race

Illinois: Percent Registered Voters Percent Republican Percent Conservative Senate Race Governor’s Race Should health insurance cover birth control


Results of Evaluation(Against Election Statistics)

17

OPDemo Weighted

OPCalibrated (1.0)

OPCalibrated (2.0)


KPDemo Weighted


2625

20 20 20

16

32

29

26

24

2119

Average Mean Square Error Estimating Election Statistics

Georgia Illinois


Concluding Remarks

Traditional methods of sampling have issues: Coverage Response Rates Cost/time

Innovative methods of sampling are inevitable: To address the above But need to maintain statistical integrity

Non-probability samples can provide pragmatic alternatives: Geodemographic weighting does not wash out all biases Calibration can reduce bias beyond geodemographic weighting

Calibration can also help probability-based samples with: Incomplete sampling frames High nonresponse rates

18


Thank You

19

Mansour Fahimi, Ph.D.Senior Vice President, Chief StatisticianMarketing & Data SciencesGfK Custom Research, LLC

T +1 240-477-4570C +1 240-565-8711

[email protected]

© GfK 2012 | Title of presentation | DD. Month 2012 1 Mansour Fahimi, Frances Barlas, Randall...

Documents

Transcript of © GfK 2012 | Title of presentation | DD. Month 2012 1 Mansour Fahimi, Frances Barlas, Randall...