CELLPHONE SAMPLING AT THE STATE LEVEL: GEOGRAPHIC …
24
Stephanie Marken Kafka—Methodologist, The Gallup Organization Manas Chattopadhyay—Senior Statistician, The Gallup Organization Anna Chan—Senior Statistician, The Gallup Organization CELLPHONE SAMPLING AT THE STATE LEVEL: GEOGRAPHIC ACCURACY AND COVERAGE CONCERNS
Transcript of CELLPHONE SAMPLING AT THE STATE LEVEL: GEOGRAPHIC …
Stephanie Marken Kafka—Methodologist, The Gallup OrganizationManas Chattopadhyay—Senior Statistician, The Gallup OrganizationAnna Chan—Senior Statistician, The Gallup Organization
CELLPHONE SAMPLING AT THE STATE LEVEL:GEOGRAPHIC ACCURACY AND COVERAGECONCERNS
This document contains proprietary research, copyrighted materials, and literary property ofGallup, Inc. It is for the guidance of your company only and is not to be copied, quoted,published, or divulged to others outside of your organization. Gallup® and GallupConsulting® are trademarks of Gallup, Inc. All other trademarks are the property of theirrespective owners. No changes may be made to this document without the express writtenpermission of Gallup, Inc.
COPYRIGHT STANDARDS
The percentage of cellphone only households has increased steadilysince 2005
3
CELLPHONE PENETRATION CONTINUES TOINCREASE NATIONALLY
The Percentage of Cellphone Only Households in the U.S.
4
WIRELESS ONLY HOUSEHOLD PERCENTAGESVARY GREATLY BY STATE
States with Highest % Wireless Only
State % Wireless OnlyHouseholds
Wyoming 54.6
Hawaii 51.3
Idaho 50.3
D.C. 49.8
New Mexico 49.4
Alaska 48.1
Arizona 47.5
Texas 46.4
Utah 46.2
Mississippi 45.7
States with Lowest % Wireless Only
State % Wireless OnlyHouseholds
Vermont 21.4
Connecticut 23.6
Delaware 24.3
Rhode Island 24.4
New Hampshire 25.5
Massachusetts 26.1
Pennsylvania 26.6
Maryland 29.8
Alabama 30.2
Iowa 31.1
Source: Marketing Systems Group (MSG)
The growing percentage of cellphone users poses a significant samplingchallenge The state associated with a cellphone record on the cellphone sample file is
often different from the state in which the respondent actually lives This introduces both cost and coverage concerns when conducting studies at
the local level (state, MSA, city or town)– Cost
At the introduction to the interview, respondents that do not reside inthe targeted state are deemed ineligible and an otherwise eligiblerespondent is terminated
Study of Virginia residents– Sample designation: Virginia– True state of residence: Maryland
5
SAMPLING DIFFICULTIES
Coverage– Residents that would be eligible to participate in the study are
excluded because their designation on the sampling frame is a stateother than their current state of residence Study of Virginia residents
– Sample designation: Maryland– True state of residence: Virginia
– If excluded residents differ systematicallycoverage concerns increase
6
SAMPLING DIFFICULTIES
In their JSSM paper, Skalland et. al. defined this type of movement inand outside of a state as “overcoverage” and “undercoverage”¹
Overcoverage– The percentage of cell-phone only adults sampled from that state’s
cellphone sampling frame that currently reside in a different state Example: Percentage of cell-phone only adults sampled using
Virginia’s cellphone sample frame that live elsewhere i.e., Maryland
Undercoverage– The percentage of cellphone-only adults that reside in that state
although they appear on a different state’s cellphone sample frame Example: Percentage of cell-phone only adults that reside in Virginia
although they appear on other state’s cellphone frames such asMaryland
7
DEFINING MOVEMENT BETWEEN STATES
Skalland and Khare, 2013. Geographic Inaccuracy of Cell Phone Samples and the Effect on TelephoneSurvey Bias, Variance, and Cost. Journal of Survey Statistics and Methodology, pp. 45-65.
8
DIFFERENCES IN RESIDENCY AND SAMPLEDESIGNATION, BY STATE
Purpose of discussion today we will focus on all cellphonehouseholds—including those that completed an interview viatheir cellphone device and are cellphone only, cellphonemostly and dual users
Overcoverage– The percentage of cellphone households sampled from that
state’s cellphone sampling frame that currently reside in adifferent state
Undercoverage– The percentage of cellphone households that reside in that
state although they appear on a different state’s cellphonesample frame
9
DEFINING MOVEMENT BETWEEN STATES INGALLUP’S STATE OF THE STATES POLL
About the study– June-December of 2013– Telephone study conducted with adults ages 18+ in all 50 states
(excluded D.C.)– Phone numbers selected using random-digit-dial (RDD)– Approximately 600 completed interviews per state– Dual-frame design with cellphone/landline distributions tailored
on a state-by-state basis but generally approximating 50/50– English/Spanish– Weighted to national targets on age, race/ethnicity, gender,
education and phone status and normalized those weights to statetargets
10
GALLUP 2013 STATE OF THE STATESMETHODOLOGY
The survey covered the following topics:– Business and Economics– Citizen Engagement– Education– Government and Politics– Food and Shelter– Health and Well-Being– Religion– Social Issues– Work– Law and Order
11
STATE OF THE STATES SURVEY DESIGN
12
OVERCOVERAGE PERCENTAGES BY STATE,RHODE ISLAND LEADS AND TEXAS FOLLOWS
States with Highest Overcoverage %
State Overcoverage %
Rhode Island 22.0
Delaware 21.6
North Dakota 21.5
Hawaii 20.4
Connecticut 19.2
Massachusetts 17.2
Nevada 17.1
New Hampshire 16.1
Virginia 14.7
Maryland 14.7
States with Lowest Overcoverage %
State Overcoverage %
Texas 5.6
Oklahoma 5.8
California 6.6
Alabama 7.8
Wisconsin 8.6
Minnesota 8.7
Iowa 8.8
Pennsylvania 9.0
Colorado 9.2
Alaska 10.1
Estimates are not weighted
13
UNDERCOVERAGE PERCENTAGE BY STATE,VERMONT LEADS AND MICHIGAN FOLLOWS
States with Highest Undercoverage %
State Undercoverage %
Vermont 29.8
North Dakota 28.9
South Dakota 28.1
Wyoming 22.8
Wisconsin 20.3
Kentucky 20.1
Kansas 19.0
Massachusetts 18.8
Hawaii 18.4
West Virginia 18.2
States with Lowest Undercoverage %
State Undercoverage %
Michigan 2.9
Ohio 4.7
New Hampshire 4.9
California 5.1
Arkansas 6.1
New Jersey 6.3
Oregon 6.8
Alabama 6.8
Louisiana 7.3
Illinois 7.5
Estimates are weighted to national demographics
14
DEMOGRAPHIC DISTRIBUTION BY MATCHSTATUS
Demographic Sample State =Resident State
Sample State ≠Resident State
Gender Male 53.43 49.28
Female 46.57 50.72
Age* 18-34 49.49 59.91
35-44 18.70 19.66
45-54 16.06 9.53
55-64 8.95 5.76
65+ 6.80 5.14
Race/Ethnicity* White 60.29 63.12
Black 13.30 10.98
Asian 2.69 6.11
Hispanic 21.68 17.66
Other 2.04 2.12
*Differences between the point estimate for a match and non-match are statistically significant
15
DEMOGRAPHIC DISTRIBUTION BY MATCHSTATUS
Demographic Sample State =Resident State
Sample State ≠Resident State
Education* High school degree or less 45.74 32.89
Training/some college, nodegree 29.99 27.97
College degree and higher 24.28 39.14
Census Region Northeast 13.91 13.91
Midwest 21.58 20.07
South 39.93 45.18
West 24.57 20.84
*Differences between the point estimate for a match and non-match are statistically significant
16
RESPONSE DISTRIBUTION ON KEY SURVEYITEMS BY MATCH STATUS
Survey Question Sample State =Resident State
Sample State ≠Resident State
Are you satisfied with the city or area where you live?
Satisfied 81.36 83.62
Dissatisfied 18.64 16.38
Would you recommend the city or area where you live to afriend or associate as a place to live, or not?
Recommend 77.54 78.90
Not recommend 22.46 21.10
Regardless of whether you will move, if you had theopportunity to move, would you like to move to anotherstate or would you rather remain in your current state?
Yes, would like to move* 39.78 53.27
No, would rather remain* 60.22 46.73
*Differences between the point estimate for a match and non-match are statistically significant
17
RESPONSE DISTRIBUTION ON KEY SURVEYITEMS BY MATCH STATUS
Survey Question Sample State =Resident State
Sample State ≠Resident State
Thinking about the job situation in the city or area whereyou live today, would you say that it is now a good timeor a bad time to find a job?
Good time* 45.07 53.63
Bad time* 54.93 46.37
In the city or area where you live, are you satisfied ordissatisfied with the availability of quality health care?
Satisfied* 73.54 76.94
Dissatisfied* 26.46 23.06
Right now do you think the economic conditions in yourstate as a whole, are getting better or getting worse?
Getting better 93.64 94.19
Getting worse 6.36 5.81
*Differences between the point estimate for a match and non-match are statistically significant
Are you satisfied or dissatisfied with the city or area where youlive?– Differences in % satisfied
South Dakota -3.66%
Would you recommend the city or area where you live to a friendor associate as a place to live, or not?– Differences in % recommend
South Dakota -3.49 Wyoming 1.92
18
DIFFERENCES IN DUAL-FRAME ESTIMATES BETWEENMATCHED INTERVIEWS AND ALL INTERVIEWS
Regardless of whether you will move, if you had the opportunityto move, would you like to move to another state or would yourather remain in your current state?– Differences in % will move
Kentucky 2.54% Hawaii 2.34% Nevada 2.04% Wyoming 1.83%
19
DIFFERENCES IN DUAL-FRAME ESTIMATES BETWEENMATCHED INTERVIEWS AND ALL INTERVIEWS
Thinking about the job situation in the city or area where youlive today, would you say that it is now a good time or a bad timeto find a job?– Differences in % good time
South Dakota -3.71 Kansas 3.04 Kentucky 2.61 Alaska -2.31
20
DIFFERENCES IN DUAL-FRAME ESTIMATES BETWEENMATCHED INTERVIEWS AND ALL INTERVIEWS
In the city or area where you live, are you satisfied or dissatisfiedwith the availability of quality health care?– Difference in % satisfied
North Dakota -3.36 Hawaii 1.80% West Virginia -1.66%
21
DIFFERENCES IN DUAL-FRAME ESTIMATES BETWEENMATCHED INTERVIEWS AND ALL INTERVIEWS
Percentage of over and under coverage varies by state Match statuses differ for subgroups of the population Overcoverage will probably continue to be more problematic as
respondents become more reliant on cellular devices Should we be worried?
– The coverage bias associated with this type of movement may ormay not affect the overall precision of your estimates
– Important to consider your research questions and tolerance forerror
– As the percentage of cellphone only, mostly and dual usersincrease, overcoverage and undercoverage percentages willcontinue to challenge researchers
22
KEY FINDINGS
Future research should focus upon:– Identifying the optimal sample design that samples from rate
centers in adjoining states for states in which undercoverage isproblematic
– Identifying topics and items that are most sensitive to biasassociated with undercoverage
– Leveraging the advantages of other methodologies that reduceresearchers reliance upon dual-frame samples when sampling atthe local level
Gallup is currently conducting the 2015 State of the States Study– Data will be available in November of 2015– Will repeat this analysis to identify potential changes– Will be including DC in the sampling process
23
FUTURE RESEARCH
