Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr....

Examining Best Practices for Sampling

and WeightingDual Frame Surveys

Liz KantorAdvised by Dr. David Redlawsk

Rutgers, The State University of New Jersey

PANJAAPOR 2015 Spring Event


We know we need to include cell phones in telephone RDD samples and weight accordingly…

…but we don’t agree on how


Considerations

• Sampling What % of the sample

should be cell phones? Landlines?

• Weighting Base weight? Frame overlap adjustment?

Calibration adjustments? Trimming? Low variance!


Data and Methods• Data

Rutgers-Eagleton Poll

• Methods Demographics: Chi-square Sampling analysis: weight C & LL samples

to make up varying proportions of overall sample

Weighting: 17 weighting & trimming combinations


Weight Calculations• Base weight: sample selection probability

multiplied by “household” selection probability

• Frame adjustment: ½ Compositing, Effective Sample Size

From Kennedy (2012)


Weight Calculations• Calibration

“Basic Sample Balancing” – one manual iteration of each demographic variable

Raking – multiple iterations of each demographic variable; SPSSINC RAKE Extension

• Trimming Hard cap: .2 and 5 Percentile cap: 2nd and 98th percentiles


Weighting CombinationsTaking a cue from Kennedy (2012):

COMBINATION BASE WEIGHT? FRAME OVERLAP CALIBRATION TRIMMING METHOD0 No N/A Basic Sample Balancing Hard Cap, .3 & 3A Yes 1/2 Compositing Basic Sample Balancing Hard CapB Yes 1/2 Compositing Basic Sample Balancing Percentile CapC Yes 1/2 Compositing Raking Hard CapD Yes 1/2 Compositing Raking Percentile CapE Yes Effective Sample Size Basic Sample Balancing Hard CapF Yes Effective Sample Size Basic Sample Balancing Percentile CapG Yes Effective Sample Size Raking Hard CapH Yes Effective Sample Size Raking Percentile CapI No 1/2 Compositing Basic Sample Balancing Hard CapJ No 1/2 Compositing Basic Sample Balancing Percentile CapK No 1/2 Compositing Raking Hard CapL No 1/2 Compositing Raking Percentile CapM No Effective Sample Size Basic Sample Balancing Hard CapN No Effective Sample Size Basic Sample Balancing Percentile CapO No Effective Sample Size Raking Hard CapP No Effective Sample Size Raking Percentile Cap

*0 = Rutgers-Eagleton Poll practices at the time of research


HypothesesI. Cell-only respondents will follow

patterns in the literature of demographic distinctiveness and geographic mobility

II. Dual users from the cell frame will differ demographically from dual users from the landline frame

III. A higher % of cell phones than used in the Rutgers-Eagleton Poll will be required in the sample to match NHIS estimates


HypothesesIV. Half compositing will be less efficient

(higher DEFF) than effective sample size

V. No difference in efficiency between hard cap and percentile cap; can’t know a priori which will increase variance more


Demographics Statistically significant differences at α = .001

* α = .01

C Only vs. Dual vs. LL only

Dual from cellvs. Dual from

LL

Gender ✔ ✔

Age ✔ ✔

Race ✔ ✔

Party ID ✔

Education

✔ ✔*

Income ✔


DemographicsDifferences in gender composition…



Geographic MobilityMatch between FIPS code (county) from sample provider and from respondent



Simulated Sample Composition

CELL-ONLY

CELL-MOSTL

YDUAL USE

LANDLINE-

MOSTLYLANDLINE-ONLY

NHIS (NJ) 19.4% 25.7% 31.1% 15.2% 6.9%30% C 70% LL 10.4% 33.7% 26.6% 23.3% 6.0%40% C 60% LL 13.9% 34.1% 25.8% 21.1% 5.1%50% C 50% LL 17.4% 34.4% 24.9% 19.0% 4.3%60% C 40% LL 20.8% 34.8% 24.1% 16.9% 3.4%70% C 30% LL 24.3% 35.2% 23.2% 14.7% 2.6%



Design Effects



Margins of Error

*Calculated using Langer Research Associate’s MoE Machine



Average deviation from NHIS estimates

CELL-ONLY

CELL-MOSTLY

DUAL USE

LANDLINE-MOSTLY

LANDLINE-ONLY

NHIS (NJ) 19.4% 25.7% 31.1% 15.2% 6.9%

* indicates that the weighting combination came within 1% of the cell-only estimate



Discussion• Support for Hypotheses I, II, and III:

Significant demographic differences among cell only, dual users, and landline only• Estimated geographic mobility much

greater for Rs from cell sample vs. Rs from landline sample

Significant demographic differences between dual users from cell and landline samples

More cell phone interviews (~50%) are needed to match NHIS estimates of cell only


Discussion• No support for Hypothesis IV

½ Compositing was not less efficient than Effective Sample Size

• Little support for Hypothesis V Percentage cap was more efficient than

the hard cap


Weighting Recommendations• Ultimately, no obvious best practices• Criteria for choosing weighting and

trimming combinations: Includes a frame adjustment (avoid

biased estimators) Inclusion of base weight

• Brings percentage of cell-only respondents within 1% of NHIS estimates

Percentile cap – more efficient


Weighting Recommendations• Under these criteria, good weighting

and trimming combinations are: B: Base Weight, ½ Compositing, Basic

Sample Balancing, Percentile Cap D: Base Weight, ½ Compositing, Raking,

Percentile Cap F: Base Weight, Effective Sample Size,

Basic Sample Balancing, Percentile Cap


Future Research• Replicate using other data• Strengthen the theoretical basis of

the criteria used to recommend weighting combinations

• Experiment with more aggressive percentile caps (e.g. 5% and 95%)


Thank [email protected]

Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr....

Documents

Transcript of Examining Best Practices for Sampling and Weighting Dual Frame Surveys Liz Kantor Advised by Dr....