Finding the Perfect Match Minimizing Bias & Increasing Representativness Through Sample Matching...

Finding the Perfect Match

Minimizing Bias&

Increasing RepresentativnessThrough Sample Matching

Lynn Vavreck UCLA Political Science & Polimetrix

Lynn Vavreck UCLA Political Science

The Representative Sample

1. How to Judge Representativeness

2. Ways that Surveys Miss the Target

2. Methods that Generate Representativeness in Theory

3. How to get Representative Samples in Practice

4. Does it “Work”?

Representativeness

Practice

Theory


How to Judge Representativeness

1 Possibility: Know population characteristics, compare sample to population

2 Possibility: Identify target that represents population, compare sample to target

• General Population – Census, US CPS• Registered Voters – Exit Polls, Registered Voter Files

What we often do: Comparing Internet to Phone samples -- comparing 1 sampling mode/design to another instead of a sampling mode/design to the target.


Ways we can Miss the Target

Sampling ErrorThe standard deviation of estimates obtained through repeated sampling following the same procedure.

Error due to chance.

BiasError as a result of gaps in the data are not benign but are systematically related to the phenomenon being modeled.

Violation of ignorability.


Ways to Get Close to Target

Theory • Simple Randomness – Each observation has an

equal chance of being selected into the sample

Practice• Various sampling techniques (cluster, stratify,

choose blocks, households, and then people, RDD, List-based methods … )

• Weights to compensate for non-response (different techniques, different constructions)

• As usable sample data deviates farther from targets, weights get bigger


Sample Matching

As a method of reducing sample bias

Can be used with various sampling methods

Involves generating matches among people on a given set of characteristics that are known before the survey is administered


How it Works Enumerate target population at individual

level (with data on characteristics) if possible

Draw a random sample, T,of size N from the target population.

For every i in T, find the closest match in an available pool (panel) by minimizing a distance function, d(x,y).

More than one match per i can be identified if desired (this allows for replacement)

Available Respondent

sPopulation

Target Sample Matche

d Sample


What is Needed to Do this? A large number of people from whom to

select matches

For Political Scientists, coverage of specific types of people

• Low interest/political sophistication/knowledge• Independents/Moderates• Low propensity voters

How do you ensure coverage of these parts of the population?


Everyone Cares about Something

People like to give their opinions … about something!

Find things that people identify with and ask them about those things

• Sports• Hobbies• Entertainment• Celebrity Gossip• True Crime


Other Methods

Many other methods for finding hard to reach populations

Panel Managements is critical• Incentives• Frequency of Surveys• Signals that Someone is listening• Interesting tools/widgets


Even if You Get Them …

Are low propensity voters who are willing to take surveys on the Internet “like” low propensity voters in general?

You can get the marginals right, but do the mechanisms still work?


Party ID and Ideology

Using NES 04, Annenberg 04, and CCES 06 we looked for the relationship between party identification and ideology for different levels of political knowledge

• Looking for degree of “constraint” among these two things

• A relationship/mechanism we know exists in regularity (different modes, samples, years, question wordings)


Hill, Lo, Vavreck, & Zaller


2004 Bush Vote Share by State

MS

NH

30%

40%

50%

60%

70%

80%

30% 40% 50% 60% 70% 80%

Bush Percentage in Preelection Survey

Act

ual

Bu

sh V

ote

Per

cen

tag

e

Finding the Perfect Match Minimizing Bias & Increasing Representativness Through Sample Matching...

Documents

Transcript of Finding the Perfect Match Minimizing Bias & Increasing Representativness Through Sample Matching...