Reducing Bias in Public Opinion Polls

Autumn CarterProfessor Jon Krosnick

Summer Research College 2009

Satisfy Your Constituents...

Or Suffer the Consequences.

So, how do we effectively determine what Americans

are feeling?

Public Opinion Polling

But Are the Polls Accurate?

?=

Critics Don't Think So...

“The dirty little secret of the polling industry is that, all too often, its findings are based on flawed methodology and dubious assumptions.”

-Arianna Huffington, Huffington Post Founder

Sample Public Opinion Question

Form 1

Individuals are more to blame than social conditions for crime and lawlessness in this country.

Agree

Disagree

Sample Public Opinion Question

Form 1

Individuals are more to blame than social conditions for crime and lawlessness in this country.

Agree: 59.6%

Disagree: 40.4%

Sample Public Opinion Question

Form 1

Individuals are more to blame than social conditions for crime and lawlessness in this country.

Agree: 59.6%

Disagree: 40.4%

Form 2

Social conditions are more to blame than individuals for crime and lawlessness in this country.

Agree

Disagree

Sample Public Opinion Question

Form 1

Individuals are more to blame than social conditions for crime and lawlessness in this country.

Agree: 59.6%

Disagree: 40.4%

Form 2

Social conditions are more to blame than individuals for crime and lawlessness in this country.

Agree: 56.8%

Disagree: 43.2%Acquiescence Effect: 16.4%

What is Acquiescence?

“...a presumed tendency for respondents to agree with attitude statements presented to them.”

-Schuman and Presser (1981)

Why do Respondents Acquiesce?

Norms of Conduct: Be polite and agreeable

Status Differential: Defer to higher status

Satisficing: Choosing the easiest response because it requires less thinking

Can we reduce acquiescence by

administering the survey via the Internet instead of

by telephone?

Would Internet Administration Reduce Acquiescence?

Norms of Conduct: Be polite and agreeable Do you have to be polite to a computer?

Status Differential: Defer to higher statusIs there a status difference without a human interviewer?

Satisficing: Choosing the easiest response because it requires less thinking

Are people less likely to satisfice on Internet surveys? (Chang & Krosnick, in press)

The Present Research

Replicate Schuman & Presser (1981)

Mode effects: Telephone vs. the Internet

Sample effects: Representative vs. Non-Representative

How Do You Survey a Representative Sample Via the Internet?

Knowledge Networks randomly dialed phone numbers and invited people to join a panel (of about 50,000 adults).

Provided Internet access to homes without it.

Periodically emails about 1,000 panelists to complete surveys and receive incentives.

A Classic Acquiescence Experiment

Form 1

Individuals are more to blame than social conditions for crime and lawlessness in this country.

Agree

Disagree

Form 2

Social conditions are more to blame than individuals for crime and lawlessness in this country.

Agree

Disagree

Replication in a National Survey

Telephone survey of a representative sample:

Internet survey of a representative sample:

* p < 0.05+ p <0.10

11.5%*

7.1%+

TelephoneXInternet Interaction: n.s.

What about Internet surveys of people who are not representative of the

national population?

Non-Probability Sample Internet Surveys

Volunteers click on banner ads and “opt-in” into the panel.

You've all seen these...

Non-Probability Sample Internet Surveys

Volunteers click on banner ads and “opt-in” to the panel.

Participants receive money or prize.

Perhaps they acquiescence less because they like taking surveys and have more practice.

In fact, previous research has found less acquiescence vs. a telephone survey (Chang and Krosnick, in

press)

Acquiescence BiasT

elep

hone

Pro

babi

lity

Web

Non

-Pro

b W

eb 1

Non

-Pro

b W

eb 2

Non

-Pro

b W

eb 3

Non

-Pro

b W

eb 4

Non

-Pro

b W

eb 5

Non

-Pro

b W

eb 6

Non

-Pro

b W

eb 7

0.0

5.0

10.0

15.0

20.0

25.0

Acquiescence Bias

Survey

Acq

uies

cenc

e B

ias

(in p

erce

nt)

*

*

**

*

†*

† †

*

If anything, non-probability samples increase acquiescence.

So we should be cautious when interpreting these

types of surveys.

What if we just didn't ask questions that ask

respondents to agree or disagree with a

statement?

Balanced Forced-Choice Questions

Form 1

Which do you think is more responsible for crime and lawlessness in this country: individuals or social conditions?

Individuals

Social Conditions

Form 2

Which do you think is more responsible for crime and lawlessness in this country: social conditions or individuals?

Social Conditions

Individuals

Response Order EffectsT

elep

hone

Pro

babi

lity

Web

Non

-Pro

b W

eb 1

Non

-Pro

b W

eb 2

Non

-Pro

b W

eb 3

Non

-Pro

b W

eb 4

Non

-Pro

b W

eb 5

Non

-Pro

b W

eb 6

Non

-Pro

b W

eb 7

-6.0

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

Survey

Diff

eren

ce b

etw

een

For

ms

(in p

erce

nt)

ns nsnsnsnsnsnsnsns

Conclusions

Acquiescence bias is real and can undermine public opinion polls and possibly the democratic process.

Internet administration did not significantly reduce acquiescence.

Non-probability sample surveys showed, if anything, more acquiescence.

BUT WAIT! You can simply ask the questions using the balanced forced choice format, eliminating order effects.