The 41st Election

1

The 41st Election:

A Polling Retrospective

By Frank Graves

Ottawa, Ontario

September 22, 2011

Implications for Methodology and Democracy

2

Outline

1.0 Evaluating Our Polls and Industry Overall

2.0 Understanding the Gap between Final Polls and Re sults

2.1 First Hypothesis: Flawed Poll 2.2 Second Hypothesis: Late Shift2.3 Third Hypothesis: Turnout

3.0 Implications for Improving Polling Methodology a nd Reporting

4.0 Implications for Democracy, the Country, and Soc ietal Engagement

3

1.0 Evaluating Our Polls andOverall Industry

“…an unprecedented amount of polling (failed) to forecast accurately the key regional results that determined the shape ofthe House of Commons. For example, no company released results that were close to correct about Conservative support inOntario, which provides a third of the seats in the House.”

-Jennifer Espey

4

• Successes:

– Overall accurately / swiftly understands key themes of the

electorate

– First detected NDP surge / Bloc collapse

– Accurately predicted Liberal Party decline / Green Party collapse

– Correctly noted the new fault lines (generational, knowledge class)

• Failures:

– Final Conservative majority outcome was unexpected (but this

turns out to be a failure to forecast the voter turnout accurately)

Successes and Failures of EKOS Polling

5

39

35

36.4

34.5

0 10 20 30 40 50

Nanos (April 29th)

EKOS (April 29th)

Angus Reid (April 27th)

Environics (April 25th)

Summary of Pre-Election Polls – Friday, April 29 th

% Conservative

• Remarkable Consensus

– Late shift

– All biased?

– Turnout?

6

• EKOS’ federal vote intention polls are conducted using Interactive Voice Response (IVR) technology,

which allows respondents to enter their preferences by punching the keypad on their phone, rather than

telling them to an operator

– In an effort to reduce the coverage bias of landline only random digit dialing (RDD), we created a

dual landline/cell phone RDD sampling frame

• Advantages

– Perhaps closest to national population

– Minimises social desirability

– Cost-effective

– Higher reliability due to large sample sizes

• Disadvantages

– Higher non-response

– Survey must be shorter

– Some design limits

– Intrusiveness

– Reputation

IVR versus Other Polling Methodologies

7

4333

25 20

80

32 34 34

13

87

32 37

14

85

31 33 36

15

85

31

0

25

50

75

100

Actual Population IVR Landline CATI Land-line IVR Dual Frame Land-Mobile

12

36 35

186

2943

22

3

27

46

248

3140

21

0

25

50

75

100

Age

<25 25 to 44 65+45 to 64

High school College ImmigrantUniversity Born in Canada

Education Immigrant Status(%)

IVR versus CATI

BASE: IVR: Canadians; December 2010 (n=1,976) CATI: Canadians; January 2011 (n=3,009)

8

2.0 Understanding the Gap between Final Poll and Results

• Hypothesis & Test Results:

1) Flawed poll (bias) – measurement / sample

2) Late shift (“blue” Liberals in Ontario?)

3) “Get-out-the-vote” / Turnout

9

• Evidence shows polls were accurate in a number of key areas:

1. Polls correctly captured 2008 vote throughout campaign

2. Polls correctly captured 2011 vote post hoc (reproduced

election result)

3. Samples were close to census on all key parameters

• Question: Why would samples accurately give their 2008 and 2011

Conservative vote choices (behaviour) but erroneously give vote

intentions? (makes no sense)

– Conservative vote intention is lower in non-voters

• Prompting Green → mixed social desirability / turnout effect?

– Evidence shows lingering Green Party effect in both intentions

and behaviour

2.1 - First Hypothesis: Flawed Poll

10

18.2

26.3

6.8

10.0

1.2

15.1

27.1

7.49.7

37.7

2.8

37.9

0

10

20

30

40

CPC NDP LPC GP BQ Other

2008 Election Results EKOS Poll (April 29-May 1)

Other

BASE: Those who voted in 2008; April 29-May 1, 2011 (n=2,789)

Bias Test 1 – 2008 Vote Behaviour in Final Poll

Q. How did you vote in the federal election held in 2008?

Note: The data is based on those who say they voted in 2008. Our survey also finds that 11.7% of respondents did not

vote.

11

39.6

30.6

18.9

3.96.0

0.9

37.9

29.3

20.3

6.0 5.3

1.2

0

10

20

30

40


May 2nd Election Results EKOS Poll (May 20-28)

Other

BASE: Those who voted on May 2nd; May 20-28, 2011 (n=2,209)

Note: The data is based on those who say they voted on May 2nd. Our survey also finds that 20.9% of respondents either

did not respond or did not vote.

Q. How did you vote in the most recent federal election, held on May 2nd?

Bias Test 2 – 2011 Vote Using Same Method 3 Weeks La ter

12

• Did Liberal Party supporters (particularly in Ontario) move to the

Conservative Party in fear of an NDP-led coalition?

– For this to be the case, there had to be a feedback loop to the

polls

– For this to be the case, late shifters had to move overwhelmingly

to the Conservative Party (particularly in Ontario)

• Evidence / Questions:

1) Was your final choice your original choice?

2) When did you make your final decision?

3) What was the reason for this shift?

4) Did you follow the polls?

5) Did the polls influence your choice?

2.2 - Second Hypothesis: Late Shift

13

73

21

6

Yes No DK/NR

Q. Were you originally leaning towards voting for a different party at the beginning of the election campaign?

BASE: Voters; May 20-28, 2011 (n=2,203) / Those who changed their mind; May 20-28, 2011 (n=414)

3

21

37

22

20

0 10 20 30 40

Q. [IF YES] When did you make your final decision regarding how you were going to vote?

Before the election campaign

After the leader’s debate

In the last week of the election campaign

On the day of the election

DK/NR

Timing of Final Decision

Of the 21% who changed their minds, 21% did so on the day of the election. Therefore, just 4.2% (21% of 21%) made their decision on election day.

14

3

25

29

43

0 10 20 30 40 50

3

8

13

76

0 20 40 60 80

Not at all (1-2)

Somewhat closely (3)

Very closely (4-5)

DK/NR


Not at all (1-2)

To a moderateextent (3)

To a great extent (4-5)

DK/NR

Q. Throughout the election campaign, how closely did you follow public opinion polls?

Impact of Public Opinion Polls

Q. To what extent did public opinion polls affect your final decision on Election Day?

15

43

32

6

19

0

10

20

30

40

50

60

It was time for a

majority government

We need to stay on a

sound economic

trajectory

Concerned about an

NDP-led coalition

DK/NR

Biggest Factor in Switching (Conservative Shifters Only)

BASE: Those who switched to the Conservative Party during the campaign; May 20-28, 2011 (n=99)

Q. What was the biggest factor in your final decision?

16

• With the endemic bias and late shift hypotheses

failing tests, we turn to turnout / get-out-the-vote

• Evidence:

– Analysis of demographic patterns of non-voters

– Regression / Log-linear analysis of voter

preferences of these groups

– Analysis of the cell-only population

→ mirrors these overall patterns

2.3 - Third Hypothesis: Get-Out-the-Vote

17

• While both sample and measurement biases inflate voting

intention and reported behaviour, the relative patterns are

revealing

• Age is crucial with non-voters (those under 25 are six times

more likely to not vote than those over 65)

• Even Gen X voters (i.e., 25-44) were twice as likely not to vote

as boomers

• Virtually all seniors claim to vote and vast majority actually do

• Age interacts with gender – young women even less likely to

vote

Results / Patterns of Non-Voting

18

1.5

6.4

6.0

21.0

31.2

33.9

0 10 20 30 40

Other

BQ

GP

LPC

NDP

CPC

0.9

6.0

3.9

18.9

30.6

39.6

0 10 20 30 40

Other

BQ

GP

LPC

NDP

CPC

2.5

7.0

9.3

24.3

32.1

24.8

0 10 20 30 40

Other

BQ

GP

LPC

NDP

CPC

EKOS final pre-election poll (April 29-May 1)

Actual results Estimated non-voter support

Voters (61.4%)

Non-Voters (38.6%)

Estimated Non-Voter Support

19

4

7

15

24

11

0 5 10 15 20 25

10

11

16

11

0 5 10 15 20

Non-Voters by Key Demographics

Q. How did you vote in the most recent federal election, held on May 2nd?

Age Phone type

% who did not vote % who did not vote

Overall

<25

25-44

45-64

65+

Overall

Cell only

Both cell & landline

Landline only

BASE: Canadians; May 20-28, 2011 (n=2,724)

20

• Non-voting is strongly linked to age

• Support for Conservative Party is strongly linked to age

• Cell-only use is strongly linked to age and non-support for

Conservative Party

• Cell-only use mirrors broader pattern of linkage of non-voting

and non-Conservative support

• Therefore, including cell phone-only in the sample produced

the larger gap

• Indeed, with the cell-only users out, our results are within the

margin of error of the election results

Impact of Cell Phone-Only Respondents

21

37.7

29.2

22.2

5.4 4.6

0.9

24.1

33.1

21.1

7.6

11.2

3.0

0

10

20

30

40

CPC LPC NDP GP BQ Other

Both landline and cellphone Cell phone only

Other

Cell Phone Only versus Cell/Landline Households

BASE: Decided voters; April 29-May 1, 2011 (n=1,836)

Note: The data on federal vote intention are based on decided and leaning voters only.

Q. If a federal election were held tomorrow, which party would you vote for?

22

3.0 Implications for Improving

Polling Methodology and Reporting

23

• Is the link between election outcome and final polls a good yardstick of

polling quality?

– Increasingly, no. Why not?

→ growth of non-voters (now over 50% of those under 45)

→ strong correlation between non-voting and non-support of

incumbent

• How to “fix” the forecast:

– Commitment index

– Remove <45

– Remove new voters / Eliminate non-voters

What Does This Mean?

24

• Our commitment index is an aggregation of five factors:

– the ease with which a respondent revealed/declared their voting preference

– the self-rated level of enthusiasm with current voting choice

– the expressed intensity of attachment to current voting choice

– the self-expressed likelihood of actually voting on May 2nd

– the respondent’s willingness to consider other parties

Commitment Index

25

39.6

30.6

18.9

3.96.0

0.9

38.7

31.9

18.1

5.0 5.5

0.8

0

10

20

30

40


May 2nd Election Results EKOS Voter Commitment Index (April 29-May 1)

Other


Q. If a federal election were held tomorrow, which party would you vote for?

EKOS Commitment Index versus Actual Results

Note: The data on federal vote intention are based on decided and leaning voters only.

26

• We have no assurances these adjustments would work again

(e.g., they would have blown up in US 2008 election)

• Commitment / Enthusiasm may be crucial (taps the emotional

domain)

– In particular, older, conservative voters are more

enthusiastic / committed than older, non-conservative

voters

• Self-rated likelihood to vote means nothing

• No one would seriously argue that ignoring the younger half of

the population or new market entrants is a sound research

method for any other area of the private or public centre

What Does This Mean?

27

44

15

33

4 31

48

13

32

3 4

38

18

34

52 21

0

10

20

30

40

50


Overall Men Women

Other


Q. If a federal election were held tomorrow, which party would you vote for? (highly enthusiastic voters only)

Interaction of Enthusiasm, Gender, and Party Suppor t

Note: Results are delimited to those who say they are “highly” enthusiastic regarding their current choice (i.e., 6 or 7 on a 7-point scale)

28

• More circumspect final forecast

• Greater awareness that nearness of final poll to

electoral outcome is an increasingly flawed indicator

of polling quality (it was a good test in the past)

• Shift media focus away from vanity sweepstakes of

final prediction to deeper understanding of campaign

dynamics and implications for future

• Better understanding of turnout

– Can’t assume future will resemble the past

Recommendations

29

4.0 Implications for Democracy,

the Country, and SocietalEngagement

30

• The new generational chasm

• Most of those under 45 aren’t voting (more so than in the past)

• This may be in part a product of conscious political strategy (suppression)

• Profound questions about implications for a societal “succession” strategy

• Profound differences between older and younger Canada

– Composition• Diversity

• Education

• Digital world view/ horizontality

– Values and Interests (younger Canada tilt)• Knowledge and skills

• Climate and post Carbon economy

• Cosmopolitan

• Less security-focussed

Implications for Democracy and Societal Engagement (1/2)

31

• Unusual demographic skew today. Median age is 42, up from 26 at

our centennial.

– Gerontocracy? Political agenda reflecting imagined / exaggerated

fears of older Canada? Good politics but good policy?

– Healthy economies balance grace and enthusiasm; prudence and

exuberance

– Brewing generational storm?

– Lesson of election polling should not be to ignore those voices

which weren’t registered – if anything, this heightens need to

include “unheard” voices

– Shift from obsessive focus on non-existent horse race to informed,

reflective, and representative national conversations on key issues

Implications for Democracy and Societal Engagement (2/2)

32

43

17

39

1

58

15

25

2

0

10

20

30

40

50

60

Dissatisfied (1-2) Neither (3) Satisfied (4-5) DK/NR

Overall Youth (<25)

Satisfaction with Election Results


Q. Overall, how satisfied were you with the results to the last election?

33

• Frank Graves. Accurate Polling, Flawed Forecast: An Empirical Retrospective on Election

41 (June 2011)

http://www.ekospolitics.com/wp-content/uploads/accurate_polling_flawed_forecast.pdf

• Frank Graves. The Great Canadian Poll-Off (March 2011)

http://www.ekospolitics.com/wp-content/uploads/the_great_canadian_poll_off.pdf

• EKOS Research Associates. Interactive Voice Response: The Past, Present, and Into the

Future (January 2011)

http://www.ekos.com/admin/articles/2011-01-21-MRIA.pdf

• EKOS Research Associates. EKOS’ Observations on MRIA Study – Canadian Online

Panels: Similar or Different? (January 2010)

http://ekos.com/admin/articles/MRIA-Comparison-Panel-Study-2010-01-27.pdf

For Further Reading

34

FOR MORE INFORMATION:

Frank Graves, President

EKOS Research Associates

p. 613.235.7215

e. [email protected]

The 41st Election

Documents

Transcript of The 41st Election