Panel Response Rates

By

Ted Vonk - Onderzoekpaleis

LONDON 3 October 2006

Advances and opportunities in market research online

Dutch Online Panel Study 2006

COPYRIGHT

This PowerPoint presentation contains the outcomes of the Dutch Online Panel Comparison study 2006 (NOPVO 2006).

It is allowed to publish the outcomes of this study if you refer to:

• The name of the study (NOPVO 2006)

• The authors: Ted Vonk (Onderzoekpaleis), Robert van Ossenbruggen (ProCression) and Pieter Willems (Millward Brown)

Contact: info@onderzoekpaleis.nl

Agenda

• An industry survey: current state of online panel research

• Response in NOPVO

• How important is response?

• Conclusions and recommendations

Gastspreker

Els Molenaar

“Als je wint heb je vrienden”

An industry survey: current state of online panel

research

Some Facts and figures about the Netherlands

Demographics

• Population: 16,3 million

• 18-65 years: 10,5 million

• Internet penetration: 82%

• Broadband penetration: 63% (of all housholds, not internet users)

Market Research

• Total market in 2005: € 278 million (UK: € 1.940 million)

• Spending per capita: € 17 (UK € 32)

• Online data collection: 25% of revenu quantitative (UK 9%)

• Over 25 online panels

Reasons for strong and fast growth of online research

• Growing costs of traditional research because of changing laws (social security fees)

• Decreasing response rates within traditional research

• Strong growth of internet penetration as a result of an active policy of Dutch government

Opinion and market data: over 50% is collected online

• Today, over 50% of research data is collected by online panels

• More than 25 online panels are operational in the Netherlands

• Clients have reacted differently over time

1998 – 2001: SceptisismIs it respresentative? what are the effects of mode?

2002 – 2004: Fully accepted – hardly any quality questionsSpeed and costs seem to have convinced all clients

2005 – now: Again increased quality concernlarge offering, low prices, lack of standards

The Dutch Online Panel Study (NOPVO)

Why?

1. Provide answer to quality concern

2. Provide guidelines for online panel innovation

3. Unique situation: large number of panels available

Research questions NOPVO

1. How do panels work: panel overview

2. Who is the online panel member

3. How do panellists respond?

4. To what extent is opinion and marketing data influenced by panel choice

NOPVO method

1. Make inventory of all online panels in the Netherlands

2. Empirical study

• One survey

• Every panels draws sample of 1000 respondents

• Same fieldwork period of 7 day (no reminders)

19 panels participated:

Representing in totalover 90% of panellists available in NL

NOPVO method

Extensive data available:

Per panel:-Type of incentive-Age of Panel-Invitation policy -etc. Per panel member:

-Panel history-Way of recruitment-Number of invitations-Individual response rate-socio demo, etc.

Per respondent:-Response (time)-Membership of other panels-data of the questionnaire

NOPVO Questionnaire

• Omnibus study to avoid selective response

• Labour and transport

• Political preference

• Judgement of PM and his image

• Religion, going to church

• Moving houses and satisfaction with house and neighbourhood

• Spontaneous and aided awareness of beer-TV brands

• Advertising awareness

• Participation in online research

• Internet usage

• Different types of scaling questions, open ended questions etc.

Participating panels NOPVO

Traditional Full Service part of international chain

Internet start ups Only completed

25 questions of ESOMAR

GfK Benelux

Intomart GfK

Millward Brown Centrum

Synovate

TNS NIPO

Flycatcher

Metrixlab

Multiscope

Netpanel

USP-MC

CIAO

PanelWizard

Trendbox

Traditional Full Service Panel Only

IBT/Team4

Interview/NSS

Motivaction

MarketResponse

RM Interactive

GMI

Lightspeed Research

SSI (Bloomerce)

Panelclix

Interesting facts of the panels

• Majority of panels exists longer than 5 years

• Different ways of recruiting used

• Almost all panels give incentives for cooperation

• Average € 1,20 for 10 minutes

• Most panels have a restriction policy in inviting panel members for a survey

• Overview of all details of all panels available at MOAWEB.NL

• Several panels do not have full details about interview history of panel members

Panel response

19.000 panel members invited….

9.462 completed the questionnaire…..

Response of 50%

75% of response is realised within 24 hours

Time Thursday Friday Saturday Sunday Monday Tuesday Wednesday Total

01.00 - 07.00 0% 4% 4% 2% 1% 4% 5% 2%

07.00 - 09.00 0% 9% 5% 3% 5% 6% 4% 4%

09.00 - 12.00 0% 23% 20% 21% 22% 17% 17% 11%

12.00 - 14.00 1% 11% 13% 17% 11% 12% 13% 7%

14.00 - 17.00 34% 17% 20% 17% 17% 13% 19% 26%

17.00 - 20.00 28% 16% 16% 17% 18% 16% 14% 22%

20.00 - 23.00 32% 13% 13% 18% 21% 21% 24% 23%

23.00 - 01.00 5% 6% 8% 5% 5% 12% 4% 6%

Total 50% 27% 8% 6% 5% 3% 1% 100%

Substantial differences in response per panel

Level of response is a result of several factors

• Panel age

• Way of recruitment

• Incentives

• Panel care

• Etc.

It is hard to determine the effects of a single factor

Response per panel

18 19

36 39 39 39 40 4149

54 55 56 60 60 62 66 68 7077

50%

Reasons for high/low response (1)

• Panels with a high response rate did not only invite their best responding panel members …

Panels with a high response rate are mainly new panels

• Panels with a low response rate are panels that do not clean up their panel database

• Panels invited panel members that did not complete one questionnaire in over a year despite many invitations

• Number of panel members is an important communication topic to the market

Reasons for high/low response(2)

• Panel age:

• Composition of samples to panel age differs a lot per panel

• New panel members respond better than old panel members

• Method of recruitment:

• Recruitment via traditional research (-instruments) gives best response (are willing already)

• Self registration or via links/banners gives significant lower response

• Incentives

• Consistency of incentive is important

• Positive effect of incentive stops at 1 euro

Composition of samples to panel age differs a lot per panel

Panel age in gross sample per panel

97

75 74

4841 37 35 32

2013 11 8 8 5 5 4 2

30

3

15 12

69

60

517

80

6 10 11 11 175 4 11

16

10 14

46 5060

51

81 79 82 81 7989 92 86

53

Panel

%

<6 months 6-12 months > 1 year

New panel members respond better than old panel members

Response and duration of panel membership

67%

61%

56%

54%

47%

47%

0 months

1-2 months

3-4 months

5-6 months

6-12 months

> 1 year

respons %

Recruitment methods vary substantially across panels

Recruitment method of panel members in gross sample

16%

15%

22%

6%

18%

1%

1%

5%

10%

5%

Buying addresses

Self registration

Via links/banners

Snowballing

By telephone

From existing panels

Advertising in print media

Via website surveys

Via traditional research

Other

Most panels use a mixture of recruitment methods

Recruitment method per panel

100

61 56

31

142

22

2 9

1814

23 18

15

36

25

62 7416

5

13

11

81

2

18

27

1

27

11

43

15

10

5

35

6

37

17

35

19

64

18

100 100%

Other

Via traditional research

By telephone

Buying addresses

Self registration

Via links/banners

Self registration or recruitment via links/banners gives significant lower response

Response per recruitment method

73%

66%

61%

57%

53%

51%

39%

39%

Existing panel

Snowballing

Via traditional research

Buying addresses

By telephone

Via website surveys

Self registration

Via links/banners

respons %

Certainty about level of incentive is important

Response per type of incentive

70%

62%

58%

49%

39%

29%

Cash

No incentive

Value points

Charity and lotteries

Value points and lotteries

Lotteries

response %

Positive effect of incentive stops at 1 euro

Response and value of incentive

59%

57%

37%

42%

> 1 euro

1 euro

< 1 euro

No (fixed) incentive

response %

Panel members love to complete questionnaires……

Reasons to participate in online research

43

38

34

21

20

26

22

17

9

8

I love completing questionnaires

I love to express myself

Because of the incentives

Curiosity

Interest in new products/ideas

%

Total reasons Most important reason

Incentives drives young people

"Because of the incentives" per age category

28

22

15

10

17

18 - 24 year

25 - 34 year

35 - 49 year

50 - 65 year

Total

%

Some other results

• Heavy Internet users respond quicker

• Socio demographics

• Women respond better than men

• Older people respond better than younger people

• Average completion time 12,5 minutes

• Fastest panel: 11,1 minutes

• Slowest panel: 15,5 minutes

• Better respondents respond ……better

Better responding panel members respond …… better

• Average response rate of individual panel members during last 12 months calculated with data about invitations and completes from panel

Average respons during last 12 months per DOPS respons category

67%

54%

31%

Complete

Stopped

Non respons

%

Is a high response rate important?

Research outcomes do not depend on response percentage

0

5

10

15

20

25

30

35

H L min max

# w

ord

s

number of words typed in open answer

Research outcomes do not depend on response percentage

voting in upcoming national elections

0

0,05

0,1

0,15

0,2

0,25

0,3

0,35

0,4

CD

A

Chr

iste

nUni

e

D66

Gro

enLi

nks

Gro

ep W

ilder

s

LPF

(Li

jst

Pim

For

tuyn

)

Pvd

A

Par

tij v

an d

e D

iere

n

SG

P

SP

(S

ocia

listis

che

part

ij)

VV

D

H

L

Research outcomes do not depend on response percentage

awareness of beer brands

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1

De

Kon

inck

Bud

wei

ser

Car

lsbe

rg

Leff

e

Cor

ona

Mae

s

Mur

phys

Ste

lla A

rtoi

s

Alfa

H

L

min

max

Research outcomes do not depend on response percentage

number of words typed in open answer

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

Working Housekeeping Non employed Unable to work Retired Student other

H

L

Research outcomes do not depend on response rate

• Low versus high responding panels

• No different means on survey questions

No response effects explained

• Non-response effects only occur if there are differences between respondents and non-respondents

• Non-response effects take place while recruiting panels

• double opt-in

• explicit consent to participate in research

• So, the relevant non-response group has been filtered out before panellists are invited for surveys

• Potential risk for non-response effects in panels:

• when response relates to survey topic (selective response)

Reponse rate: panel strategy

• A high response rate does NOT indicate a high sample quality

• Response rate is a panel strategy component, driven by rules of economics

• panel setup and maintenance determine response rate

• the value of response rate is evaluated against recruitment costs

Conclusions

Conclusions

• With online panels, only a specific group of respondents is interviewed.

• Selection takes places during recruitment, not with inviting panellist for an individual study

• Therefore, response rate does not reflect sample quality: it reflects a panel business strategy

• This self selection issue is not new to market research: we’ve seen the same with CATI and P&P

• Challenges for online panel research are to reach the other group of people

Recommendations

• Deploy other, new recruitment methods

• Remove badly responding panellists

• Use online panels for the right kind of research goals

use panels for

• continuous research or trend measurement

• concept testing

but not for

• incidence rating

• Do not swap panels in a continuous research project

• For 0 versus 1 measurement studies, match sample on panel historical data

Thank you

tedvonk@onderzoekpaleis.nl

pieter.willems@nl.millwardbrown.com