What’s really going on inside panels?

And does it matter?

________

NORSTAT SUMMER LOUNGE June 2014

Dan Kvistbo

SORT OF AN AGENDA

HOW FAR WE ARE WHERE WE ARE GOING

THE BEGINNING

BARRIERS ALL BUT GONE

1999-2001

_____

• Toluna

• GMI

• Ciao!

• Lightspeed

• Zapera

• Research Now

• etc.

_____

LOW COST | HIGH SPEED

LightSPEED | FLYresearch | ResearchNOW | SPEEDfacts etc.

FOCUS ON BENEFITS | DATA ASSUMED REP2

RESEARCH BUYERS IN LINE TO MOVE STUDIES ONLINE

CASE IN POINT | P&G

P&G | 1999: 15% 2000: 40% 2001: 50%

”To save money and expedite market research surveys”...

_____

”[...] because surveys are completed 75% faster over the

Web-and at half the cost of conventional methods [...]”

TROUBLED WATERS | 2003‘ish - 2006 ()

CULMINATION | SEP 2006 RESEARCH INDUSTRY SUMMIT

“Two surveys a week apart yielded different recommendations…

_____

I never thought I was trading data quality for cost savings”

A WOMAN WITH A BUDGET

Kim Dedeker, Procter & Gamble's VP-Global Consumer & Market Knowledge, 2006 (Now Chairman, the Americas at Kantar)

WHAT HAD GONE WRONG?

PART (I) OF THE TALE

“...clients, have put many

pressures on the research

suppliers […]”

_____

”We’ve focused on cost and

speed and an unintended

consequence was erosion in

quality. Now, we’re starting to feel

the effects”…

Kim Dedeker

PART (II) OF THE TALE

• Heavy survey taking

• Panel conditioning

• Declining cooperation

• Increase in demand

• Persistent mode differences

• Long and complex questionnaires

• Multi panel membership

• The 0.25% doing 30+%

• ‘Yes we can’

• Etc.

_____

(Source: Reg Baker, Market Strategies International)

MR INDUSTRY RESPONSE?

”It’s like the whole in the ozone

layer. Everyone knows it’s a

growing problem.”

_____

”But they just ignore it and go on to

the next project.”…

(S. Morwood, VP Worldwide Market Research for

IBM. AdAge oct. 2006)

TRUTH BE TOLD

ESOMAR 25

2005 – 2006 – 2007 – 2008 – 2009 – 2010 – 2011 – 2012 – 2013 – 2014 –

The Grand Mean Project

ARF ORQC

”Million dollar” FoQ project

ISO 26362

AAPOR Report on Online Panels

FoQ 2 (Reduce the Risk)

Report on the AAPOR task force on non-probability sampling

ESOMAR/GRBN (draft) Guideline For Online Sample Quality *(Consultation draft)

<

26 28

HALFWAY THROUGH!

BUT IN ESSENCE

“75 years of basing quantitative market research

on a paradigm of projecting the results of

samples onto populations via assumptions of

random probability sampling, has been replaced

by a willingness to use online access panels,

which do not approximate to this model.”

_____

- Ray Poynter

(Handbook of Online and Social Media Research, 2010)

AND DESPITE THE EFFORTS

Lemon market | Conditions: _____ • Asymmetry of information (buyer vs. seller) • Incentive for the seller to pass off a low-quality

product as a higher-quality one • Sellers with a bettter product have no way of

demonstrating this credibly to buyers • Deficiency of effective quality assurances/guarantees _____

http://en.wikipedia.org/wiki/The_Market_for_Lemons (Paraphrased)

Ref: The Market for "Lemons": Quality Uncertainty and the Market Mechanism

- George A. Akerlof

The Quarterly Journal of Economics, Vol. 84, No. 3. (Aug., 1970), pp. 488-500

BLACK BOX

White-box testing

_____

a method of testing [software] that tests

internal structures or workings [of an

application], as opposed to [its] functionality.

(http://en.wikipedia.org/wiki/White-box_testing)

SO... WHAT IS GOING ON INSIDE?

CLOSER...

DIRTY LITTLE SECRETS OF ONLINE PANELS

ISSUE #1 CONDUCT

_____

N = 1000, whatever it takes

ISSUE #2 RECRUITMENT

ISSUE #3 RESPONSE RATES

Human Interaction?

Trust that the survey is legitimate?

Time involvement with each respondent?

Attention given to each respondent?

Respondent control over access?

Respondent control over whether to respond?

CONSEQUENCE

The concept of TSE

gets a whole new

meaning…

• “Total” sampling error

• “Total” coverage error

• “Total” non-response error

• Etc.

IS IT EITHER OR?

A Data

Q sample

A convenience sample of a convenience sample

Random probability sample

Convenience sample

A quota sample of a random sample

Yes, you are ”...trading data quality for cost savings...”

TO CONCLUDE

“If you pay peanuts for your sample, chances are

you're getting monkeys”

(Gordon Morris Global Insights Manager at Sony Ericsson)

”I don’t want it done scientifically, I just want it

done right!”

(Client, Ron Sellers)

“Sometimes close counts in more than

horseshoes”

(Reg Baker)

LOOKING FORWARD

A few words on the future of online research

1. It’s (increasingly) mobile

2. It moves beyond self-reported attitudes,

emotions and behaviours to merge survey data and

behavioral data

3. Success depends as much on buyers

acknowledging their responsibility (ask better

questions and adapt to the realm of mobile!) as on

sample providers to deliver the appropriate sample

& platform

THANK YOU

Or google ...there’s only one of me

_____

DAN KVISTBO dan.kvistbo@norstat.dk

+45 27772016

@Kvistbo

Questions?