Will goodhand advanced quant - 2011 - 1

Event sponsored by Affinnova All copyright owned by The Future Place and the presenters of the material For more informa=on about Affinnova visit h>p://www. affinnova.com/

For more informa=on about NewMR events visit newmr.org

Advanced Quant Techniques July 14, 2011

9,000 Concepts Later: How the wisdom of the crowd compares with monadic screening

Will Goodhand, Brainjuicer

2

9,000 Concepts Later: How the wisdom of the crowd compares with monadic screening

Will Goodhand, BrainJuicer, UK NewMR Advanced Quant Techniques, July 14, 2011 Feb 2011

3

S  IEM is 500 non-rep Americans betting on election outcome

S  IEM predictions of 600+ elections over 20 years were more accurate than most accurate opinion poll 75% of the time

S  IEM shows Predictive Markets’ potential vs. MR’s current gold standard (large, rep samples etc)

S  Francis Galton 1822-1911 (self-professed elitist) unexpectedly discovered crowds could be wise, whilst trying to prove that democracy was a poor idea

S Guess the weight of the Ox showed a crowd of 800 was essentially perfect [1197 vs. 1198lb], better than any individual (including livestock experts) within it

The Wisdom of Crowds

The Wisdom of Crowds The crowd is wise when, Diverse,

Independent & faithfully aggregated James Surowiecki (2004)

4

Market Research Application Predictive Markets

Will Goodhand, BrainJuicer, UK NewMR Advanced Quant Techniques, July 14, 2011


5

Predictive Markets: Changing the rules of concept screening; 3 main benefits

Convenience sample of 500, not targeted

Ask how market will react, not personal preference

Comparative x15, not monadic

Proven Accuracy

Greater discrimination

Spots breakthrough ideas


6

Predictive Markets – ‘Imagine you owned shares in all these ideas...’

Probably Sell Shares In... Probably Buy Shares In...

Which One To Sell All Shares In... Which One To Double Shares In...

Net Preference % %

Full evaluation of ‘reasons why’

Full evaluation of ‘reasons why’

Provides Normable Benchmark


7

-5 -4 -4-2

-4-2 -2

-4 -4-2

-7-5 -5

-9 -9 -9

-14

-20

-5

-9-7

-12 -12

-20

16

12 129

7 75

75

25

2 24 4 4 4 4

11 129

119

14

11

9 9

74

-5

-4-2

2

45

-16-11-5-5-5-4

2 -2

-40

54

4

-20

-10

0

10

20Co

ncep

t A

Conc

ept B

Conc

ept C

Conc

ept H

Conc

ept G

Conc

ept L

Conc

ept F

Conc

ept J

Conc

ept I

Conc

ept E

Conc

ept D

Conc

ept T

Conc

ept K

Conc

ept O

Conc

ept S

Conc

ept N

Conc

ept M

Conc

ept W

Conc

ept U

Conc

ept R

Conc

ept Q

Conc

ept P

Conc

ept V

Conc

ept X

% o

f res

pond

ents

Net PreferenceMost Successful - would double sharesLeast Successful - would sell shares

And Discrimination via Net Preference = Better Decision Making Please select which one of these ideas you would immediately sell / double your shares in

Total Sample: 889

Strength of Portfolio*55 51

Food Portfolio Norm This Test


8

1. Proven accuracy

2. Greater discrimination

3. Spots breakthrough ideas

Predictive Markets


9

In market validation – 4 Quick Case Studies

Green lighted ideas that have gone on to perform well in market….


10

Picks out the Same Winners as Monadic Testing

Monadic Ideas Tested No. of Green Lighted Concepts Snacks 15 5 out of 5 Household cleaners 10 5 out of 5 Air Care Products 5 2 out of 2 Non-alcoholic Drinks 8 1.5 out of 2# Still Waters 11 1 out of 2# Average Correlation 0.91 Sequential No. of Green Lighted Concepts Telecoms 42 8 out of 11 Chilled Food 11 3 out of 3 Cleaners 29 6.5 out of 9 Chewing gum 7 4 out of 4 Lighting ideas 20 9 out of 10 Air Care Products 28 5 out of 7# Average Correlation 0.81

# ‘0.5’ indicates a concept given a green light in tradiGonal test and pause for thought in PM (i.e. progress aKer re-‐framing)


11

0

10

20

30

40

50

60

70

80

90

100

% o

f res

pond

ents

Concept A on 97th percentile vs Predictive Market Probably

Buy Norms

Concept A on 96th percentile vs Monadic Concept Test Purchase Intention Norm

Predictive Market Normative Database

Monadic Concept Test Normative Database

97th percentile 96th percentile

PM consistent with later stage traditional monadic testing


12

Most encouraging

Least encouraging

Encourage to find out more Least likely

Most likely

Like

lihoo

d to

sto

p an

d re

ad PCE

BM

PPF

SW

NR

MH

Predictive Even in Niche Markets: Crowd Selected Same Top 3 as Rheumatologists

Net Preference Score

SW

MH BM

Monadic Test Key Metrics Highly Targeted Audience = Rheumatologists

Predictive Market Key Metrics Convenience Sample of 500

= Same Result

% H

appi

ness

NR

PCE PPF

Spearman’s Rank Correlation Net Pref vs Rheumatologists survey overall metric = 0.89

Represents a 90% reduction in the cost of medical concept testing


13

Comparability of tests for screener over time

Yes; ‘probably buy / probably sell’ sifting question allows comparisons over time, in different contexts, and is normable

35 38 39 39 41 45 46 48 49 50 51 52 57 61 6248

BaggedSnack

39 40 45 46 47 47 48 51 54 54 6247

BaggedSnack

36 40 41 41 41 42 44 48 49 50 5348

BaggedSnacks

% probably buy

Bagged Snack

Bagged Snack

Bagged Snack

Test 1

Test 2

Test 3

More Stable Cross Country Norms

53 Germany 59 Germany

54 UK 52 UK

56 US 56 US

59 France 70 France

17 Range 48 Range

50 Min 43 Min

67 Max 91 Max

58 Russia 68 Russia

67 Turkey 65 Turkey

50 Netherlands 43 Netherlands

67 Central/South America 91 Central/South America

59 Spain 60 Spain

Food & Bevs %

probably buy more

shares

Country Food & Bevs % top2box

PI Country

PREDICTIVE MARKETS NORMS

CONCEPT OPTIMIZER NORMS

S  Allows easier cross country comparisons and means fewer databases have to be kept

CONCEPT OPTIMIZER NORMS

PREDICTIVE MARKETS NORMS

Category

All UK concepts

% Top2Box PI

Category

All UK concepts

% probably buy more

shares

Average Average

Food 54 Food 54

Technology 60 Technology 60

Household 61 Household 52

Beverages 47 Beverages 55

Toiletries 55 Toiletries 55

Games/Toys 42 Games/Toys 58

Max 61 Max 60

Min 42 Min 52

Range 19 Range 8

More Stable Cross Category Norms too


16

1. Proven accuracy



Predictive Markets


17

Greater discrimination vs. monadic tests Please select which one of these ideas you would immediately sell / double your shares in

Total Sample: 500

-9 -7 -10-3

-30

31

19 17

7 713

6

22

127

4

-26

-14

-24

-13

-7

-35

-25

-15

-5

5

15

25

35

Concept 1 Concept 2 Concept 3 Concept 4 Concept 5 Concept 6 Concept 7

% o

f res

pond

ents

0

10

20

30

40

50

60

70

80

90

100

Sequ

entia

l Mon

adic

% D

ef/P

rob


Flat line monadic scores

Wheat from chaff


18

Better Distinguishes Wheat from Chaff (normalised range for each test type)

Scores shown represent normalised range ie. the range divided by the standard error (of the % net preference in the PM or %TTB in the monadic test)

Monadic Ideas Tested

Range of Net Preference [Predictive Markets]

Range of PI [Monadics]

Air Fresheners 5 19.6 4.8

Savoury snacks 15 26.0 6.1

Household cleaners 10 26.9 10.7

Average 24.2 7.2 Sequential

Telecoms UK 14 12.8 7.8

Telecoms France 14 14.2 8.9

Telecoms Spain 14 20.3 18.4

Chilled Food 11 14.3 14.0

Window cleaners 17 27.6 18.1

Chewing gum 7 20.8 4.1

Cereals 11 17.2 2.0

Average 18.2 7.3


19

-‐24

-5-7 -7 -6

-10 -11 -9

-17

-28

20 2016

13

8 84

74

15 13

9

7

-5

-9-4-3

-25

-20

-15

-10

-5

0

5

10

15

20

25

Concept 1 Concept 2 Concept 3 Concept 4 Concept 5 Concept 6 Concept 7 Concept 8 Concept 9

% o

f res

pond

ents


Nigeria: Very clear results even in a high scoring ‘everyone is a winner’ culture Please select which one of these ideas you would immediately sell / double your shares in

Strength of Portfolio

69 76

Portfolio Norm (SubSaharan Africa)

This Test

Performance v norms Top Quar5le 2nd Quar5le Bo9om 50%

%s based on total seeing each concept: c500


20

1. Proven accuracy



Predictive Markets


21

Amber Lights: Alvin and the Chipmunks

S  The Predic=ve Market revealed this as a polarising film idea, with detractors seeing it as boring and only for nostalgic baby boomers and proponents seeing it as being great for children and families.

S  The concept clearly had energy with a large propor=on of the sample choosing to trade the idea. 17% traded Alvin & the Chipmunks vs an expecta=on of less than 12%. A classic light concept with poten=al given the right targe=ng

−  “Sorry, it is about a boring subject, can't be improved.”

−  “Done right, this flick has enormous poten5al.”

−  The predic=ve market indicated −  targe=ng would be key; families,

baby boomers with grandchildren −  music should appeal to this target

group −  ‘outdated’ Chipmunks need to be

completely refreshed for this film to work

-2 -2-4 -3

-5-8

-4 -5 -4 -4 -3 -3

-9-11 -10 -11 -13

1916

129

69

5 5 4 3 2 13 2 1 1 2

-25

-20

-15

-10

-5

0

5

10

15

20

25

I AmLegend

NationalTreasure

The GoldenCompass

CharlieWilson's

War

P.S. I LoveYou

Alvin andthe

Chipmunks

Sw eeneyTodd

ThePerfectHoliday

The BucketList

The WaterHorse

The GreatDebaters

Atonement Aliens Vs.Predator

The KiteRunner

YouthWithoutYouth

JUNO Walk Hard

% o

f res

pond

ents

Most Successful - would double shares

Least Successful - would sell shares

Net Preference


22

Impressive Record of Predicting In-Market Reality

Box

Offi

ce ta

king

s

Correlation 0.81

Net Preference

$103.3

$78.4

$31.9

$22.7

$14.3

$55.9

$18.7

$3.2 $0.8

$29.6

$0.7 $0.1 $0.6

$8.8$14.7

$1.1

$19.1

-15

-5

5

15

25

35

45

55

65

75

85

95

105

115

I AmLegend

NationalTreasure

The GoldenCompass

CharlieWilson's

War

P.S. I LoveYou

Alvin andthe

Chipmunks

Sw eeneyTodd

ThePerfectHoliday

The BucketList

The WaterHorse

The GreatDebaters

AtonementAliens Vs.Predator

The KiteRunner

YouthWithoutYouth

JUNO Walk Hard

Tick

et S

ales

(in

mill

ions

)

17 14 9 6 1 1 1 0 -1 -1 -1 -1 -5 -9 -9 -10 -11

S  PM predicted 2 top films of 17 (in right order) & 3 of top four

S  Alvin & the Chipmunks a classic (‘pause for thought’) polarising concept = potentially breakthrough idea, which turned out to be successful.

0

20

40

60

80

100

120

-30 -20 -10 0 10 20 30

Industry Benchmark for Success


23

Confirming what we know about Predictive Markets

Predictive Markets Proven accuracy

Greater discrimination Stable norms

Can spot breakthrough ideas Rich qualitative diagnostics

Cost effective 15 ideas x 500 2 weeks from brief to debrief

Monadic Testing ═  Equally accurate for top ideas �  Less discrimination �  Variable to volatile norms �  Likely to miss breakthrough ideas �  Thin open-end diagnostics �  Expensive 1 idea x 150 �  5 weeks from brief to debrief

Comparing 400+ Concepts 20+ Categories

Award Winner

2007

2005

�  Less variability cross countries �  Less variability cross categories �  Little subgroup variation �  Effective in highly specialized categories �  Wide range of potential applications


Q & A

Will Goodhand BrainJuicer

Sue York The Future Place


25

Contact us www.brainjuicer.com

Will Goodhand Juicy Evangelist [email protected] +44 7980 673 186

Will goodhand advanced quant - 2011 - 1

Marketing

Transcript of Will goodhand advanced quant - 2011 - 1