PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... ·...
Transcript of PRIMARY RESEARCH - Marketing Intelligenceintelligence.marketingonline.nl/sites/default/... ·...
THE FUTURE PREDICTING
PRIMARY RESEARCH
1
SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?
PRIZE: A REAL CRYSTAL BALL
PART 1: MAKES SOME PREDICTIONS
IN SEPT 2013 WE ASKED A GROUP OF 400 UK PANELIST TO PREDICT THE SELLING PRICE OF THE IPAD MINI 3 WEEKS PRIOR TO ITS LAUNCH. HOW CLOSE DID THEY GET?
WHAT IS YOUR PREDICTION? WITHIN 10%, WITHIN 5%, WITHIN 3%, WITHIN 1%
*Note we told them the existing selling price of the old model
PREDICT THE PRICE OF 100ML OF CHANEL PERFUME?
In €
I AM GOING TO TOSS A COIN WILL IT BE HEADS OR TAILS
ADDITIONAL QUESTION…
PREDICT HOW MANY SAID HEADS?
WHAT PROPORTION OF WINE DRINKERS IN THE UK
PREFER RED WINE? BASED ON A POLL OF 400 WINE DRINKERS IN UK WHO WERE ASKED IF THEY PREFER RED OR WHITE
WINE
WHAT % DRINK RED?
PREDICT HOW MANY RESEARCHERS CHECK THEIR EMAIL BEFORE BREAKFAST? BASED ON POLL OF ATTENDEES AT ESOMAR CONGRESS 2014
DO YOU CHECK YOUR EMAILS BEFORE BREAKFAST?
England v Montenegro +3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3
PREDICT WHAT MARGIN OF VICTORY OUR UK PANELISTS PREDICTED FOR THESE 2 FOOTBALL MATCHES
WILL THE MARKET RESEARCH INDUSTRY BE BIGGER OR SMALLER IN 10 YEARS TIME?
The CXO Advisory group gathered 6,582 buy or sell predictions from 68 different investing gurus made between 1998 and 2012, and tracked the results of those predictions. How accurate were they?
WHAT % WERE CORRECT?
SWAP YOUR QUIZ SHEET WITH THE PERSON NEXT TO YOU READY FOR MARKING
BACKGROUND Gamification à More prediction protocols in surveys à Fostered an interest in the science of prediction à Led to a series of dedicated prediction experiments à Exploration of the world of prediction market trading à Prediki
• 30+ Primary research experiments
• 500+ Predictions analysed
• 60+ Prediction markets v traditional research comparisons
THE TYPES OF EXPERIMENTS WE HAVE RUN
• Betting on the future of brands • Predicting why people buy things • Predicting the behaviour of other people • Predicting the price of things • Predicting the election prospects of political parties • Predicting football match results • Predicting the outcomes of TV game shows • Predicting the success of adverts • Predicting future sales of products • Predicting the future more generally
SO WHAT HAVE WE LEANT ABOUT PREDICTION?
20%
42% 43%
32% 30%
38%
11%
21% 21% 22% 22%
29%
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Consumer purchasing estimates
Observed behaviour of
others
Observed opinion
Forecast Price prediction
Guesswork
Correct prediction Random chance
WHAT ARE WE GOOD AT PREDICTING AS INDIVUALS?
SOME OF US ARE BETTER AT PREDICTING
0%
20%
40%
60%
80%
100%
120%
140%
160%
180%
score 1 score 2 score 3 score 4 score 5 score 6 score 7
Index of Prediction performance over 7 waves of experiments
top 100
bottom 100
!"#$%&'%() *+,-%'. = /)0("1,'%() ×200("' ×345#&'%6%'.×(1−8%00%&+-'.) × :,)$(1)#;;
Note: Not prediction accuracy is not directly dependent on sample size
One person with access to the right information and able to objectively process it can be just as good as much larger groups. e.g. Nate Silver: Correctly predicted the outcome of all 52 states in the 2012 UK election
16 IS A CROWD Jed Christianson, University of Birmingham calculates
LESS ABOUT SAMPLE SIZE MORE ABOUT SAMPLE DIVERSITY & INTELLIGENCE
Jed Christianson, estimates that the quality of consolidated group predictions does not improve much beyond 16 participants, if they are all well informed.
..AND HOW YOU AGGREGATE CROWD WISDOM
MEAN, MODE, MEDIAN V TRADING & DOUBLE AUCTION TRADING
1906 Plymouth County fair
Actual weight = 1198 lb Median average guess = 1207lb Error = <1%
WHAT WE KNOW
THE WISDOM OF CROWDS
Using larger samples and median aggregation techniques can help consolidate wisdom in situations where information and knowledge is dissipated.
CROWD WISDOM IS BASED ON FILTERING THE SIGNAL FROM THE NOISE
Each person’s prediction is made up of 2 components: information & error.
If each individual’s judgement is independent & unbiased then the error will largely cancel itself out and the aggregation process then distils off the
inherent knowledge.
Actual selling price = £319
Median average guess = £316 Error = 1%
SCORE: 1 POINT
2013 GMI online sample
WHAT WE KNOW
THE WISDOM OF CROWDS
WITHOUT COLLECTIVE KNOWLEDGE CROWDS CAN BE PLAIN IGNORANT
WARNING
2014 GMI online sample
Actual weight = 550kg
AN UNWISE CROWD
Median average guess = 350kg Error = 36%
Where there is no wisdom to consolidate, crowds just generate noise.
CROWD WISDOM CAN BE A BIT BEHIND THE TIMES
-44%
-34% -33% -30%
-28% -26% -25% -25%
-17% -14%
2% 5%
9% 9%
21% 23%
-50%
-40%
-30%
-20%
-10%
0%
10%
20%
30%
Examples of price predicition errors
We measured an average 9% price lag on aggregated price predictions – our memory of price is based on historical purchase activity which is often lower that current pricing.
94% 87%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Women Men
Price prediciton accuracy
Men are less price savvy than women
Crowd €80 Actual €120
YOUR PREDICTIONS?
SCORE: +/-$10 2 POINTS +/-$20 1 POINT
Aggregating crowd wisdom: “If each individual’s judgement is independent & unbiased then the error will largely cancel itself out”
THE PROCESS OF MAKING PREDICTIONS IS LITTERED WITH COGNITIVE BIASES
68% predict heads DUE TO ORDER BIAS
HEADS OR TAILS?...YOUR PREDICTIONS
SCORE: +/-5% 2 POINT +/-10% 1 POINT
WITH NO INFORMATION TO GUIDE US TINY NUDGES CAN HAVE BIG EFFECTS ON OUR PREDICTIONS
HOW THIS EFFECT CORRUPTS PREDICTIONS….
54%
46% 46%
54%
0%
10%
20%
30%
40%
50%
60%
White Red
What percentage of people do you think prefer white wine?
What percentage of people do you think prefer red wine?
V
A 20% shift in prediction based simply on naming the colour of wine in the question!
IN THE ABSENCE OF KNOWLEDGE WE PREDICT THE MAJORITY OF OTHER PEOPLE WILL DO & THINK THE SAME AS US
- 10 20 30 40 50 60 70 80 90
100
Believers
Believe this How many people believe this
- 10 20 30 40 50 60 70 80 90
100
Non believers
Don’t believe How many people don't believe this
those holding minority opinions assume more people agree with them than those holding the majority opinion: is this the definition of delusion?
WHAT WILL THE WORLD BE LIKE IN 2050?
55% 68%
CHECKING EMAIL YOUR PREDICTIONS
SCORE: +/-5% 2 POINT +/-10% 1 POINT
60%
20%
I check my emails before breakfast
I don't check my emails before breakfast
Prediction of how many other people at Lightspeed GMI check their emails
before breakfast*
*Source: office poll!
80% of Researcher said they check their email before breakfast
OUR EMOTIONS REALLY CAN DOMINATE & BADLY DISTORT OUR PREDICTIONS
England v Montenegro +3 +2 +1 0 -1 -2 -3
Germany v Rep. Ireland +3 +2 +1 0 -1 -2 -3
PREDICT WHAT SCORES OUR UK PANELISTS PREDICTED!
SCORE: CORRECT = 1 POINT PER QUESTION
0%
10%
20%
30%
40%
50%
60%
new by 3 new by 2 new by 1 draw liv by 1 liv by 2 liv by 2
Newcastle Liverpool
0%
10%
20%
30%
40%
50%
chel by 3 chel by 2 chel by 1 draw car by 1 car by 2 car by 3
Chelsea Cardiff
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
man by 3 man by 2 man by 1 draw south by 1
south by 2
south by 3
Southhampton Man U
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
man by 3 man by 2 man by 1 draw south by 1
south by 2
south by 3
Southhampton Man U
FOOTBALL SCORE PREDICTIONS
These are the comparative football score predictions of competing fans.
0%
10%
20%
30%
40%
50%
60%
70%
80%
Conservative Government
Labour Pary Government
Liberal Democrats Government
Conservative & Liberal Coalition
Labour and Liberal Coalition
Conservative, Liberal, & UKIP
Coalition
Conservatives Labour LiberalDemocrats UKIP
PREDICT WHO WILL FORM THE NEXT UK GOVERNMENT: BY PARTY AFFILIATION
The issue of emotional bias is particularly significant in the political arena where voting intention and predictions on who is likely to form a future government almost completely correlate.
DIFFERENCES BETWEEN PREDICTING WHAT OTHERS WILL DO
V
WHAT I WILL DO
SOCIAL COGNITIVE BIASES RENDER PREDICTIONS ABOUT OUR OWN BEHAVIOUR PARTICULARLY DIFFICULT
Will you tidy up after the meeting? Yes = 50%
TIDIED UP =13%
Predict how many will tidy up? = 15%
…we are much better at predicting the behaviour of others than our own behaviour. We like to think well of ourselves!
WE ARE OFTEN TOO TIED UP IN THE DETAIL TO SEE THE BIGGER PICTURE
WILL THE MARRIAGE LAST? Yes/No Parents are much better than the married couples at predicting this
Source: Queens University Canada
0,9 0,91 0,92 0,93 0,94 0,95 0,96 0,97 0,98 0,99
1
Experts Dillettantes (non
experts)
Chimps (random guesses)
Future Predictions accuracy
CA
LIBR
ATI
ON
Highly recommended reading
PHILIP TETLOCK STUDYING 15,000 GEO-POLITICAL PREDICTIONS
0
0,01
0,02
0,03
0,04
0,05
0,06
Experts Dillettantes (non experts)
Chimps (random guesses)
Descrimination
Philip Tetlock
EXPERT POLITICAL JUDGEMENT
What he describes as “Dillettantes”, smart people with no specific expertise in the topic, prove to be more effective at making geo-political predictions than political pundits/experts.
HOW MANY INVESTMENT GURUS 6,500 STOCK MARKET PREDICTIONS WERE CORRECT?
SCORE BELOW 50% = 1 POINT
48% 2% less accurate than a coin toss!
WE ARE UNABLE TO SEPARATE THE SIGNAL FROM THE NOISE
Will the Market Research industry be bigger or smaller in 10 years time? Yes/No?
I don’t know but we in the market research industry are not probably the best people to ask.
HOW TO WORK THE CROWD! EXPLORING THE BEST TECHNIQUES TO USE TO EXTRACT RELIABLE PREDICTIONS
SWAP BACK YOUR QUIZ SHEETS
10 VOLUNEERS
WHICH OF IS THE MOST POPULAR AD?
BET 1, 2, 3, 4, 5 POINTS
WHICH OF THESE CANDLES SELLS THE MOST
BET 1, 2, 3, 4, 5 POINTS
WHICH OF THESE MUGS SELLS THE MOST?
BET 1, 2, 3, 4, 5 POINTS
Clue: These are mostly purchased as gifts for other people Clue 2: these are sold to English people!
Camping Cooking Gardening DIY (Home improvement)
-200%
-150%
-100%
-50%
0%
50%
100%
150%
200%
250%
300%
350%
Evaluation of 20 different ads Monadic rating Performance prediction: Net Investment
0.89 correlation 5x differentiation
ADVICE: MAKE IT REWARDING
All the research out there show us that prediction made under competitive conditions where you can have the opportunity to win or lose, results in participants investing more thought and effort and this in tern improves the quality of predictions.
WEIGHT BASED ON CONFIDENCE
33% 37% 36%
44%
0% 5%
10% 15% 20% 25% 30% 35% 40% 45% 50%
Total Guess Have a Hunch Fairly Sure Very Confident
Prediction accuracy
In our analysis of 500 predictions, where in each case we asked respondents how confident they were, we found the predictions from the most confident groups to be slightly more accurate.
MONEY BET IS A PROXY TO CONFIDENCE
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
-3 -2 -1 +1 +2 +3
Bet amount: correlation with outcome
We also found an interesting relationship between money bet and prediction accuracy. A non-linear relationship. Small bets were less accurate than medium bets but larger bets were not more accurate than medium bets. The most accurate bets overall were in fact those that bet small amount against.
EXPLOIT PREDICTION MARKET TRADING
• Prediction markets trading is a process where participants are given money to buy and sell shares in the outcome of an event with a chance to win real money.
• The most famous example being the Iowa Electronic Market a prediction market set up to predict the outcome of elections around the world: ◦ In 451 out of 596 head to head comparisons, prediction
market trading by groups on average of less than 20 people has more accurately predicted the result that the most accurate local opinion polls.
• We conducted 32 head to head experiments to compare the prediction from prediction markets trading to predictions made in traditional surveys
• A survey based approach with random cells* of 15 participants who were asked to predict which products would sell more vs 15 people prediction markets trading – asked to buy or sell variable amounts to create a confidence weighted market
TESTING OUT USING PREDICTION MARKETS FOR MORE TRADTIONAL MARKET RESEARCH PURPOSES
37%
55%
65%
0%
10%
20%
30%
40%
50%
60%
70%
Random guess Micro survey (sample of 15)
15 people prediction market trading
HEAD TO HEAD COMPARISON
Source: GMI/Prediki based on 32 direct head to head comparisons
Comparing the performance of small head to head samples, prediction markets trading groups did more accurately make correct predictions. The trading process more effectively aggregating group wisdom
SOME ISSUES THOUGH
OPINIONS IN PREDICTION MARKET TRADING CAN QUICKLY BE SET IN STONE IF NO NEW INFORMATION IS ADDED
ADDING MORE PEOPLE AFTER A CERTAIN POINT DOES NOT CHANGE THE RESULT
When we used larger samples of respondents in prediction markets, adding more respondents did not improve the quality of predictions – predictions rather rapidly got fixed in stone. The betting intentions of the earlier participants was having a significant impact on the later participants.
37%
55%
65% 69%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Random guess Micro survey sample of 15
15 people prediction
markets trading
Standard survey: sample of 200
HEAD TO HEAD COMPARISON
The aggregated prediction of larger samples of people made standard surveys proved to be more accurate. These predictions were all made in isolation from the influence of other people’s opinion.
INFORMATION SHARING IS KEY PREDICTION MARKETS FEED OF INFORMATION
• These initial experiments we did not allow the participants in the prediction markets trading process to share opinions all they could see was what others had bet.
• We then conducted a second series of experiments where we injected some intelligence by encouraging participants to discuss their decisions and allowed them all to see what other people thought.
MARKETS WOULD REACT WHEN WE ADDED INFORMATION
Self generating clues
The participants began to self issue clues that would help the group collectively make more thoughtful predictions. The example above show how the market reacted to thoughtful comments made about which product they felt would sell more.
THINK OF A QUESTION AS A CONUNDRUM: INFORMATION
PROVIDES CLUES TO HELP PEOPLE SOLVE THE PROBLEM
DIALECTICAL BOOT STRAPPING ENCOURAGING CROWDS TO SELF-GENERATE THE INSIGHTS
NEEDED TO SOLVE PREDICTION CONUNDRUMS
Example = Board room decisions
Useful reference: Herzog and Hertwig (2009)
37%
55%
65% 69%
81%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Random guess Micro survey (sample of 15)
15 people Micro prediction market
trading
Standard survey (sample of 200)
Micro prediction market with shared information & free
comments
THE VALUE OF ADDING INFORMATION TO PREDICTIVE MARKET TRADING SYSTEM
In this second series of experiments the prediction accuracy of these small group of individuals sharing information in prediction markets moved ahead of the predictions from larger samples of respondents in standard surveys
THE PROBLEM OF HERDING EFFECTS
Information cascade
However there predictions were still not perfect, stray comments from individuals could occasionally steer the prediction markets off course.
STUDYING THE IMPACT OF NUDGE EFFECTS: THE INFLUENCE ONE PERSON’S OPINION HAS ON ANOTHER
2%
6%
11%
15%
20%
Personal preferences Self evident predictions (e.g. ad
evaluation)
Factual (requiring knowledge)
Inverted personal preference (e.g. pedicting relative levels of dislike)
Complex estimates
Nudge influence by prediction task
Source: GMI research 2014
We conducted some unique experiments to measure the influence that one persons opinion could have on another and found the less certain people are and the harder the prediction is, the more we rely on other people’s opinions
OUR SOLUTION = DIVIDING UP THE HERD
A simple solution to this is to run multiple prediction markets in isolation from each other and then aggregating their predictions.
37%
55% 65% 69%
81% 93%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
100%
Random guess Micro survey (sample of 15)
15 people Micro
prediction market trading
Standard survey
(sample of 200)
Micro prediction
market with shared
information & free comments
Aggrigation of 3 divided
herds
HEAD TO HEAD COMPARISON
By doing this we were able to improve the prediction accuracy from prediction markets even further.
Please note these experiments must be treated only as anecdotal evidence to illustrate these issues. This is only one small series of experiments. A lot more work is need to properly validate these findings.
EFFECTIVE USE OF PREDICTION MARKETS
• Incentivise - ideally with real money! • Allow active & dynamic trading • 16 is a crowd • Share as much information as possible • A moderator is important to stimulate debate and share
information • Divide the herd: run multiple micro markets
SAMPLING 100’S SMALLER SMARTER GROUPS BEING ASKED SMARTER QUESTIONS & SHARING THOUGHTS & OPINIONS
SO WHO IS THE BEST PREDICTOR IN THE AUDIENCE?
PRIZE: A REAL CRYSTAL BALL