Run 13 RHIC Machine/Experiments Meeting 21 May 2013 Agenda : Status Reports
When to run more experiments?
161
iLive Latvia, Riga, November 12th 2015 Ton Wesseling – CEO – Testing.Agency #iLive2015- @TonW Run more experiments
-
Upload
ilive-conference -
Category
Marketing
-
view
108 -
download
0
Transcript of When to run more experiments?
i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Thank you for having me
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
On this stage
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Visit ing Riga
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
On november 11th
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
http:/ / test ing.agency
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Meeting you al l
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Conversion people
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Always fun
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
When we start cooking
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Open minded, outgoing
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
High energy & fun
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
We are
Evidence Based
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And
LEAN
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I t has been a long day
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What about your energy?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Value for your t ime and money
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Reboost your energy
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Please stand up
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Raise your hands
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Raise your hands
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
High energy conversion fun
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Calm down and focus
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Why this song?
ROAR
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Because of Katy Perry?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
No, i t ’s the name of the song!
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Today
the
ROARmodel
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
20 years ago
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I learned about him
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Quali ty circle – the start of LEAN
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
LEAN
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So I looked at data
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Made some changes
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And measured i f i t was successful
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And measured i f i t was successful
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Why can this lead to fai lures?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Average conversion rate looks stable
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
But in fact
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
There is a big weekly change!
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I only did small improvements
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So I made many wrong decisions
26 out of 44 weeks: over 5% change!
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Then I learned about this story…
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Bil l Murray
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Trying to get the gir l
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Constant test ing environment
A/B-Testing
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Methods used to improve conversion
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Methods companies plan to use
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-test ing is
HOT
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-test ing is
HOT
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Everyone is pushing to test more
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Like Amazon, Like ZalandoG
erm
an
Ecom
me
rce
mark
et,
thank
you
An
dré
Mory
s
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Never sat isf ied with conversion rates
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Start test ing or test more
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
NO
Growth problems wil l disappear…
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-test ing is not a solut ion
It’s a methodology
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I t ’s part of a:
Continuous
Optimization
Program
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
I t ’s a way of working
It’s company DNA
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You need to know:
When
How
What
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Growth curve – how it was teached
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
the
ROARmodel
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
How much impact do you need?
http://ondi.me/size
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month
15% impact
needed
5% impact
needed
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Optimizat ion phases - ROAR
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Phase 1: br idge the gap – Risk
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Phase 1: br idge the gap – Risk
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Get out of the off ice
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Risk
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Yes, I ’ve got those 1000+ conversions
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
With 1000+ conversions a month
Your tes t capac i t y is max.
20+ tests a year
wi th a s ta t i s t i ca l power o f 80%
on pred ic ted up l i f t s o f 15%
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests with measurable impact
On average
1 out of 3
Simple truth: you’re just not always able to create a winner with enough impact
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
So in this case, you’ve got impact
Every 7 or 8 weeks
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Slower? Energy wil l run out!And your continous optimization program will die
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Business cases!
Delivery –test with
in stock. 4 weeks
or 2 days?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Business cases!
Service charge, do
it or not? Include in
price or not?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
What to test? Big design changes
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Optimizat ion – moving up
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Final ly you can test detai ls…
Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You should grow to
Customer Buying Reasons
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Reasons behind the customer journey
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A continuous optimizat ion team
Analyst Psychologyst
Designer Developer
Team Lead
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
They keep on asking:
Why are these users here?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And they want to know
How they can
help their users’ brains
to fullf i l their needs?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Data & Psychology
Digital Data Persuasion Psychology
Analyze
&
Experiment
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A lack of resources
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Testing takes resources
Use it to LEARN
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A n d t h e n s t a r t : FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFindFind
Your optimization team needs a good analyst / researcher
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Many sources out there to Find
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Analyze
Really a good analyst / researcher!
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Use your analyt ics data
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Create heatmaps
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Create screen recordings
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
To find out: The cr i t ical spots
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Create
And yes, your team needs an economic / consumer Psychologyst and a UX designer
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Source: Psychology
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Two systems
107
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Two systems that intervene
108
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
System 1 taking over
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
And the automated respons rules
110
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Even when i t ’s wrong
111
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Use that persuasion knowledge
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Tests that can be done
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind Test
And your team needs a good front-end developer
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Get your A/B-solut ion in p lace
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You need Front end development
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellConcludeAnalyze
TestCreateAnalyzeFind
Analyze
Your analyst will analyze the results
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Look at detai ls
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
FA C T & A C T
&
TellCombineAnalyze
TestCreateAnalyzeFind
TellConclude
Your psychologyst should conclude and your conversion lead should tell
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Store your learnings
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Push your behavioral insights
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Everyone should use your knowledge!
Data Insights Action
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
+ Optimization + Automation
Back to the ROAR model
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Optimizat ion – moving up
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now you’re test ing
On your way to
4 tests a week
Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Test capacity
maximum number of
experiments per yearthat your site or app
can handle
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Regret
amount of money
you are losing per year by
not using your full test capacity
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Automation
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Automation
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You tested which locations are cr i t ical
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You know which dialogues are cr i t ical
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Automation
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now you’ve moved to
Algorithm testing
Which internal and external influencers do have an impact on which segment of users?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
+ Optimization + Automation
ROAR – Automation
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – You don’t stop optimizing
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
ROAR – Re-think
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
1.000 conversions
per month
10.000 conversions
per month
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-test ing is the ult imate resource
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
More experiments
“If you double the number of
experiments you do per year
you’re going to double your
inventiveness”
Jeff Bezos, CEO Amazon
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
CRO brings user knowledge
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Store your learnings
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
You wil l make big impact on Discover
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now, that ’s
HOT
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Now, that ’s
HOT
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Done test ing?
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Done test ing? No!
TESTING NEVER STOPS
Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
A/B-test ing
Minimize regret
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Test as much as you can
Time span
Co
nve
rsio
ns
pe
r m
on
th
Risk + Optimization + Automation Re-think
10.000 conversions
per month
1.000 conversions
per month
15% impact
needed
5% impact
needed
Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Like Amazon, Like ZalandoG
erm
an
Ecom
me
rce
mark
et,
thank
you
An
dré
Mory
s
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Experimenting is a way of working
It’s company DNA
T o n W e s s e l i n g | @ T o n W R u n m o r e e x p e r i m e n t s # i l i v e 2 0 1 5
Run more experiments
i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5
To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y
# i L i v e 2 0 1 5 - @ To n W
Run more experiments
