Run more experiments - Ilive 2015 Latvia, Riga

i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# i L i v e 2 0 1 5 - @ To n W

Run more experiments

Ton Wesseling | @ TonW Run more experiments #ilive2015

Thank you for having me


On this stage


Vis i t ing Riga


On november 11th


http://test ing.agency


Meeting you al l


Conversion people


Always fun


When we start cooking


High energy & fun


We are

Evidence Based


And

LEAN


I t has been a long day though


What about your energy?


Value for your t ime and money


Reboost your energy


Please stand up


Raise your hands


High energy conversion fun


Calm down and focus


Why this song?

ROAR


Because of Katy Perry?


No, i t ’s the name of the song!


Today

the

ROAR model


20 years ago


I learned about him


Qual i ty c i rc le – the start of LEAN


LEAN


So I looked at data


Made some changes


And measured i f i t was successful


Why can this lead to fa i lures?


Average conversion rate looks stable


But in fact


There is a big weekly change!


I only did smal l improvements


So I made many wrong decis ions

26 out of 44 weeks: over 5% change!


Then I learned about th is story…


Bi l l Murray in Groundhog Day


Trying to get the gir l


He has a stable test ing environment

A/B-Testing


Methods used to improve conversion


Methods companies plan to use


A/B-test ing is

HOT


Everyone is pushing to test more


Like Amazon, Like Zalando G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon


Never sat isfied with conversion rates


Start test ing or test more


NO Growth problems wi l l d isappear…


A/B-test ing is not a solut ion

It’s a methodology


I t ’s part of a:

Continuous Optimization

Program


I t ’s a way of working

It’s company DNA


You need to know:

When

How What


Growth curve – how i t was teached


the

ROAR model


Optimizat ion phases - ROAR

Time span!

Con

vers

ions

per

mon

th!



Time span!

Con

vers

ions

per

mon

th!

Risk!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation!



Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation! Re-think!



Time span!

Con

vers

ions

per

mon

th!


1.000 conversions!per month!



Time span!

Con

vers

ions

per

mon

th!


10.000 conversions per month!



How much impact do you need?

http://ondi.me/size!



Time span!

Con

vers

ions

per

mon

th!




15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!



Time span!

Con

vers

ions

per

mon

th!




Phase 1: br idge the gap – Risk


Get out of the office


Risk


Yes, I ’ve got those 1000+ conversions

Time span!

Con

vers

ions

per

mon

th!





With 1000+ conversions a month

Your tes t capac i ty i s max .

20+ tests a year

wi th a s ta t i s t i ca l power o f 80% on pred ic ted up l i f t s o f 15%


Tests with measurable impact

On average

1 out of 3

Simple truth: you’re just not always able to create a winner with enough impact


So in th is case, you’ve got impact

Every 7 or 8 weeks


Slower? Energy wi l l run out! And your continous optimization program will die


What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!


What to test? Business cases!

Service charge, do it or not? Include in

price or not?!


What to test? Big design changes


ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Final ly you can test detai ls…

Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!


You should grow to

Customer Buying Reasons


Reasons behind the customer journey


A cont inuous opt imizat ion team

Analyst Psychologyst

Designer Developer

Team Lead


They keep on asking:

Why are these users here?


And they want to know

How they can

help their users’ brains

to fullfil their needs?


Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment


A lack of resources


Test ing takes resources

Use it to LEARN


FA C T & A C T

&

Tell Conclude Analyze

Test Create Analyze Find


A n d t h e n s t a r t : FA C T & A C T

&


Test Create Analyze Find Find

Your optimization team needs a good analyst / researcher!


Many sources out there to Find


FA C T & A C T

&


Test Create Analyze Find Analyze

Really a good analyst / researcher!!


Use your analyt ics data


Create heatmaps


Create screen recordings


To find out: The cr i t ical spots


FA C T & A C T

&


Test Create Analyze Find Create

And yes, your team needs an economic / consumer Psychologyst and a UX designer!


Source: Psychology


Two systems

104


Two systems that intervene

105


Use that persuasion knowledge

Wheel-of-Persuasion.com

Resource & courses


Tests that can be done


FA C T & A C T

&


Test Create Analyze Find Test

And your team needs a good front-end developer!


Get your A/B-so lut ion in p lace


But you need Front-end resources!


FA C T & A C T

&



Analyze

Your analyst will analyze the results!


Look at detai ls


FA C T & A C T

&

Tell Combine Analyze


Tell Conclude

Your psychologyst should conclude and your conversion lead should tell!


Store your learnings


Push your behavioral ins ights


Everyone should use your knowledge!

Data Insights Action


+ Optimization! + Automation!

Back to the ROAR model

Time span!

Con

vers

ions

per

mon

th!

Risk! Re-think!




ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Now you’re test ing

On your way to 4 tests a week

Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!


Test capacity

maximum number of

experiments per year that your site or app

can handle


Regret

amount of money you are losing per year by

not using your full test capacity


ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!





Automation


You tested which locat ions are cr i t ical


You know which dia logues are cr i t ical


Automation


Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!


+ Optimization! + Automation!

ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!

Risk! Re-think!




ROAR – You don’t stop opt imiz ing

Time span!

Con

vers

ions

per

mon

th!





ROAR – Re-think

Time span!

Con

vers

ions

per

mon

th!





A/B-test ing is the ul t imate resource


More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon


CRO br ings user knowledge


Store your learnings


You wi l l make big impact on Discover


Now, that’s

HOT


Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!


A/B-test ing

Minimize regret


Test as much as you can

Time span!

Con

vers

ions

per

mon

th!




15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!


Like Amazon, Like Zalando G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Enjoy the high energy fun!


Exper iment ing is a way of working

It’s company DNA


Run more exper iments

i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# i L i v e 2 0 1 5 - @ To n W

Run more experiments

Run more experiments - Ilive 2015 Latvia, Riga

Data & Analytics

Transcript of Run more experiments - Ilive 2015 Latvia, Riga