Run more experiments - Ilive 2015 Latvia, Riga

i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# i L i v e 2 0 1 5 - @ To n W

Run more experiments

Ton Wesseling | @ TonW Run more experiments #ilive2015

Thank you for having me

On this stage

Vis i t ing Riga

On november 11th

http://test ing.agency

Meeting you al l

Conversion people

Always fun

When we start cooking

High energy & fun

We are

Evidence Based

I t has been a long day though

What about your energy?

Value for your t ime and money

Reboost your energy

Please stand up

Raise your hands

High energy conversion fun

Calm down and focus

Why this song?

Because of Katy Perry?

No, i t ’s the name of the song!

ROAR model

20 years ago

I learned about him

Qual i ty c i rc le – the start of LEAN

So I looked at data

Made some changes

And measured i f i t was successful

Why can this lead to fa i lures?

Average conversion rate looks stable

But in fact

There is a big weekly change!

I only did smal l improvements

So I made many wrong decis ions

26 out of 44 weeks: over 5% change!

Then I learned about th is story…

Bi l l Murray in Groundhog Day

Trying to get the gir l

He has a stable test ing environment

A/B-Testing

Methods used to improve conversion

Methods companies plan to use

A/B-test ing is

Everyone is pushing to test more

Like Amazon, Like Zalando G

More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon

Never sat isfied with conversion rates

Start test ing or test more

NO Growth problems wi l l d isappear…

A/B-test ing is not a solut ion

It’s a methodology

I t ’s part of a:

Continuous Optimization

Program

I t ’s a way of working

It’s company DNA

You need to know:

How What

Growth curve – how i t was teached

ROAR model

Optimizat ion phases - ROAR

Time span!

Risk! + Optimization!

Time span!

Risk! + Optimization! + Automation!

Time span!

Risk! + Optimization! + Automation! Re-think!

Time span!

1.000 conversions!per month!

Time span!

10.000 conversions per month!

How much impact do you need?

http://ondi.me/size!

Time span!

15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

Time span!

Phase 1: br idge the gap – Risk

Get out of the office

Yes, I ’ve got those 1000+ conversions

Time span!

With 1000+ conversions a month

Your tes t capac i ty i s max .

20+ tests a year

wi th a s ta t i s t i ca l power o f 80% on pred ic ted up l i f t s o f 15%

Tests with measurable impact

On average

1 out of 3

Simple truth: you’re just not always able to create a winner with enough impact

So in th is case, you’ve got impact

Every 7 or 8 weeks

Slower? Energy wi l l run out! And your continous optimization program will die

What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!

What to test? Business cases!

Service charge, do it or not? Include in

price or not?!

What to test? Big design changes

ROAR – Opt imizat ion – moving up

Time span!

Final ly you can test detai ls…

Is Multi Variate Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!

You should grow to

Customer Buying Reasons

Reasons behind the customer journey

A cont inuous opt imizat ion team

Analyst Psychologyst

Designer Developer

Team Lead

They keep on asking:

Why are these users here?

And they want to know

How they can

help their users’ brains

to fullfil their needs?

Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment

A lack of resources

Test ing takes resources

Use it to LEARN

FA C T & A C T

Tell Conclude Analyze

Test Create Analyze Find

A n d t h e n s t a r t : FA C T & A C T

Test Create Analyze Find Find

Your optimization team needs a good analyst / researcher!

Many sources out there to Find

FA C T & A C T

Test Create Analyze Find Analyze

Really a good analyst / researcher!!

Use your analyt ics data

Create heatmaps

Create screen recordings

To find out: The cr i t ical spots

FA C T & A C T

Test Create Analyze Find Create

And yes, your team needs an economic / consumer Psychologyst and a UX designer!

Source: Psychology

Two systems

Two systems that intervene

Use that persuasion knowledge

Wheel-of-Persuasion.com

Resource & courses

Tests that can be done

FA C T & A C T

Test Create Analyze Find Test

And your team needs a good front-end developer!

Get your A/B-so lut ion in p lace

But you need Front-end resources!

FA C T & A C T

Analyze

Your analyst will analyze the results!

Look at detai ls

FA C T & A C T

Tell Combine Analyze

Tell Conclude

Your psychologyst should conclude and your conversion lead should tell!

Store your learnings

Push your behavioral ins ights

Everyone should use your knowledge!

Data Insights Action

+ Optimization! + Automation!

Back to the ROAR model

Time span!

Risk! Re-think!

ROAR – Opt imizat ion – moving up

Time span!

Now you’re test ing

On your way to 4 tests a week

Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!

Test capacity

maximum number of

experiments per year that your site or app

can handle

Regret

amount of money you are losing per year by

not using your full test capacity

ROAR – Automation

Time span!

Automation

You tested which locat ions are cr i t ical

You know which dia logues are cr i t ical

Automation

Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!

+ Optimization! + Automation!

ROAR – Automation

Time span!

Risk! Re-think!

ROAR – You don’t stop opt imiz ing

Time span!

ROAR – Re-think

Time span!

A/B-test ing is the ul t imate resource

More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon

CRO br ings user knowledge

Store your learnings

You wi l l make big impact on Discover

Now, that’s

Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!

A/B-test ing

Minimize regret

Test as much as you can

Time span!

15% impact needed!

5% impact!needed!

Impact needed: all conversions with an average conversion rate of 2%, a test length of 3 weeks and a power of 80%!

Like Amazon, Like Zalando G

Enjoy the high energy fun!

Exper iment ing is a way of working

It’s company DNA

Run more exper iments

i L i v e L a t v i a , R i g a , N o v e m b e r 1 2 t h 2 0 1 5 To n W e s s e l i n g – C E O – Te s t i n g . A g e n c y

# i L i v e 2 0 1 5 - @ To n W

Run more experiments

Run more experiments - Ilive 2015 Latvia, Riga

Data & Analytics

Transcript of Run more experiments - Ilive 2015 Latvia, Riga

Socioeconomic Development of Cities & Towns in Latvia (Riga, 2010)

1st TETRA Baltic States CONFERENCE Riga, Latvia 20th October 2005 1 1st TETRA Baltic States CONFERENCE Hotel Riga, Riga, Latvia 20th October 2005 TETRA.

Ice Arena Brīvības gatve 333, Riga, Latvia. Ir īpašumi visā Latvijā! Brīvības gatve 333, Riga, Latvia Ice Arena Ir īpašumi visā Latvijā! Description of.

14 15 April, 2016 Riga, Latvia - uni-svishtov.bg

Riga, Latvia - GARMTECHkristalice.lv/wp...VOLVO-OPEN-CUP-40th-Riga-Latvia.... · Jurmalas gatve 78 D, Riga, Latvia. (an indoor ice-rink with ice surface of 28 x 58 m, air-conditioned

ITERA Latvia LLC - balticsheart.combalticsheart.com/pdf/BalticsHeart_66.pdfcity of Riga, the president of ITERA Latvia Mr.J. Savickis was honored with the Riga City Coun-cil’s „Riga

E-learning for Regional Development: Cases in Latvia Atis Kapenieks Riga Technical University, Latvia Baltic IT&T 2006, Riga, Latvia, April 5-7, 2006.

Higher Education in Latvia Indrikis Muiznieks, University of Latvia Riga, March 24, 2006.

11 13 June 2015, University of Latvia. Riga, Latvia International ... › fileadmin › user_upload › lu_portal › projekti › eibsr… · Riga, Latvia International Scientific

2009 NATIONAL REPORT (2008 data) Latvia...Riga, November 2012 Centre for Disease Prevention and Control of Latvia* 22 Duntes Street, Riga LV-1005, Latvia Tel.: +371 67043700 Fax.:

Latvian Biomedical Research and Study Centre (BMC), Riga, Latvia.

Riga, Latvia - European Space Agencyemits.sso.esa.int/emits/owa/loadfiles.showfile?p_file=F1467... · Riga, Latvia Founded in 1962 RD ALFA Microelectronics was a pioneer in electronic

RIGA SHOPPING MALL RIGA, LATVIA FEASIBILITY STUDY Preparation and Analysis: Bank Financing of Construction Project.

1st TETRA Baltic States CONFERENCE Hotel Riga, Riga, Latvia 20th October 2005 TETRA

Rural Development Perspectives in Latvia (Riga, 2012)

Proexpo Plant, Riga Latvia

Women’s Political Participation. Latvia University of Latvia, Riga Feministica Lettica Ausma Cimdina.

Art education in Latvia. Janis Rozentals Riga Art Highschool Address: 2a Hāmaņa Street, Riga LV-1007, Latvia E-mail: info@jrrmv.lv Phone: (371)7601783;

23 - 25 April 2019 Riga, Latvia · 2019-04-29 · PROGRAMME OUTLINE Ziedonis Hall, National Library of Latvia Mūkusalas iela 3, Riga, LV-1423, Latvia 29th ECBC Plenary Meeting 24

WHY RIGA - cache.radissonhotels.com · 55 Elizabetes Street, Riga, LV – 1010, Latvia +371 6777 2345 / latvia.events@radissonblu.com radissonblu.com/en/latvijahotel-riga. Title: