1 5 Kyle Rush Architecting Great Experiments MozCon 2014

8/12/2019 1 5 Kyle Rush Architecting Great Experiments MozCon 2014

1/61


2/61

MEANS

WRITE IT DOWNOR

TWEET IT


3/61

EXPERIMENT CONCEPTS


4/61

SIGNIFICANCEThe risk of encountering a false positive.

95% = out of 100 a/a tests, 5 will inaccurately report a difference.


5/61

POWERThe risk of encountering a false negative.

80% = out of 100 a/b tests, 20 winners will not be reported.


6/61

ONE-TAIL VS. TWO TAILOne tail = is the variation better?

Two tail = is the variation better OR worse?


7/61

MDEMinimum detectable effect


8/61

SAMPLE SIZEHow many subjects are in your experiment.


9/61

Always use a sample size calculator to calculate sample sizebefore starting an a/b test.

bit.ly/VUBti8


10/61

SAMPLE SIZE CALCULATOR

bit.ly/SWR3YC


11/61

EXAMPLEAbsolute lowest MDE


12/61

CONVERSION RATE: 4%

MDE: 1%

POWER: 80%SIGNIFICANCE: 5%

TAILS: ONE

2,972,435Visitors per branch


13/61

EXAMPLEFocus on time

1 month = 170,000 unique visitors


14/61

CONVERSION RATE: 4%

MDE: 6%


TAILS: ONE

83,230Visitors per branch


15/61

EXAMPLESmall startup

1 month = 3,000 unique visitors


16/61

CONVERSION RATE: 4%

MDE: 45%


TAILS: ONE

1,567Visitors per branch


17/61

Sample size calculators tell you how many subjects,but not which subjects should be in your experiment.


18/61

SAMPLINGcan be really hard

week day vs. weekend traffic

campaign vs. organic traffic

returning vs. new visitors


19/61

WHERE SHOULD I TEST?


20/61

EXAMPLEE-commerce website


21/61


22/61

RUN EXPERIMENT FOR 1 MONTH (100K VISITORS)


23/61


CONVERSION RATE: 2.5%


TAILS: ONE

MDE: 10%


24/61

CHECKOUT PAGE


25/61

CHECKOUT PAGE

ASSUMPTIONS1. Lower traffic2. Relatively high conversion rate

Let's assume:

50% conversion rate

10,000 monthly unique visitors



26/61


CONVERSION RATE: 50%


TAILS: ONE

MDE: 5%


27/61

Start by focusing a/b tests on thelast step in your conversion funnel.


28/61

WHAT SHOULD I TEST?Depends on MDE and time.

LANDING PAGE


29/61

LANDING PAGE

OPTIMIZATION(MOZCON 2013)

bit.ly/1wkpgye

GOING BEYOND THE LOW


30/61

GOING BEYOND THE LOW

HANGING FRUIT(CONVERSION CONFERENCE 2014)

bit.ly/1kU4sZ0


31/61

BE FEARLESS


32/61

EXPERIMENTTerse vs verbose


33/61


34/61

RESULT+31% LEADS

99.9% CONFIDENCE


35/61

BEFORE YOU TEST


36/61

GOALSMeasure as many goals as possiblemicro: form field errors, time on pagemacro: purchase, revenue

Choose a primary goalDon't forget about down the funnel goals

Repeat purchase

Save payment information


37/61

EXAMPLESuccess vs. submit


38/61

13.72% CONVERSION RATE


39/61

17.15% CONVERSION RATE


40/61

25% DIFFERENCE


41/61

Measure as many goals as possible for every experiment.


42/61

QUALITY ASSURANCENo bugs in the variationNo bugs in the control

Tracking works correctly


43/61

ELIMINATING BIASDouble blind experiments


44/61

EXPERIMENT BRIEFHypothesisAudience descriptionGoals tracked

Stopping conditionsScreenshots

QA summary


45/61

EXPERIMENT BRIEF

TEMPLATEbit.ly/1nvNJjx


46/61

AFTER YOU TEST


47/61

STATISTICAL TIENot enough data to conclude that there is a difference.


48/61

The overwhelming majority of

experiments results are a statistical tie.


49/61

EXAMPLEStatistical tie


50/61


51/61

RESULT


52/61

RETESTING


53/61

FIRST TEST


54/61

+36% revenue

SECOND TEST


55/61

statistical tie

THIRD TEST


56/61

Statistical tie


57/61


58/61

Always, always record detailed

experiment results in an archive.

EXPERIMENT ARCHIVE


59/61

EXPERIMENT ARCHIVEexperiment date

audience/url

screenshots

hypothesisresults

link to experiment

link to result csv


60/61

EXPERIMENT ARCHIVE

TEMPLATEbit.ly/1q9tRWI


61/61

1 5 Kyle Rush Architecting Great Experiments MozCon 2014

Documents

Transcript of 1 5 Kyle Rush Architecting Great Experiments MozCon 2014