A/B Testing the WordPress Way

A/B Testingthe WordPress Way

mitcho (Michael 芳貴 Erlewine)

WordCamp Phoenix, January 29, 2011

Introduction

Introduction

• Hi, I’m mitcho.

Introduction

• Hi, I’m mitcho.• Linguist, coder, teacher.

Introduction


• http://mitcho.com; @themitcho

http://mitcho.com

http://mitcho.com

http://twitter.com/mitchoyoshitaka


Introduction


• http://mitcho.com; @themitcho• Linguistics grad student at MIT

http://mitcho.com

http://mitcho.com



Introduction


• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla

http://mitcho.com

http://mitcho.com



Introduction


• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla• Yet Another Related Posts Plugin

http://mitcho.com

http://mitcho.com



Introduction


• http://mitcho.com; @themitcho• Linguistics grad student at MIT• Automattic, Mozilla• Yet Another Related Posts Plugin• Lightning talk later today!

http://mitcho.com

http://mitcho.com



Today

• A/B testing in a nutshell• Testing on the web?• Theory• Practice

• The WordPress Way

Testing on the web?

Testing on the web?

• Websites aren’t just for passive communication.

Testing on the web?


• Every website has a purpose.

Testing on the web?


• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

Testing on the web?


• Every website has a purpose.• Common actions: register, sign up, log in, pay, donate, read more...

• tinyurl.com/everywebsite

http://tinyurl.com/everywebsite

http://tinyurl.com/everywebsite

Example:

Example:• highrisehq.com

http://highrisehq.com


Example:• highrisehq.com• Action: sign up



Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.



Example:• highrisehq.com• Action: sign up• Tried many different button texts: “Free Trial”, “Sign-up for Free Trial”, etc.

• “See Plans and Pricing” increased sign-ups by 200%

thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/



http://thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

http://thinkvitamin.com/business/how-to-increase-sign-ups-by-200-percent/

Example:

Example:

+30% +27%

+7%+15%

37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

http://37signals.com/svn/posts/1525-writing-decisions-headline-tests-on-the-highrise-signup-page

Others:

Others:

• Huffington Post titles

Others:

• Huffington Post titles

• Most everything at Amazon, Google

Why test?

Why test?

• We know that {design, content, layout, titles, copy, interface, load time} matters.

Why test?


• How do you know you’re making the right decision? What if you disagree?

Why test?


• How do you know you’re making the right decision? What if you disagree?

• Get the “hard data.”

The theory of A/B testing


• So we can do a randomized experiment.


• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.


• So we can do a randomized experiment.• Randomly assign each user to a different treatment and keep track of a metric.

• How can we interpret the data?

Is this coin a fair coin?

TTTTHHHHTHTTHHTTHTHHHHHTTHHHHTHTTTTHHHHTTHHTTHTHTHHHHHTTTHHHHHHTHTHHHHTHHTTTTHHHHHTHTHHTTTTTHTHHHTHTTTHHTTHHHHHTTHTHHHHHTTTHHHTHTHHHHHHHTTTHTTHHHHTHHHTTTHHHHHHHHHHTTHTHHHHHTTHHHTHTHHHHTHHHHHTHHTHHHHHHHTHTHTHHHHHHHHHTHHHTHTHHTHHTHHHHHTHHHHTHTHTHHHHTTTHHHTHHHHHHHHHHTTTHHHHTTHTTHHHHTHTHHHHHTHTHHTHHHHTHTTTHTTTTHHTHHHTTHTHTTHHTHTHHHHHTHHHHHHHHTHTTHTTHTTHHHHTTHTHTHHHHTTHTHTHTHTHTTHHTHTTHTTTHHHHHHHHHHTHTHTHHTHTHHTHTHHHHHTHHHHHHHTHTHTTHTHHTHHHHTHHHTTTTHTTHHTHHHHTTHHHHTHTHTTHTHTHHHHHTHTTTHTHHTHTHHTTHTHTH


N = 500, P(H) = 0.632


N = 250, P(H) = 0.568


N = 10, P(H) = 0.5

Not a fair coin:

P(H) = 0.6P(T) = 0.4

HHHHTHHHTTHTHHHTHTTHTHTTTTTHHTTTTHHTTHHHHHHHTHTHHHHHTTHHHHHHHTTHHHHTTHTTHHTHTHHTTHHTHHHTHHTTHHTTTHHHTHTHHTHHHTTTHTHHHHHHHHHHHHTTHTHHTTHHHTTHHTTHHHHTTHHTHHHHHHHHTTTHHTTTTHHHTHHHTHHTTTHHHTTHTHTHTHHTHHTHHHHHTTTHHTHHTHHTTHHHTHHTTHTTTTHTHTTHHTHHHTHTHTHTTTTTHHTHTTHHTTHHTHHTTHHTHHHTHTHHHHHHTTHTHTTHHHHHHHTHHHHTHTTHTHHHHHHHHTHTHTTTHHHHTHTHTTHHHHTHTHTHTTHTHHHHHHHHTTHTHTHHTTTTHHHHHHTTTTTHTHHTHTHTHTHHHTHHHTHHHHHTTTTTHTHHHTHHHHTTTHTHHTTTTHTTHTTTTHHTTTHTHTTHHTTHTHHTHHTTHHTHTHTHHHHHHHHTTTHHHTTTHTHTHHTTTHTHTTHHHHTHHHTHHHTHTHHHTHHHTHHTTHHHTHHTHHHHTHHTHHTHTTTTTHHHHTTHHHTHTTHHHHTHHTHHTHTHTHHTHHTTTHTTHHHTTHHTTHTHTTHTHTHHTHHHTTTHHTHHTHTHHHTHTTHTHHHHTHHTTTHTHHHHTHHTHHTTHHTHTTHTTHHTHTHTHHHHHTHHHHTHHHHHHTHTHHHHHHHHTHTTTTHTTHHHTHHHTHHTTHTTHTTHHHTHHHHHTHHHHTHHTHHTHHHHTTHHHHHHTTTHHHHHHHHTTHHTTHTTHHTTHTHHHTHTTTHHHTHHHHTTTHTHHHHHHHHHTTHHHTHHTHHHTHHTHTHHHTHHHTHHTTHTHHTTTTHHTHHHHTTTHHTHHTHHTTTTHHTTHTTHHTHHHHHHTHHTTHHHTHHTHTHTHTHHHHTTHTHHTTHHHTHTHHHHHTHTHHHHHTTTHHHTHHTHHHTHTHTHHHHHHTHHHTHTHHHHHTHTHHHHTHHHHHTHHHHTHTHHTTHHTHHHHTHHTTHHHHTHHTTHTHTTTHHHHHHHHHTHHHHTHHHTTHHHTTTHHHHTTHTHHHHTHHHTTHTTTHTTTTHTHTHTTHTHHTHTHHTTHHHHTTTHTHTHHHHTTHHTHTTTHTHTHHHTTHHHHHHHHHTTTHHTTHHHTHTHTTHHHTTHHHHHHHHHHTTHTHHHTTTHHTHHHTHTHHHTHTHTTHTHHTHTHHTHTTTTTHHTTTTHHTHHHHHHHTTHTTHHHHHTTTHTHHTTHTHHHHTHHTHTHTTHTHTTHHHTTTHTTTHHTTHHHHTTTHTHTTTTHHHTHTHHHTHHTHHHHHTHHHHHHHHTTTHHHTTTHHHTHHTTHTHHTHHHHHHTHTHTHTHHHTTHTHTHTHHTTTHHHTHHTHHTHHTTHTHTTHHHTHHHTHHTHTHTHHHTHTTTTTTHTTHHHHHTTTTTHTTHTHHTHTHTTHHHTHTHHTTTTTTHHHTTTHTTHHHHTHHHTHTHTHHHHHHTHHTTTHHHHHTHHHHTTTHTHHTHHHTHHTHTHHTHHTHHHHHHHTHHHHHTTTTTHHHHHTTHHHHHHHHTHHHHHHHTTTTTHHTHHHHTHHTHHTHTHHHTHHTHTHTHHTTHHTTHTHHHHTTTHTHTTHTHTHHTHHHTHHTHHHTHHHHHHHTHTHHHHHTTHTHHHHTHHTTTHHTTHHHHTTTTHHHTHHHTHTHTTTTHTHHHHTHHHHHTHHHHHHHHTHHTHHHHTHHTHTHTTTHTHHTTHHTHTTHHHHTHHHHHHHHTHHHHTHHHTTHHHTHHHHTTTTHHHHTHHHHHHHTTTHHTHHTTHHTTTTHHTHHHHHHTHTHTTHHTHHHHTTTHTHHTHTTHHHTHTHTTTTTHHTHHHHTHHHHHHHHHHHHHHHHHTTHHHHTHHHTTTTHTHHHHHHHHTHTHHTTTTHTTHTHHTHHHHHHHTHTHTHHHTTTTHHTHHHHHHTHTTHTHHTHTHHHHHHHTTHHHHTHTHTHTTTHTTHHHTHHHHHHTTHHTHTTHTTTHHHHHTTTHHTHHHTTHHHHHTHHHHHTTHTHTHHTHHHHTTHHTTTHTHHHTHHHHTTTTHTHHHHHHHTHTHHHHHHHHTTHTHHHHTHHHHTHHHHHTTHTTHHHHTTHTTHHHHTHHTHHHTTTHTHTTHTTTTHTTHTHHHHTTTHTTHHTTTTTHTHTTHTHTHTTHHTHTTHHHTHTTHTHHHHTHHTHHHHHHHHTTTHTHHTHTHTHHHHTHTTTHHHTHHHHHTHTTTHTTHHTHTTHHHHHTTTTTHHHHHTTTTHTHHHHTTHHHHHHHHHTHTHHHTTTTHTHTHTHHTHHTTTHHTHHTHHTHHTTHHHHHTHHHHTHTHHHHHTHTTHTHHHHTHHHTHHTHHTHHHTHHTHTTHHHHHTTTHHTHTHTHTHHTTHTHTTHHTTTHTHTHTHTTHHTHTHTHHHTHHHHHHHTTHHHHTHHTTHTHHTHTHTHHHHHHHHTTHHTHHHHTHHHTTHHHHHTTHTHHHHHHHHHTHTHHTTTHHTHHHHHTHTHHTHTHHHTTHHHHTTTHHTTHHTTTHTTTTTTTTHHHHTTHTHHHHHHTTHHHTHHHHTHTTHHHHHHTHHTTHHTTTHHHHTHHTTHTHTHHHHTTHTTTHHHTHTHHTHHHTTHTHHHTTHHTHHHHTTHHTHTTTTHHTTHTTTTTHHHHHHHHHHHTTTTHTTTHTHHTHHHHHHHHHHHTHHHTTHHHHTHTTHHHTTHHHTHHHTTTHTHHTHHHTTTTHTTHHHHTTHTTTHTHHHHHTHTHTHHTTTTTHTTHHTTHHTTTHHTHTHHTTTHHHHHHHHTTTTTTTHTHHHHTTTHHTHHHHTHHHTHHHTHHTHHHHTHHHHHHTHTTHHHHTTHHTHTTHHHTHTHTTHHTHHHTTTHHTTTHHHTHHHHTHHHTHTHTHHTHTTTTTTHTHTTHHTHHHHHHHHHHHHHTTHHTTHTHTTTTTTHHHHHTTHHHHHTTHTHHHHHTTTHHTHTHHTHHHTHTHHTHTHTHHHTHHHHHHHHHHHTHHHHHHHTTHHHHTTHHHHTHHHTHHHTTHTTTHHHHHHHHHTTTHHHTTHTHHTHTHTTHTTTHHHTTHHHHHHHHHHTTHHHTTHTTTHTHHHTTTHHTHHTTTHHHHTTHHTHTTTTHTTHHHHHHTHTTHHHHHTHHTTHHHTTHHHHHTTTHTTHHTTHTTHTTHHHHHHTTTHHHTTTHTHHTHHHHHTHTTHHHHHHHHTTHHTHHTTHHHHTTHHHTHHHHHHHTTTHHTHHHHTHHHHHHHHHTHHTTHHTHTTTHHHTTHTTTHHHHHHHTTTHHHHTHHHHHHHTTTTTHHHTTHTTTHHHHTHTTTHTHHHTHTHHHTTTTHHTTHHTTTHHTHHTHHHTHTHTHHHHTHHTHHHTHHHTTTHHHTHHTHTHHTTHHTTHTTHHHTTTHHHHHHHHHHHTTHHTTHHHHHHHHHHTHTTHHHHHHTTHHTTHTTHTTHHTTTHTTHTHTHHHHTHHHTHHTHHTTHTTHTHHTTTHTHHTHTHHHHHTHTTHHTTHTHHHHHHTHTHHTHTHHTTHHHHHTHTHHTHTHHTTHTTTHHTHHHTTTTTTTTHHHHTHHTHTHHHHTHTHHHHTHHTHTTTHHTTHHTTTHHHHTTHHHHHHTHTTHTHTTHHTHHHTTHHTTTTTHHHHHTTTTTHHHTTHHTHTTHTHTTHHTTHHHTHHHHHHHTTHTTHHHHHHTTHHHHTHTHHTTHHHTHHHHTHHTHTTHHHTTTHTHTHTTHHHHHTHHHHHTTHTHHTTTHTTHHHHHHHTHTHHHHHTTTTTTHHHTTTTTHHTTTHHHHTHHTTHTHTHHTTHHHHTHHHTHHHHHHTHHHHHHHTTTHHHHHTTHHTHHTTTTTHHTHTTTTHHHTHTTHHTTTHTHHHTHHTHTHHTTHHTHTTTHHHTHHHHHHHHHHHHTHTTTHHTTHHHTHTTTHHTHHTTHHHHHHTHTTHTHHHTHTTTTHHHTHTHHHHHHHHTHHHTHHHHTTHHHHTTTTHTTHTHTHHHHHTHHHHHHTHTTHHTHTHHHHTHHTHHTHHHTHTHTHTTHTHHHHTHHTHHHTHHTHHTTHTHHTHHHHHTHHTTHHTTHTTTHTTHHHTTTTHTHHHHHHHHTHTTHHHTHTHTHHTHTTTHHHHHHTHTTTHHHTHHHHHTTHTHHHHHHTTTHTTHTHTHHTHTTTHHTTHTHHTTHTTHHTHHTHHHHTHTHHHTHHHHHTTHTTTHHTHHHHTTHHHTTTHHHTTTTHTHTHTHHTTTTHTHTHHHHTHHTHHTHHTTHHTTHHHTHHTHHHTTTTHHTHHHHTTHHHHHTHTTHHHHTTHHTHHTHHTHHHHHHTHHHTTHHHHHTTHTTHHTHHHTTTTTHTTTHTTTHTTHTHTTTTHHTHHHHTHHHHHTTHTTHTHTHTTHHHHTTTHHHHHHHHHTHTHHHTTHHTHTHTTHHHTHTTHHHTTHHTTHHHHTHTHHHHHHTHTHTHHTTHTTTHHTHTHTHHHTTTTTHHHHHTHTHTTTTHTTHHHTHTTTTHTTHTTHHTHTTHHHHHTTTHHHHHHHHHTTHTTHTTHHTHTTHTTTTTTHTTTHHTHTTHTTHHHHHHTHHHHHHHHHTHHTHHTHHHTHHHHHTHHTHTHTHHTHHHTHTHHTHTHHTTHTHTHTTHTHTTTHHTHHHTHTHHHHHTHHHTHHHTHTTHHHHHHTHHHHTHHHTTHHTHHTTHTTHTHTTHHTTHHHTTHHHHHHHTHTHHHTHHTTHHHHHTHTHHHHTHHTTHTTTHHTHHHHHTTTHHHHHHHHTTHTTTTTTHHHTHTTHHHTTTHTTHHHHHHHTTHTHTHHTHHTHTHHTTTHHHTTHHHHTHHHHHHHHHTHHHTTTHTTHTHHTHHHTHTHHHTTHHHHHHHHHHHTHTHTT

HHTHHTHTHHTTTTTHTHHHHHHHTTHHHHHTTTTTTTHHTTHTTHHTHHHTHHHHHHTHHTTTHHTHTHHHTHHHHHHHHHTTTTHTHHTTHHHHTHTHHHHHHHHTTHHHHTHTT

But maybe we should run N = 5000 ?

Statistics to the rescue!


• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.


• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.• The question: could this coin (variant) actually be just as “effective” as a fair coin (control), and any variance we see is due to chance?


• After 500 trials, it’s very rare even for a known fair coin to give you exactly 50% heads.• The question: could this coin (variant) actually be just as “effective” as a fair coin (control), and any variance we see is due to chance?• We can calculate this probability: p.


• Normally, if p < 0.05, we can be confident. If p < 0.01, we can be very confident.



• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.



• We want to be 95% sure that the different results we see are due to a real difference in “effectiveness”.

• This 95% confidence interval is standard.

Can we say this is an unfair coin?


N = 10, P(H) = 0.5, p = 1 unclear!


N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

unclear!

confident!


N = 10, P(H) = 0.5, p = 1

N = 250, P(H) = 0.568, p ≈ 0.03

N = 500, P(H) = 0.632, p < 0.01

unclear!

confident!

very confident!

Best practices

CC-BY-NC flickr.com/photos/kylemacdonald/3199283481/

http://www.flickr.com/photos/kylemacdonald/3199283481/


Best practices

• Decide on a clear, measurable, long-term metric ahead of time.




Best practices

• Decide on a clear, measurable, long-term metric ahead of time.• Clear metric = actionable results.




Best practices


• Test bold changes.




Best practices


• Test bold changes.• Keep the potential costs in mind.




Best practices: how you test


• Assignment of visitors to variations must be truly random.



• Each visitor must have a consistent experience across visits.




• The site must still work, both during and after the experiment!




• The site must still work, both during and after the experiment!

• Beware of interactions.

A/B testing on the web


• Get clear, actionable results with hard data.


• Get clear, actionable results with hard data.• Take the guesswork out of design.


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!


• Get clear, actionable results with hard data.• Take the guesswork out of design.• Increase your bottom line with little cost.• You can’t afford not to!

• Unfortunately, if it’s not drop dead easy and convenient, it won’t happen...

Today

• A/B testing in a nutshell• The WordPress Way

ShrimpTest

CC-BY flickr.com/photos/gastev/2630047705/ CC-BY-SA flickr.com/photos/stuart_spivack/315481947/

A B

http://www.flickr.com/photos/gastev/2630047705/

http://www.flickr.com/photos/gastev/2630047705/

http://www.flickr.com/photos/stuart_spivack/315481947/

http://www.flickr.com/photos/stuart_spivack/315481947/

ShrimpTestA/B Testing for WordPress

ShrimpTest

• A/B testing right in WordPress:

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use

ShrimpTest

• A/B testing right in WordPress:• More convenient and easier to use• More robust and extensible

ShrimpTest


• A project of Automattic, written by mitcho

ShrimpTest


• A project of Automattic, written by mitcho• Released as a free plugin.

ShrimpTest

Custom variant selection and metrics


$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}


$variant = shrimptest_get_variant( $my_experiment_id );// $variant is going to be an integer.switch ( $variant ) { case true: // if $variant > 0 echo "This is variant #{$variant}."; break; default: // if $variant is 0, or if ShrimpTest is down. echo "This is the control.";}

// the user did something worth xshrimptest_update_metric( $my_experiment_id, $x );

Features

✓ Written as a WordPress plugin

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection

Features

✓ Written as a WordPress plugin✓ Server-side (PHP) selection✓ Tools for easy setup of experiments: [ab] shortcode, variant viewer

Features


✓ API for custom metrics, variant types

Features


✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

Features


✓ API for custom metrics, variant types✓ “eat my own dogfood” approach

✓ Works with caching plugins

Features

Start testing now!

• Download at http://shrimptest.com

Start testing now!

http://shrimptest.com


• Download at http://shrimptest.com• Stay up to date: @ShrimpTest

Start testing now!



Thank you! Questions?mitcho (Michael 芳貴 Erlewine), Automatticmitcho.com, @themitchoshrimptest.com, @ShrimpTest

A/B Testingthe WordPress Way

http://mitcho.com

http://mitcho.com

http://shrimptest.wordpress.com

http://shrimptest.wordpress.com

A/B Testing the WordPress Way

Technology

Transcript of A/B Testing the WordPress Way