[#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of [#500Distro] Measuring for Impact: Knowing When, What & How to A/B Test

A/B Testing

Measuring For Impact: Knowing What, and How to A/B [email protected]_greenfieldCEO/Co-Founder, [email protected]_greenfield


You know you should A/B [email protected]_greenfield

You also know you should exercise more eat less sugar spend less on coffee wear sunscreen etc., [email protected]_greenfield

(Dont worry, Im not going to say anything else about sugar or sunscreen.)@mike_greenfield

So, how do you create a culture in which people will constructively A/B test?

Do six [email protected]_greenfield

1. Embrace I dont knowWe have 2+ ideas.

I dont know which one will be more [email protected]_greenfield


2. Have Data, Choose MetricsTo test, you need:People using your product(Approximate) agreement on the metrics that matter


Not Many Users? Dont A/B test!Laserlike, has ~60 users and has never run an A/B testWe will run many, many tests when we have enough usersA test should have at least a few hundred instances (and a lot more if effect sizes are likely to be small)Test iff you can have business [email protected]_greenfield

Know What You Want to OptimizeIf its important, you should be running tests to improve itIf its not important, spend time on other thingsMost tests should be aimed at improving 1-2 specific [email protected]_greenfield

3. Have Clear Process, Tech for Testing



A/B Testing ProcessNew feature: if possible, roll out to a small test subset first (10s or 100s of thousands)Version change: always test things that could (cumulatively) have business impactEveryone on the product team should be running and resolving tests


A/B Testing TechUsing a third party testing service is akin to building your site on Wordpress: great at some scales/competency levelsNo matter how youre testing, a new test should be at most a few lines of codeIt should be easy to see how each side of a test compares across many [email protected]_greenfield

4. Understand the Math of What to Test


Process: Same vs. New TweakWhats the probability your tweak will have a positive effect?What kind of effect might that have, and how might that effect change the companys prospects?Will you be able to measure the change?Optimize on one variable, but look at others


Process: Same vs. Big ChangeWhats the probability that your change will have a negative impact?How big an impact might there be?Will you be able to measure the change?Holistic approach @mike_greenfield

A/B Test for Quality

Circle of Moms: test warning users when questions seemed short, low qualityResulting questions were graded for quality, without grader knowing test bucketEnd result: warning yielded ~5% fewer questions, but much higher [email protected]_greenfield

5. Understand the Math of Picking Winners


Resolving Too Soon vs. Resolving Too LateHow big is the potential audience for this test?Example 1: end of year most popular baby names email that will never be sent againExample 2: Facebook signup [email protected]_greenfield

Longitudinal Tests vs. Immediate TestsLongitudinal: change home page, email frequency, product framingNeed to examine effect over a long period

Immediate: change button color, email subjectLikely that long-term effects will be [email protected]_greenfield

Automatically Resolve Tests?Longitudinal tests should not be automatically resolvedExample: new home page design

Immediate tests can be automatically resolved when speed is important and there is one clear objective functionExample: Circle of Moms email subject [email protected]_greenfield

Choose robust statisticsBad: # of page viewsGood: % of users viewing at least [5, 25, 100] pagesPotentially bad: # of sales (when small)Potentially good: # of people getting through the second step of a sales [email protected]_greenfield

6. Celebrate A/B Testing Succe[email protected]_greenfield



[email protected]@[email protected]_greenfield