Mumbling Crap

about A/B

Testing

Let’s go back in time…

UX Camp Brighton 2012

Two things happened at UX Camp Brighton 2012:

• An A/B testing talk from Gareth Llewellyn

And

• I presented the following…

My First A/B Test…

A/B Testing is Great!

Back to Today…

Why A/B Testing Might

not be Great…

Some (Obvious) Things

that I Learnt from a

year of A/B Testing

The Positives

Easy to Setup

Clients Love Them

Quantitative Results

Small Changes - Big Difference

A/B Testing is Great!

The Negatives

It can Take a Long Time

Book a Demo

Need Lots of Traffic

Quick results often wrong

Often Inconclusive

Minor Changes = Minor Changes

Dehumanises Users

Use Other Methods

Conversion First Mind-set

Dark Patterns

darkpatterns.org

Problems with the Testing Tool

Limited HTML/CSS

Cloaking?

Google Guidelines

Can’t Polish a Turd

Ethicare

Misses the Bigger Picture

Why is Nobody Clicking that Button?

It’s the wrong colourIt’s not big enoughIt says “Buy Now” not “Add to Cart”It should be roundIt should be flat!

Your product is shitNobody know what your product doesYour product is overpricedYou competitor has a sale onYour website doesn’t look professionalYour user journey is confusingYou charge too much for postageYour website is slowYour product is out of seasonYour website doesn’t work on mobile

No Room for User Learning

Change Aversion

1. Warn users about major changes

2. Clearly communicate the nature and value of the change

3. Let users toggle between old and new versions

4. Provide transition instructions and support

5. Offer users a dedicated feedback channel

6. Tell users how you’re addressing key issues they’ve raised

In Summary• A/B testing should be part of a bigger user testing plan• Don’t rely on tests to be conclusive• Explain their limitations to clients/stakeholders• Most problems cannot be solved by A/B testing• Set aside a lot of time for tests to find a conclusion• Make sure tests are fair 50/50 split, on all browsers and devices• Test your tests thoroughly! • Treat ‘failed’ tests as a learning process

A/B testing can still be useful if run in the right way and for the right reasons. Be sure to include other types of testing, both qualitative and quantitative, to get the full picture.

Remember…

“Nobody has ever A/B tested

their way to innovation!”

Luke Hay

Research & UX Director

@hayluke

Questions?

@hayluke