SPLIT TESTINGUsing Real Data

Follow @ShameOnUX

Split Testing 101

KISS If you can’t measure it, you can’t test it For clean data you can only test ONE

element at a time Attribution is everything

Follow @ShameOnUX

Types of Conversions

Follow @ShameOnUX

• Purchase• Contact• Phone Call• Sign-up• Subscription (on-going purchase)• Content• Share• Time on site / page• Pages per visit• Download

A Quick Note…

Follow @ShameOnUX

When testing for specific conversions, focus on specific signals that drive the specific conversion.

• Purchase vs. Sign up• Submit content vs. Get the click• Social Share vs. Link-bait

Split Test 1 Goal: Increase Content Acquisition

Follow @ShameOnUX

Conversion: Form Submittal

Type: user Generated Review

Motivation: Share frustration, publicize experience, receive sentiment, receive compensation

Follow @ShameOnUX

Split Test 1 Control

Follow @ShameOnUX

Split Test 1

Variation 1: Remove Unnecessary and potentially distracting page elements

Element: Main Navigation

Follow @ShameOnUX

Split Test 1, Variation 1 Test ResultsConversion Increase: 63%

Follow @ShameOnUX

  Visitors Conversions Conversion Rate Standard Error

Control 11643 961 8.25% 0.26%  

Variation 12564 1689 13.44%   0.30%  

95% Conversion Rate Limits

  From To  

Significant At     7.83% 8.67%  

95% confidence: YES   12.94% 13.95%  

99% confidence: YES

    99% Conversion Rate Limits

Z-score -13.069389     From To  

      7.75% 8.75%  

P-value 2.5E-39     12.85% 14.04%  

Follow @ShameOnUX

Split Test 1

Variation 2: Add elements of social proof

Element: Most recent reviews from other users

Follow @ShameOnUX

Variation 2 Test ResultsConversion Increase: 105%

Follow @ShameOnUX

  Visitors Conversions Conversion Rate Standard Error

Control 15263 2106 13.80% 0.28%  

Variation 14007 3957 28.25%   0.38%  

95% Conversion Rate Limits

  From To  

Significant At     13.34% 14.26%  

95% confidence: YES   27.62% 28.88%  

99% confidence: YES

    99% Conversion Rate Limits

Z-score -30.628883     From To  

      13.25% 14.35%  

P-value 3E-206     27.50% 29.00%  

Split Test 2 Goal: User Sign-Up

Follow @ShameOnUX

Conversion: Form Submittal

Type: Personal Information Submission

Motivation: Interested in starting free trial

Follow @ShameOnUX

Split Test 2 Control

Follow @ShameOnUX

Split Test 2

Variation 1: Change Call-to-Action text

Element: Button

Follow @ShameOnUX

Split Test 2, Variation 1 Test ResultsConversion Increase: 25%

Follow @ShameOnUX

  Visitors Conversions Conversion Rate Standard Error

Control 1254 112 8.93% 0.81%  

Variation 1304 145 11.12%   0.87%  

95% Conversion Rate Limits

  From To  

Significant At     7.60% 10.26%  

95% confidence: YES   9.68% 12.56%  

99% confidence: NO

    99% Conversion Rate Limits

Z-score -1.8450767     From To  

      7.35% 10.51%  

P-value 0.03251     9.41% 12.83%  

Follow @ShameOnUX

Split Test 2

Variation 2: Prominent Display of Price

Element: Sub-title / Text

Follow @ShameOnUX

Split Test 2, Variation 1 Test ResultsConversion Increase: 42%

Follow @ShameOnUX

  Visitors Conversions Conversion Rate Standard Error

Control 1366 151 11.05% 0.85%  

Variation 1284 201 15.65%   1.01%  

95% Conversion Rate Limits

  From To  

Significant At     9.65% 12.45%  

95% confidence: YES   13.98% 17.33%  

99% confidence: YES

    99% Conversion Rate Limits

Z-score -3.4791822     From To  

      9.39% 12.72%  

P-value 0.00025     13.67% 17.64%  

Split Test 3 Goal: Increase click-through rate

Follow @ShameOnUX

Conversion: Interest-based

Type: Click

Motivation: Interested in learning more about offer

Follow @ShameOnUX

Split Test 2 Control

Follow @ShameOnUX

Split Test 3

Scenario: Data supports that visitors who first visit the product detail page prior to entering the checkout are 50% more likely to complete their purchase

Variation 1: Remove Buy Now Option

Element: Button

Follow @ShameOnUX

Split Test 3, Variation 1 Test ResultsConversion Decrease: 18%

Follow @ShameOnUX

  Visitors Conversions Conversion Rate Standard Error

Control 10112 543 5.37% 0.22%  

Variation 11213 511 4.56%   0.20%  

95% Conversion Rate Limits

  From To  

Significant At     5.00% 5.74%  

95% confidence: YES   4.23% 4.88%  

99% confidence: YES

    99% Conversion Rate Limits

Z-score 2.72335304     From To  

      4.93% 5.81%  

P-value 0.99677     4.17% 4.94%  

Split Testing Key Takeaways

Follow @ShameOnUX

• Don’t Guess. Test.• If at First you don’t succeed, test again. If

at first you do succeed, test again.• Remove emotion from testing• Tests will fail• Tests will succeed• Put yourself in the shoes of your users; feel

their pain, now design a test

QUESTIONS?

Follow @ShameOnUX

Fin!Follow @ShameOnUX

Visit ShameOnUX.com