9. Applying scientific tools & techniques for improving customer satisfaction.
The Scientific Approach to Improving Conversion with A/B Testing
Transcript of The Scientific Approach to Improving Conversion with A/B Testing
![Page 1: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/1.jpg)
The Power of A/B Split TestingPresentation By: Alex Foley
August 4, 2016
![Page 2: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/2.jpg)
My pain point: “I’m not sure how to best open up my
presentation at #DOYOLive.”
![Page 3: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/3.jpg)
HYPOTHESIS 1: ENTHUSIASTIC, CONFIDENT; WWE TYPE
My hypothesis: The crowd may need to be energized!
Pro:Crowd attention and energy
Con:Potentially make a fool of myself
![Page 4: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/4.jpg)
FUN FACT…
![Page 5: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/5.jpg)
HYPOTHESIS 2: RIDE THE POP CULTURE WAVEMy hypothesis: A Pokemon Go reference can’t fail, right?
Pro:Widely known and top of mind
Con:Some folks may already be sick of the Poke-mania
![Page 6: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/6.jpg)
The Results?
![Page 7: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/7.jpg)
VOTE ON MY NEXT TEST
• Justin Bieber
• Drill Instructor
• Trump
![Page 8: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/8.jpg)
![Page 9: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/9.jpg)
WHAT WE WILL LEARNWhat is it?
Why it’s important?
Best Practices
The New Wave of Website Development
Resources
![Page 10: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/10.jpg)
How Marketing “Used to” Be Done
![Page 11: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/11.jpg)
HOW MARKETING “USED TO” BE DONE
Source: isha.sadguru.org
![Page 12: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/12.jpg)
87% of Marketers Consider Data Their Most Underutilized Asset.
-Teradata 2015 Global Data-Driven Marketing Survey
![Page 13: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/13.jpg)
What is A/B Split Testing?
![Page 14: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/14.jpg)
A/B Split Testing: testing a hypothesis by measuring the performance of one version (A) against another (B) in an equal sample size.
![Page 15: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/15.jpg)
In reality – it is the scientific method
![Page 16: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/16.jpg)
Why is this important for us as marketers?
![Page 17: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/17.jpg)
It keeps you thinking, it keeps you sharp
It is the formula for continuous improvement
You don’t have to be right the first time
It can give you an edge on the competition
![Page 18: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/18.jpg)
It directly impacts the bottom line
74% of companies who take a structured approach to conversion (including A/B testing) have improved sales
Source: Econsultancy and RedEye 2014
![Page 19: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/19.jpg)
Modern Origins
![Page 20: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/20.jpg)
![Page 21: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/21.jpg)
TAGUCHI’S LOSS FUNCTION—A CAR PART EXAMPLE
Bad Car Part
Take Car to Dealer
Dealer Staffs
Accordingly
Time Wasted on Preventable Problem
Missing Work to
Deal with Repairs
![Page 22: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/22.jpg)
YOU MIGHT SAY THIS IS…
THE BUTTERFREE EFFECT!
![Page 23: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/23.jpg)
THE FOUNDATION OF TODAY’S LEADING COMPANIES
![Page 24: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/24.jpg)
THE BASIC A/B TESTING METHODOLOGY
A/B Learnings
Define Goals and
Hypothesis
Test a variable
Analyze and
Conclude
Record Findings
![Page 25: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/25.jpg)
The Golden Rule of A/B Testing:Only test 1 variable at a time!
![Page 26: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/26.jpg)
WHAT SHOULD I BE TESTING?• Email subject lines• Email preview text• CTAs• Colors• Size• Copy• Lengths of copy• Types• Sales copy• Buy now vs. learn more vs. x• Headlines• Font• Font color• Bullets vs. no bullets• Amount of content • Amount of choices or CTAs
• Page templates• Layouts• Heavy images or video vs. text• Email personalization tokens• Pricing• Pricing copy• Form aesthetics• Images – size, positioning, type, #• Email sending times• Email senders• Persona-based content• Number of form fields• Form field wording
![Page 27: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/27.jpg)
So…everything?
![Page 28: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/28.jpg)
THE CONUNDRUM – WHERE TO START!?
• Start with your goals and identify primary influencers
• Start with your leading hypotheses
• Start with key assets that underperform
• Start big and go narrow
![Page 29: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/29.jpg)
A/B Testing —the Basis for Growth Driven Design
![Page 30: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/30.jpg)
TRADITIONAL WEB DESIGN
1.5 - 2 Years
1.5 - 2 Years
3 Month Redesign
Legacy Website
Project Budget$25,000 - $250,000
3 Month Redesign
WebsiteV01
WebsiteV02
Project Budget$25,000 - $250,000
Impa
ct
Time
![Page 31: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/31.jpg)
05/03/2023
31
THE GDD DIFFERENCEIm
pact
1.5 - 2 Years
1.5 - 2 Years
3 Month Redesign
Launch Pad
Website
Project Budget$25,000 - $250,000
3 Month Redesign
Time
WebsiteV01
WebsiteV02
Project Budget$25,000 - $250,000
Gained Opportunity
Gained Opportunity
![Page 32: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/32.jpg)
LOOK AT YOUR TOP-VIEWED PAGES
![Page 33: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/33.jpg)
BRAINSTORM AND HYPOTHESIZE
A
B
![Page 34: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/34.jpg)
UTILIZE HEATMAPS AND USER SESSION DATA
![Page 35: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/35.jpg)
UTILIZE HEATMAPS AND USER SESSION DATA
![Page 36: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/36.jpg)
A/B Testing Software
![Page 37: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/37.jpg)
A/B TESTING TOOLS
![Page 38: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/38.jpg)
A/B Testing Without Tools
![Page 39: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/39.jpg)
It is an imperfect science, but it is quite doable!
![Page 40: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/40.jpg)
MANUAL TESTING
![Page 41: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/41.jpg)
CREATE A TRACKING SPREADSHEET
![Page 42: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/42.jpg)
HOW LONG IS LONG ENOUGH?It depends on:• Overall traffic• Traffic to that asset• The scale of data for your business
Pro Tip
Test it!
Find your benchmarks.
![Page 43: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/43.jpg)
USE A CONFIDENCE CALCULATOR
![Page 44: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/44.jpg)
A Client Case Study
![Page 45: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/45.jpg)
Our Hypothesis: 3D book covers will drive more clicks than flat book covers
![Page 46: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/46.jpg)
Flat Image 3D Image
Success Metric: % of views to clicks
![Page 47: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/47.jpg)
THE RESULTS MAY SURPRISE YOU
Slight difference in views to clicksBigger difference in clicks to submissions
![Page 48: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/48.jpg)
An A/B Testing Case Study – Sim City
![Page 49: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/49.jpg)
AN A/B TESTING CASE STUDY – SIM CITY
![Page 50: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/50.jpg)
MULTIVARIATE TESTING VS. A/B TESTING
A/B = testing 1 variable
Multivariate testing = testing many variables and combinations of those
Ideal for high traffic websites
Source: optimizely.com
![Page 51: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/51.jpg)
A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN
Success metric: sign-up rates
24 total combinations: 4 button – 6 image multivariate test
Hypothesis: video media elements will perform better
Source: blog.optimizely.com
![Page 52: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/52.jpg)
A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN
The Results• Hypothesis disproved
• 40.6% improvement rate
• 2,880,000 additional emails
• $60 million in donations
Source: blog.optimizely.com
![Page 53: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/53.jpg)
A REVIEW OF KEY CONCEPTS• A/B testing is not an option, it is a must to stay competitive
• Only test one factor at a time per hypothesis
• You must be patient and wait for a significant sample size before drawing conclusions
• You are never done with A/B testing – don’t settle
• Track the results! Set regular times to update and revisit your findings
• Make it part of your culture and daily thought process
![Page 54: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/54.jpg)
RESOURCES• How to Test Your Landing Pages infographic
• The A/B Test Calculator and A/B Testing Guide from HubSpot
• The A/B Test Flow Chart – print this out and hang it up!
• Hotjar – a free heat mapping tool
• 71 A/B Testing Ideas
• 10 A/B Testing Tools
• Growthdrivendesign.com
• http://hubs.ly/H03W7hV0
![Page 55: The Scientific Approach to Improving Conversion with A/B Testing](https://reader036.fdocuments.net/reader036/viewer/2022070603/5871b0171a28abda6a8b66ab/html5/thumbnails/55.jpg)
Thank You!