Always Be Testing - Learn from Every A/B Test (Hiten Shah)
-
Upload
future-insights -
Category
Data & Analytics
-
view
3.793 -
download
0
Transcript of Always Be Testing - Learn from Every A/B Test (Hiten Shah)
Simple 4-step experimentation process
Data Figure out what to improve
Hypothesis Make an educated guess
Experiment Test your guess
Act Decide what’s next
Data helps you identify the most painful problems people have. (not the ones that bother you the most)
Find pages with high traffic and high bounce rate
HTTP://GOOGLE.COM/ANALYTICS
Increase Conversions By A/B Testing
Do a Happy Dance But Watch it Closely
Get More Traffic So You Can Start A/B Testing
Get More Traffic So You Can Do a Happy Dance
High Traffic
Low Traffic
High ConversionsLow Conversions
How To Figure Out What To Improve
What percentage of your
customers or users have
adopted each feature? Intercom on Product Management
When you’ve already got usage, analyze it.
Where do you have the most visitors, users, customers, emails, or notifications?
More volume helps you test faster.
๏ Website Traffic / App Installs
๏ Welcome Emails
๏ Web / Mobile Onboarding
๏ Email Digests
๏ Triggered Notifications
๏ Cart / Funnel Abandonment
๏ Dormant / Churned users
Common areas with large volume and high impact potential
A few qualitative tools
HTTP://USERTESTING.COM HTTP://SURVEYMONKEY.COM
Use simple equations to assess potential impact of each idea
DAN MCKINLEY HTTP://KISS.LY/DANETSY
visits x conversion rate x order value x li4 = ?
users x order value x resurrec7on rate = ?
ESTIMATED
ESTIMATED
MODELED
MODELED
Assess the potential impact,
difficulty and priority of
each test idea you have.
Prioritize your tests
Sensitivity tests help you
learn what your visitors care
about and what they don’t.
Run sensitivity tests on your web pages
1. Your starting hypothesis.
2. Dates for when you ran the test.
3. Screenshots of control and all variations used during the test.
4. Expected change in conversion.
5. Probability of change.
6. Screenshot of the raw data used in your analysis.
7. A screenshot to the A/B Test Report in KISSmetrics.
8. The decision you've made from the test.
9. What you learned from the test.
Document every A/B test and create a playbook
What learnings do we want to carry over from the past?
Helps you avoid unknowingly decreasing your conversions.
✓ Keep the footer links
✓ Autofocus on the first form field
✓ Do not include social proof
✓ Link to the features page
✓ Use “Log in” copy on the bu"on
Create experiment checklists based on learnings
Thank Y0u =) Got questions? Need help? Contact me any time!
Hiten [email protected]