Post on 08-Jan-2017
Brian Gardner - Director, Product ManagementIlya Izrailevsky - Engineering Manager
Implementing analytic and optimization tools on your website or app
WiFi: QBConnect Password: not required#QBConnect
2
Today’s Speakers
Brian GardnerDirector, Product Management@brianwgardner
Ilya IzrailevskyEngineering Manager@iizrailevsky
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your product is working (or not)
Agenda
Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your product is working (or not)
Agenda
5
“The sooner you run an experiment, the sooner you are likely to find a surprise, and the surprise is the market speaking to you, telling you something you didn't know. Several of our businesses here came out of surprises.”
– Scott Cook, Intuit founder & chairman
Intuit’s tradition of experimentation
Deep Customer Empathy
Go Broad to Narrow
Rapid Experiments
with Customers
CustomerDelight
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
6
What is an A/B test?
BA#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Why A/B Test?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Discover which experiences delight your customer
Accelerate your learning loop
Make faster, better decisions
with data
8
Who We Serve
9
Introducing:
WASABI
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
10
First time use test: QuickBooks Self EmployedTest: Complete tax profile first Control: Connect account first
In the test cell, we see:• Significant increase in tax profile completion• Slight increase in categorizing a transaction from a card• No change to connecting a bank• No change to categorizing on the transactions page
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
11
First time use test: QuickBooks Self EmployedTest: Complete tax profile first Control: Connect account first
In the test cell, we see:• Significant increase in tax profile completion: +20%• Slight increase in categorizing a transaction from a card: +6%• No change to connecting a bank: Flat• No change to categorizing on the transactions page: Flat
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your product is working (or not)
Agenda
13
• Marketing vs Product (changing content & UI vs features or back-end)• How much engineering is involved?• Do you have sensitive data?• What platforms and devices do you need to test across?• Cost• Analytics provided / options• Efficient use of traffic• Performance• Ease of use
How do you choose a tool for A/B testing?What are the dimensions you should think about when choosing A solution?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
14
Landscape of Popular Options
• WYSIWIG• Minimal Engineering• Marketing focused
• Developer focused• Flexible engineering effort• Own your own data
• Customized• Ongoing maintenance• Own your own data
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
COMMERCIAL OPEN SOURCE IN-HOUSE
WASABI
Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your product is working (or not)
Agenda
16
The A/B Testing Lifecycle: Iterate to Awesomeness
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Step 1:Plan an experiment
Step 2:Develop a feature to test
Step 3:Instrument your application with Wasabi API Calls
Step 4:Create and run the experiment using the Wasabi UI
Step 5:Deploy and track results quickly
17
• Is this user already assigned for this experiment?
• Does the user qualify for the experiment? (Segmentation Rules)
• Is the user already in a conflicting experiment? (Mutual Exclusion)
• Randomization• Check vs. Sampling Percentage
• Assign to an experience (Bucket)
• Return assignment
How does user assignment actually work?What should I show this user when a WASABI test is active?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Demo
19
Architecture: behind the scenes, here’s how it’s built
Web tier
Admin UI API API
Segmentation
Mutual Exclusion
Sampling
Bucket Dice Roll
Application tier
Cassandra MySQLData tier
Authentication
User Directory
Logging
Metrics
Pluggable Components
Bucket assignments
Events / analytics
Kafka
Stream
HDFS SparkVertica S3
Pluggable Components
Wasabi A/B Testing Service
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Introduction & Overview
Understand the landscape analytic and optimization tools
How to implement an A/B test
Gain the ability to make informed decisions about what in your product is working (or not)
Agenda
21
Best practices and considerations for a good A/B test
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Get started! (Don’t let the perfect be the enemy of the good)1
Think about what you’ll learn from a “failed” test in advance.
Be skeptical – don’t take your results at face value• Track all your key metrics – check for unintended consequences
• Think about sources of bias, assumptions: timing, infrastructure, etc.
Don’t stop at one round of experiments!
2
3
4
5
Don’t forget qualitative research as well
22
What Should You Take Away?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Everyone can A/B test! (It’s not just for marketers)
A lot of A/B testing tools in the market. Choose wisely!
Take time to come up with your hypothesis and how you’ll measure it. It can
make or break your experiment.
23
If you’re interested, here’s how you get Wasabi:
https://github.com/intuit/wasabi
Questions?
#QBConnect | @IntuitDev | @brianwgardner | @iizrailevsky
Contact us: data@intuit.com
Questions?
Thanks for coming!We’d love your feedback.Before you head out, take a minute to rate this session in the QuickBooks Connect mobile app.
Keep the conversation goingdeveloper.intuit.com/hub/quickbooks-connect
Access Session SlidesEngage with Speakers and Fellow DevelopersGet Questions Answered