Post on 16-Apr-2017
Overview
• Introduction
– What is A/B testing?
– Comparing web and field tests
• Analysis
• Effects that can affect test
– Denominator issues
– Temporal effects
– Hidden bias
• Bootstrapping
• From validation to understanding mechanisms
A/B testing
• Essentially a randomized trial
• Split traffic 50:50
• One group sees ‘normal’ site
• The other group sees the variant or ‘treatment’
• After a set period of time, calculate the difference in KPIs between the two groups
• Generally, you can attribute the difference to the treatment
http://www.smashingmagazine.com/2010/06/the-ultimate-guide-to-a-b-testing/
Web vs. field trials
• Data
– Quantity
– Quality
– Type
• Web data: large quantities, low quality until aggregation and cohort creation, observational.
• Field trials: small to medium quantities, high quality information about participants, combination of direct responses, tests and observations.
Events
• Launching the widget
• Adding item to Bag
• Rotating MeModel
• Pressing Share button
• Adding garment to try on
A single event
web 2015-01-14 04:41:20.000 2015-01-14 04:41:53.480 struct0e833b00-d2cb-436b-ad1d-21fa47474b80 primary
js-2.2.0cloudfront hadoop-0.5.0-common-0.4.0 XX.XX.XX.X2091617875 aca45a2fbc191e7b 3 BR
-XX.XXXX -XX.XXXXX https://live-cdn.me-tail.net/wanda-ui/5a180420-2416-11e2-81c1-0800200c9a66/pt-BR/?xdm_e=http%3A%2F%2Fwww.dafiti.com.br&xdm_c=default4031&xdm_p=1#init-data/%7B%22retailerPageType%22%3A%22productListing%22%2C%22open%22%3Afalse%7D http://www.dafiti.com.br/roupas-femininas/casacos-e-jaquetas/ https live-cdn.me-tail.net 80 /wanda-ui/5a180420-2416-11e2-81c1-0800200c9a66/pt-BR/
xdm_e=http%3A%2F%2Fwww.dafiti.com.br&xdm_c=default4031&xdm_p=1 init-data/%7B%22retailerPageType%22%3A%22productListing%22%2C%22open%22%3Afalse%7D http www.dafiti.com.br 80 /roupas-femininas/casacos-e-jaquetas/ unknown
TabBar OpenTab productListing
Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36 Chrome Chrome
39.0.2171.95 Browser WEBKIT pt-BR 1 1 10 0 0 1 0 0 1 24 216 0Windows Windows Microsoft CorporationAmerica/Sao_Paulo Computer 0 1366 768 UTF-8 2160
• Widget launched by clicking on tab
• Selected Information:– Timestamp
– UserID (cookie ID)
– Geolocation: Country code, longitufe & latitude (usually of ISP), Timezone
– IP address
– URLs: host, current, referrer
– Event hierarchy
• Others:– Browser information
– Device & OS information
– Session counter
Logs of millions of events
• Store all raw logs in the cloud
• Create aggregates of specific every day– Use aggregates to create cohorts
UserID Retailer Engaged? Order value OrderID Group Test Ratio User Type
0157dab05efbef6f XX null 68.11 68137749 out 50 ExistingBin
0158ee5980cc75ad XX null null null in 90 NewBin
015ab3acaba4c770 XX TRUE null null in 90 NewBin
015e3a8e1d5ad181 XX null null null in 90 NewBin
015e3da4002e861a XX null null null in 90 NewBin
0160ae8d4465773b XX TRUE null null in 90 NewBin
0161f081a2c51d9f XX null null null out 50 ExistingBin
01647bcd7185da9d XX null 96.27 27342749 out 50 NewBin
Aggregated slice over some time period by user
Basic A/B test
• Change in homepage
• Measure difference in Average Order Value (AOV) between control and treatment.
https://www.google.co.uk/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwjip4_nxr3JAhWFJw8KHTbdAaoQjRwIBw&url=http%3A%2F%2Fpadicode.com%2Fblog%2Femail-marketing-2%2Fab-testing-resources%2F&psig=AFQjCNEnv6hj7n9VW-z0tx54iLl5c3srIA&ust=1449158617165351
Data distribution
• Skewed distribution
• Prices are often distributed log normally.
• So, depending on the skew and extreme values, the mean can fluctuate without it indicating a real effect.
Significance testing
• z-test of lognormal means between groups
• Critical threshold: 0.05
• H0 = 0; Ha > 0
• Single-tailed p-value:– 0.00682
• Double-tailed p-value– 0.0137
Pitfalls
• Temporal effects
– Novelty
– Spikes
• Noise
– Dilution
• Bias
– Bucketing bias
– Asymmetric cohorts
• Bugs
Temporal effects
• Novelty: strong effect at the beginning wanes over time.
• Spikes/spurious data: Atypical mechanisms.
• Monitoring only p-value going below a critical threshold will misinterpret the effect.
Spikes
• Toy Scenario
Property Group A Group B
Stable conversion 0.2 0.22
Sale conversion 0.3 0.5
Visitor rate (per group)
N(200,20) N(200,20)
Where 𝑁 𝜇, 𝜎 are normal distribution paramters. Conversion data generated using a binomial distribution.
Spikes
• Spurious effects like flash sales can quickly push p-value below critical threshold.
• Sale on Day 6 increases conversion in both groups but difference is higher in Group B.
• Increase in cumulative conversion reduces p-value dramatically.
Spikes
Spurious effects like flash sales can quickly push p-value below critical threshold.
days all_visitors all_orders_a all_orders_b conversion_a conversion_b difference p_value
1 194 32 34 0.165 0.175 0.010 0.393
2 355 63 76 0.177 0.214 0.037 0.109
3 566 110 126 0.194 0.223 0.028 0.121
4 772 155 168 0.201 0.218 0.017 0.208
5 953 185 210 0.194 0.220 0.026 0.079
6 1154 244 300 0.211 0.260 0.049 0.003
7 1331 279 346 0.210 0.260 0.050 0.001
8 1537 323 386 0.210 0.251 0.041 0.003
9 1713 362 422 0.211 0.246 0.035 0.007
10 1931 404 467 0.209 0.242 0.033 0.008
11 2135 451 511 0.211 0.239 0.028 0.014
12 2341 497 555 0.212 0.237 0.025 0.021
13 2553 544 606 0.213 0.237 0.024 0.019
14 2770 595 666 0.215 0.240 0.026 0.011
15 2956 633 707 0.214 0.239 0.025 0.011
16 3143 670 749 0.213 0.238 0.025 0.009
17 3388 713 803 0.210 0.237 0.027 0.004
18 3590 757 843 0.211 0.235 0.024 0.007
Temporal variability
• Instability in effect size.
• Depending on the A/B test, the long term instability of effect size can be debilitating after roll-out.
• But can also provide a source of insight.
Dilution
• Users who can see treatment are a small fraction of population.
http://www.infoq.com/presentations/ab-testing-pinterest
• Create a counterfactual cohort from control group for correct measurement.
Dilution
• Treatment only made available in the ‘Variant’ group and only a small fraction of treatment group actually go through the treatment.
• Can use Instrumental Variables to scale the overall effect
Instrumental variables
• Bin = Instrument (Z)• Instrument is able to “predict” actual treatment
(T)• 2 Stage linear regression
– 𝑌 = 𝛼 + 𝛽𝑇 + 𝜖 (ideal equation)– 𝑌 = 𝜌 + 𝜎𝑍 + 𝜃 (measured values at bin level)– 𝑇 = 𝛾 + 𝛿𝑍 + 휀 (isolating treatment effect)– 𝑌 = 𝛼 + 𝛽 𝛾 + 𝛿𝑍 (re-stated ideal equation)– 𝛽 = 𝜎/𝛿
• When treatment is only possible in one group, 𝛿is a proportion and the effect size of ‘true’ treatment vs. control is the effect at the bin level scaled by 1/𝛿.
Simulate beforehand
• Know that adoption is important to see the difference at the level of Treatment (In) vs. Control (Out).
• Estimate how long it will take to see a statistically significant effect given:1000 digitised garments, 10% adoption and higher conversion rate for engaged users.
Bucketing bias
• Non-representative population
– Bugs in bucketing
– Bugs in treatment not showing up on selective devices
• Run A/A test
• Examine cohorts carefully
Bucketing bias
• Asymmetric bucketing– Smaller group gets
values from high density regions
– Larger group gets the full range
– Smaller group becomes a non-representative sample
• Run A/A test
• Symmetric bucketing
Bootstrapping
• No assumptions about data distribution
• Can calculate any metric
• Explicitly performs the assumptions of hypothesis testing so easier to explain
• Can also interpreted from a Bayesian perspective Reference
Bootstrapping
http://www.texample.net/tikz/examples/bootstrap-resampling/
Bootstrapping
• Resultant Gaussiandistribution of metric
• Need to check convergence of bootstrap samples.
• Can be used to get a distribution of differences.
http://rosetta.ahmedmoustafa.io/bootstrap/
Bayesian models
• Explicit mechanistic modelling of important parameters.
• Answers the question: “What are the range of conversion rates that result in the observed data?”– Use knowledge that conversion rates can be modelled
with a binomial distribution and the parameter space to be explored is p.