Post on 13-Jul-2020
Bayesian AB Testing
Chris Campbell
Bayesian AB Testing
• Use prior information• Prevent peeking problem• Reduce impact of multiple comparisons• Probability of effect more interpretable than threshold
Section title
Search Item Click AB Test
5
Search Item Click
Section title
6
7
Search Item Clicks
Paired t-test
• Sample size
• Minimum detectable effect
power.t.test(n = 28, sd = 895, power = 0.9)
delta = 789.5388
9
Power Calculation
Bayesian Estimation Supersedes the t-Test
library(BEST)BESTmcmc(clk$A - clk$B,
priors = list(muM = 0, muSD = 1400))
11
BEST
BEST
Posterior Probability
= Prior
Probability +
New Evidence
Probability of Direction
13
82%
14
Probability of Direction
Highest Density Interval
Highest Density Interval
89% of densityCredible Interval
Region of Practical Equivalence
ROPE
Overlap of Credible Interval with ROPE
library(rstanarm)stan_glm(
clicks ~ group +weekday,
data = clk)
19
Bayesian GLM
Better control of additional explanatory variables
Bayesian GLM
Summary
• Large quantity of modelling methods available• Intuitive interpretation
21