Bayesian statistics (or Bayesian inference) is a method of statistical inference in which Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available.
Bayesian statistics is widely used in A/B testing and more and more tools are implementing it as a basis for their stats engines.
Bayesian approach views probabilities like a more general concept. Following the Bayesian technique, you can use probabilities to represent the uncertainty in any event or hypothesis. Hence, it’s perfectly acceptable to assign probabilities to non-repeatable events, like the result of your new product launch campaign.
A/B testing tools using Bayesian-type statistics
- Google Optimize
- VWO
- Adobe Target
- AB Tasty
- Dynamic Yield
There’s a good amount of shorter and longer articles describing why Bayesian is a better choice for those running A/B tests, for example, “The Power of Bayesian A/B Testing“, they all seem to contain the following reasoning. And of course, Frequentists would argue on several of them.
- Bayesian gets reliable results faster (with a smaller sample)
- Bayesian results are easier to understand for people without the background in statistics (Frequentist results are often misinterpreted)
- Bayesian is better at detecting small changes (Frequentist favoring the null hypothesis).
Chris Stucchio has written a comprehensive overview of Bayesian A/B testing and the following section is mostly based on his white-paper.
Important variables of Bayesian testing:
α – underlying and unobserved true metric for variant A
β – underlying and unobserved true metric for variant B
Therefore, If we choose variant A when α is less than β, our loss is β – α. If α is greater than β, we lose nothing. Our loss is the amount by which our metric decreases when we choose that variant.
ε – the threshold of expected loss for one of the variants, under which we stop the experiment
More about statistics in A/B testing.
Last modified: June 12, 2019
Want to see more articels like What is Bayesian Statistics? Check out all definitions in the Analytics Dictionary.