Analytics dictionary › Bayesian Statistics

# Bayesian Statistics

Bayesian statistics (or Bayesian inference) is a method of statistical inference in which Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available.

Bayesian statistics is widely used in A/B testing and more and more tools are implementing it as a basis for their stats engines.

Bayesian approach views probabilities like a more general concept. Following the Bayesian technique, you can use probabilities to represent the uncertainty in any event or hypothesis. Hence, it’s perfectly acceptable to assign probabilities to non-repeatable events, like the result of your new product launch campaign.

A/B testing tools using Bayesian-type statistics

• VWO
• AB Tasty
• Dynamic Yield

There’s a good amount of shorter and longer articles describing why Bayesian is a better choice for those running A/B tests, for example, “The Power of Bayesian A/B Testing“, they all seem to contain the following reasoning. And of course, Frequentists would argue on several of them.

• Bayesian gets reliable results faster (with a smaller sample)
• Bayesian results are easier to understand for people without the background in statistics (Frequentist results are often misinterpreted)
• Bayesian is better at detecting small changes (Frequentist favoring the null hypothesis).

Chris Stucchio has written a comprehensive overview of Bayesian A/B testing and the following section is mostly based on his white-paper.

#### Important variables of Bayesian testing:

α – underlying and unobserved true metric for variant A

β – underlying and unobserved true metric for variant B

Therefore, If we choose variant A when α is less than β, our loss is β – α. If α is greater than β, we lose nothing. Our loss is the amount by which our metric decreases when we choose that variant.

ε – the threshold of expected loss for one of the variants, under which we stop the experiment