Analytics dictionary › Bayesian Statistics

Bayesian Statistics

Bayesian statistics (or Bayesian inference) is a method of statistical inference in which Bayes’ theorem is used to update the probability for a hypothesis as more evidence or information becomes available.

Bayesian statistics is widely used in A/B testing and more and more tools are implementing it as a basis for their stats engines.

AB Testing Chart - Bayesian
AB Testing Chart – Bayesian

Bayesian approach views probabilities like a more general concept. Following the Bayesian technique, you can use probabilities to represent the uncertainty in any event or hypothesis. Hence, it’s perfectly acceptable to assign probabilities to non-repeatable events, like the result of your new product launch campaign.

A/B testing tools using Bayesian-type statistics

  • Google Optimize
  • VWO
  • Adobe Target
  • AB Tasty
  • Dynamic Yield

There’s a good amount of shorter and longer articles describing why Bayesian is a better choice for those running A/B tests, for example, “The Power of Bayesian A/B Testing“, they all seem to contain the following reasoning. And of course, Frequentists would argue on several of them.

  • Bayesian gets reliable results faster (with a smaller sample)
  • Bayesian results are easier to understand for people without the background in statistics (Frequentist results are often misinterpreted)
  • Bayesian is better at detecting small changes (Frequentist favoring the null hypothesis).

Chris Stucchio has written a comprehensive overview of Bayesian A/B testing and the following section is mostly based on his white-paper.

Important variables of Bayesian testing:

α – underlying and unobserved true metric for variant A

β – underlying and unobserved true metric for variant B

Therefore, If we choose variant A when α is less than β, our loss is β – α. If α is greater than β, we lose nothing. Our loss is the amount by which our metric decreases when we choose that variant.

ε – the threshold of expected loss for one of the variants, under which we stop the experiment

More about statistics in A/B testing.

Sign up for Reflective Data

5,000 sessions / month ×

Please enter your email


After submitting your email, we will send you the registration form.