analytics-dictionary / term / experimentation

Statistical significance

A result is statistically significant when it is unlikely to be explained by chance alone, given a chosen threshold.

Written by Reflective Data experts · updated Oct 5, 2026

The threshold, often a 5 percent significance level, is set before the test. Significance does not mean the effect is large or important: a tiny difference can be significant with enough traffic.

Decide in advance how big a change must be to matter to the business.

A/B test Minimum detectable effect (MDE)P-value Statistical power

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Win rate

Win rate is the percentage of experiments that produce a significant improvement.

multivariate-test.md

Multivariate test

A multivariate test changes several elements at once and measures the effect of every combination.

ab-tasty.md

AB Tasty

AB Tasty is an experimentation and personalisation platform for web and product teams.

holdout-group.md

Holdout group

A holdout group is a set of users deliberately kept from a change or campaign, to…

effect-size.md

Effect size

Effect size is the magnitude of a difference between variants, not just whether it exists.

experimentation-program.md

Experimentation program

An experimentation program is the organisation, process and tooling that lets a company run experiments consistently.

← full analytics dictionary