analytics-dictionary / term / experimentation

Multiple comparisons problem

The multiple comparisons problem is that testing many metrics or variants raises the chance of at least one false positive.

Written by Reflective Data experts · updated Oct 5, 2026

If you look at twenty metrics at a 5 percent level, one will probably look significant by chance. Corrections such as Bonferroni or controlling the false discovery rate keep errors in check.

Name one primary metric in advance and treat the rest as secondary.

Guardrail metric Statistical significance Type I and type II errors

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Minimum detectable effect (MDE)

The MDE is the smallest change your test is designed to reliably detect.

fooc.md

Flash of original content (FOOC)

FOOC is when visitors briefly see the original page before a client-side test applies the variant.

google-optimize.md

Google Optimize (sunset)

Google Optimize was Google free A/B testing tool, discontinued on 30 September 2023.

one-tailed-vs-two-tailed-test.md

One-tailed vs two-tailed test

A one-tailed test looks for an effect in one direction only; a two-tailed test looks for…

statistics.md

Statistics

Statistics is the science of collecting, analysing and interpreting data, and quantifying uncertainty.

experimentation-program.md

Experimentation program

An experimentation program is the organisation, process and tooling that lets a company run experiments consistently.

← full analytics dictionary