analytics-dictionary / term / experimentation

Type I and type II errors

A type I error is a false positive, declaring an effect that is not real; a type II error is a false negative, missing a real effect.

Written by Reflective Data experts · updated Oct 5, 2026

The significance level controls the type I rate; statistical power controls the type II rate. Reducing one usually raises the other unless you add data.

Which mistake is more costly depends on the decision: shipping a harmful change or missing a good one.

P-value Statistical power Statistical significance

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Frequentist statistics

Frequentist statistics judges evidence by how likely the data would be if there were no real…

bayesian-statistics.md

Bayesian statistics

Bayesian statistics expresses results as the probability that a variant is better, updated as data arrives.

multiple-comparisons.md

Multiple comparisons problem

The multiple comparisons problem is that testing many metrics or variants raises the chance of at…

kameleoon.md

Kameleoon

Kameleoon is an experimentation and personalisation platform covering both web and feature experimentation.

fooc.md

Flash of original content (FOOC)

FOOC is when visitors briefly see the original page before a client-side test applies the variant.

cuped.md

CUPED

CUPED reduces variance in experiments by using pre-experiment data about each user, so tests reach conclusions…

← full analytics dictionary