analytics-dictionary / term / experimentation

Statistical power

Statistical power is the probability that a test detects a real effect of a given size.

Written by Reflective Data experts · updated Oct 5, 2026

A test with low power will often miss genuine improvements. Power depends on sample size, the size of the effect, the variability of the metric and the significance level.

Run a power calculation before launching to know how many users and how long you need.

A/B test Minimum detectable effect (MDE)Statistical significance

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Client-side testing

Client-side testing changes the page in the visitor browser using JavaScript, typically with a visual editor.

frequentist-statistics.md

Frequentist statistics

Frequentist statistics judges evidence by how likely the data would be if there were no real…

aa-test.md

A/A test

An A/A test splits users into two groups that get the same experience, to check that…

multiple-comparisons.md

Multiple comparisons problem

The multiple comparisons problem is that testing many metrics or variants raises the chance of at…

sequential-testing.md

Sequential testing

Sequential testing uses statistical methods that remain valid when you check results repeatedly while a test…

optimizely.md

Optimizely

Optimizely is a digital experience platform known for web experimentation, feature experimentation and personalisation.

← full analytics dictionary