analytics-dictionary / term / experimentation

Simpson’s paradox

Simpson's paradox is when a trend that appears in several groups reverses when the groups are combined.

Written by Reflective Data experts · updated Oct 5, 2026

It occurs when group sizes differ between variants, for example if one variant happened to get more mobile traffic. Segmenting results by key dimensions can reveal it.

It is a reminder that random assignment, not post-hoc slicing, is what protects experiments.

A/B test Sample ratio mismatch (SRM)Statistics

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Practical significance

Practical significance asks whether a measured effect is large enough to matter to the business.

type-1-and-type-2-errors.md

Type I and type II errors

A type I error is a false positive, declaring an effect that is not real; a…

optimizely.md

Optimizely

Optimizely is a digital experience platform known for web experimentation, feature experimentation and personalisation.

minimum-detectable-effect.md

Minimum detectable effect (MDE)

The MDE is the smallest change your test is designed to reliably detect.

progressive-delivery.md

Progressive delivery

Progressive delivery releases a change to a small share of users first and expands gradually while…

feature-flag.md

Feature flag

A feature flag is a switch in code that turns a feature on or off for…

← full analytics dictionary