analytics-dictionary / term / experimentation

Holdout group

A holdout group is a set of users deliberately kept from a change or campaign, to measure its long-term impact.

Written by Reflective Data experts · updated Oct 5, 2026

Holdouts show the cumulative effect of many changes, or whether a marketing program really adds sales, by comparing against people who never received it.

They are a practical way to measure incrementality.

A/B test Attribution Return on ad spend (ROAS)

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Convert

Convert is an A/B testing platform known for privacy features and fast, lightweight testing.

feature-flag.md

Feature flag

A feature flag is a switch in code that turns a feature on or off for…

multiple-comparisons.md

Multiple comparisons problem

The multiple comparisons problem is that testing many metrics or variants raises the chance of at…

exposure-logging.md

Exposure logging

Exposure logging records the moment a user is actually shown a variant, so analysis only includes…

statistics.md

Statistics

Statistics is the science of collecting, analysing and interpreting data, and quantifying uncertainty.

sample-ratio-mismatch.md

Sample ratio mismatch (SRM)

A sample ratio mismatch is when the actual split of users between variants differs from the…

← full analytics dictionary