analytics-dictionary / term / experimentation

Randomization unit

The randomization unit is what gets assigned to a variant: a user, a session, an account or a device.

Written by Reflective Data experts · updated Oct 5, 2026

It should match how people experience the change. Randomising by session can show the same person different variants; randomising by user is more consistent. B2B tests often randomise by account.

The analysis unit should be the same as, or compatible with, the randomization unit.

A/B test Exposure logging Sample ratio mismatch (SRM)

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Peeking problem

Peeking is checking results repeatedly and stopping the moment they look significant, which inflates false positives.

multivariate-test.md

Multivariate test

A multivariate test changes several elements at once and measures the effect of every combination.

variance-reduction.md

Variance reduction

Variance reduction techniques make experiments more sensitive, so they reach conclusions with less data.

multi-armed-bandit.md

Multi-armed bandit

A multi-armed bandit shifts traffic towards better-performing variants while a test is still running.

guardrail-metric.md

Guardrail metric

A guardrail metric is a measure you monitor during a test to make sure the change…

statsig.md

Statsig

Statsig is a product experimentation platform with feature gates, experiments and analytics.

← full analytics dictionary