analytics-dictionary / term / experimentation

Canary release

A canary release exposes a new version to a small group of users to detect problems before everyone gets it.

Written by Reflective Data experts · updated Oct 5, 2026

It is a safety technique: if the canary group shows errors or worse metrics, the release is stopped. It differs from an A/B test in that the aim is safety, not measuring an improvement.

Many teams use the same flagging infrastructure for both.

Feature flag Guardrail metric Progressive delivery

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Multi-armed bandit

A multi-armed bandit shifts traffic towards better-performing variants while a test is still running.

experimentation-program.md

Experimentation program

An experimentation program is the organisation, process and tooling that lets a company run experiments consistently.

win-rate.md

Win rate

Win rate is the percentage of experiments that produce a significant improvement.

statsig.md

Statsig

Statsig is a product experimentation platform with feature gates, experiments and analytics.

ab-test.md

A/B test

An A/B test compares two versions of something by showing each to a random group of…

optimizely.md

Optimizely

Optimizely is a digital experience platform known for web experimentation, feature experimentation and personalisation.

← full analytics dictionary