Skip to content
~/reflective_data

analytics-dictionary / term / experimentation

A/B test

An A/B test compares two versions of something by showing each to a random group of users and measuring the difference in outcomes.

Written by Reflective Data experts · updated Oct 5, 2026

Version A is usually the existing experience (the control) and version B the change (the variant). Because users are assigned at random, differences in outcomes can be attributed to the change rather than to other factors.

Good experiments start with a clear hypothesis, a primary metric and a plan for sample size before launch.

related

Conversion rateFeature flagStatistical powerStatistical significance

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

more in experimentation

vwo.md

VWO

VWO is an A/B testing and conversion optimisation platform with visual editing, heatmaps and surveys.
effect-size.md

Effect size

Effect size is the magnitude of a difference between variants, not just whether it exists.
simpsons-paradox.md

Simpson’s paradox

Simpson's paradox is when a trend that appears in several groups reverses when the groups are…
google-optimize.md

Google Optimize (sunset)

Google Optimize was Google free A/B testing tool, discontinued on 30 September 2023.
statistics.md

Statistics

Statistics is the science of collecting, analysing and interpreting data, and quantifying uncertainty.
minimum-detectable-effect.md

Minimum detectable effect (MDE)

The MDE is the smallest change your test is designed to reliably detect.

← full analytics dictionary