analytics-dictionary / term / experimentation

One-tailed vs two-tailed test

A one-tailed test looks for an effect in one direction only; a two-tailed test looks for a difference in either direction.

Written by Reflective Data experts · updated Oct 5, 2026

Two-tailed tests are the safer default because a change can also hurt. One-tailed tests need less data but ignore effects in the other direction, so use them only with a strong, pre-stated reason.

Choose before looking at the data.

Frequentist statistics P-value Statistical power

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

Variance reduction

Variance reduction techniques make experiments more sensitive, so they reach conclusions with less data.

holdout-group.md

Holdout group

A holdout group is a set of users deliberately kept from a change or campaign, to…

multiple-comparisons.md

Multiple comparisons problem

The multiple comparisons problem is that testing many metrics or variants raises the chance of at…

optimizely.md

Optimizely

Optimizely is a digital experience platform known for web experimentation, feature experimentation and personalisation.

server-side-testing.md

Server-side testing

Server-side testing runs experiments in your backend, so users get the assigned variant before the page…

win-rate.md

Win rate

Win rate is the percentage of experiments that produce a significant improvement.

← full analytics dictionary