analytics-dictionary / term / experimentation

P-value

A p-value is the probability of seeing a result at least as extreme as yours if there were actually no difference.

Written by Reflective Data experts · updated Oct 5, 2026

A small p-value, typically below 0.05, suggests the result is unlikely to be pure chance. It does not tell you the probability that the variant is better, nor how large or valuable the effect is.

Always read it together with the effect size and confidence interval.

Confidence interval Frequentist statistics Statistical significance

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

A/A test

An A/A test splits users into two groups that get the same experience, to check that…

server-side-testing

Server-side testing

Server-side testing runs experiments in your backend, so users get the assigned variant before the page…

guardrail-metric

Guardrail metric

A guardrail metric is a measure you monitor during a test to make sure the change…

statistics

Statistics

Statistics is the science of collecting, analysing and interpreting data, and quantifying uncertainty.

type-1-and-type-2-errors

Type I and type II errors

A type I error is a false positive, declaring an effect that is not real; a…

experiment-velocity

Experiment velocity

Experiment velocity is how many experiments a team completes in a period.

← full analytics dictionary