analytics-dictionary / term / experimentation
Frequentist statistics
Frequentist statistics judges evidence by how likely the data would be if there were no real effect.
Written by Reflective Data experts · updated Oct 5, 2026
It uses p-values and confidence intervals, and it works with a fixed sample size decided in advance. It is the traditional approach and still the most common in A/B testing tools.
Peeking at results and stopping early breaks its guarantees unless the method is designed for it, as in sequential testing.
related
Bayesian statisticsConfidence intervalP-valueSequential testing
// how we can help
We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits