analytics-dictionary / term / experimentation

Peeking problem

Peeking is checking results repeatedly and stopping the moment they look significant, which inflates false positives.

Written by Reflective Data experts · updated Oct 5, 2026

If you check a fixed-horizon test every day, chance alone will eventually produce a significant looking result. Calling the test then leads to shipping changes that do nothing.

Use a pre-planned duration or a sequential method designed for continuous monitoring.

P-value Sequential testing Statistical significance

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

GrowthBook

GrowthBook is an open-source experimentation and feature flagging platform that works with your own data warehouse.

confidence-interval.md

Confidence interval

A confidence interval is a range of values that likely contains the true effect, given the…

simpsons-paradox.md

Simpson’s paradox

Simpson's paradox is when a trend that appears in several groups reverses when the groups are…

exposure-logging.md

Exposure logging

Exposure logging records the moment a user is actually shown a variant, so analysis only includes…

server-side-testing.md

Server-side testing

Server-side testing runs experiments in your backend, so users get the assigned variant before the page…

client-side-testing.md

Client-side testing

Client-side testing changes the page in the visitor browser using JavaScript, typically with a visual editor.

← full analytics dictionary