analytics-dictionary / term / experimentation

Novelty effect

The novelty effect is a temporary change in behaviour caused simply by something being new.

Written by Reflective Data experts · updated Oct 5, 2026

Users may click a new design more at first just because it is different, then return to normal. Short tests can mistake this for a lasting win.

Run tests for full business cycles and look at how the effect changes over time.

// how we can help

We set up, audit and run experimentation programs on any tool. A/B testing services → · Audits

A sample ratio mismatch is when the actual split of users between variants differs from the…

Bayesian statistics expresses results as the probability that a variant is better, updated as data arrives.

LaunchDarkly is a feature management platform used to control feature rollouts with flags.

Win rate is the percentage of experiments that produce a significant improvement.

GrowthBook is an open-source experimentation and feature flagging platform that works with your own data warehouse.

A guardrail metric is a measure you monitor during a test to make sure the change…