Statistical significance

Statistical significance is a measure of how likely it is that a difference observed in a test, such as one page version converting better than another, is a real effect rather than random chance. It is usually expressed as a confidence level, commonly ninety-five percent, set before the test begins.

Also called: significance

Published 29 September 2026

Why it matters to a buyer

Checking a test's results every day and stopping as soon as they look good is one of the most common ways a genuinely inconclusive test gets reported as a clear winner, because random noise will cross a significance threshold temporarily even when there is no real underlying difference, and stopping early catches that noise rather than the truth. For a buyer, this is why a test's planned sample size and duration should be set in advance and respected, rather than treated as a rough guide to check in on.

How it is measured or applied

A confidence level, typically ninety-five percent, is set before the test starts, along with the required sample size calculated from the site's current conversion rate and its minimum detectable effect. The test then runs for full weeks until the planned sample is reached, and the result is read once, at the end, rather than monitored daily and stopped whenever it happens to look favourable. Reaching statistical significance means the difference is unlikely to be random, but it is not the same as reaching the minimum detectable effect the test was actually sized to find with confidence.

Sources

Statistical significance, Optimizely

A/B test
An A/B test is an experiment that splits visitors at random between two versions of a page or element, the original and a variation, and compares how each performs on one chosen goal, such as sign-ups or purchases, to find out whether the change caused a real difference or just random variation.
Minimum detectable effect (MDE)
The minimum detectable effect is the smallest improvement in a metric, such as conversion rate, that an A/B test is designed to reliably detect given its sample size and duration.
Sample ratio mismatch (SRM)
Sample ratio mismatch is a data-quality problem in an A/B test where the actual split of visitors between variants differs from the intended split, such as a planned fifty-fifty split landing at fifty-six to forty-four, by more than chance would explain.
Multivariate testing
Multivariate testing is an experiment that tests multiple page elements, such as a headline, an image and a button colour, in combination at once, measuring not just each element's individual effect but how they interact.

Go deeper

From the blog

How we do it

A/B testing

Back to all terms

Thirty minutes to see where your site is invisible.

Book a free consultation with Ibrahem and Jeremy. We look at your site on search, AI answers and conversion while we talk, and you leave with a prioritised plan.

Book a free 30-minute consultation See pricing

30 minutes. No pitch deck. You leave with a prioritised plan.