How to Calculate AB Testing Sample Size
How to calculate the sample size an AB test needs before launch, why stopping early ruins the result, and what to do when your traffic is too low to test.
Calculate AB testing sample size from three numbers: your current conversion rate, the smallest lift worth detecting, and your chosen significance level — usually 95%. A free calculator using those three inputs will return the number of visitors each variant needs before you can trust the result.
This is one piece of a larger testing programme; the conversion rate optimization guide covers how sample size fits into the wider process of running CRO well.
On this page
Why does sample size matter at all?
Because small samples produce big, wrong-looking swings that mean nothing. A hundred visitors split into two groups of fifty will regularly show one variant "winning" by ten or twenty percentage points purely from noise, and that gap closes or reverses entirely as more visitors arrive. Sample size is the number of visitors it takes for that noise to settle down enough that a real difference, if one exists, becomes visible above it.
Think of it like flipping a coin. Flip it ten times and six heads is a completely unremarkable result, not evidence the coin is biased. Flip it ten thousand times and a sixty-forty split would be extraordinary evidence something is wrong with the coin. A conversion test works the same way: the same ten-percentage-point gap means nothing at fifty visitors per variant and means a great deal at fifty thousand. The visitor count is not a formality attached to the test — it is the entire reason the result can be trusted or not.
What three numbers actually determine the sample size?
Your baseline conversion rate, the minimum lift you want the test to reliably detect, and your significance level. Move any one of the three and the required sample size moves with it — a smaller lift you want to detect requires a larger sample, because a small true difference is harder to distinguish from noise than a large one.
| Input | Effect on required sample size |
|---|---|
| Lower baseline conversion rate | Larger sample needed — fewer conversions per visitor to learn from |
| Smaller minimum detectable lift | Larger sample needed — a subtler difference takes more data to confirm |
| Higher significance level (e.g. 99% vs 95%) | Larger sample needed — a stricter bar takes more evidence to clear |
| Higher statistical power (e.g. 90% vs 80%) | Larger sample needed — less willingness to miss a real effect |
A dedicated sample size calculator takes all four of these inputs and returns a number per variant, and using one is the correct move rather than estimating by feel — the relationship between the inputs is not linear, and gut estimates are reliably wrong in the optimistic direction [1].
It is worth internalizing why the relationship is not linear, because it changes how you should think about bold versus subtle tests. Halving the minimum detectable lift you want to catch does not double the required sample size — it roughly quadruples it, because the math behind detecting a difference scales with the square of the effect you are trying to see. That is the real reason experienced testers push for bigger, more confident changes rather than small tweaks: a subtle change is not just harder to notice, it is dramatically more expensive to prove statistically.
How do I actually calculate it before launching a test?
Pull your baseline conversion rate for the exact page being tested
Use the specific page or flow's own recent rate, not a site-wide average, which can be misleadingly different from any one page's actual number.
Decide the smallest lift worth the effort of the test
A one-point lift on a low-traffic page may not be worth testing for at all if it would take months to detect; be honest about what result would actually change a decision.
Put both numbers into a sample size calculator at 95% significance and 80% power
Those two defaults are the field standard for a reason — going lower on either raises your false-positive or false-negative rate more than most tests can afford.
Divide the total by your daily eligible traffic to estimate runtime
If the answer is months, either the page doesn't get enough traffic to test yet, or the change needs to be bolder so the expected lift — and therefore the required sample — is smaller.
Why is stopping a test early so damaging?
Because the significance level you set only holds if you look at the result once, at the end. Checking a live dashboard daily and stopping the moment a variant crosses your significance threshold is a different statistical test entirely — one with a much higher real false-positive rate than the number on the screen suggests, since a genuinely tied test will drift above and below any threshold repeatedly before it settles [2].
The instinct to peek is completely understandable and almost universal — watching a live number and wanting to act on it is normal. The fix is not willpower; it is deciding the sample size and the minimum runtime in advance, writing them down before the test launches, and treating the dashboard as informational only until both are met.
The test that gets stopped on day four because it 'clearly' won is almost always the one that would have flattened out by day fourteen. We ask every client to write the stopping rule down before launch, precisely because nobody trusts themselves to follow it from memory once the number looks good.
Some testing tools offer sequential testing methods designed to let you peek safely, adjusting the significance bar as you go rather than fixing it once at the end. These are legitimate and can be the right choice for a fast-moving team, but they are a different method with different math from a fixed-horizon test, not a workaround for skipping the planning step — pick one approach before launch and stay inside it.
What do I do if my traffic is too low to test?
Below a real traffic floor, running the test anyway does not fail safely — it produces a confident-looking answer that is actually noise, which is worse than admitting you cannot test yet. There are three honest paths when the math does not work.
Test only your single highest-traffic page or flow
Concentrating what traffic you have on one page reaches significance faster than spreading thin tests across several pages at once.
Make the change bolder so the expected lift is larger
A large, obvious change is easier to detect than a subtle one, which lowers the sample size the calculator asks for.
Switch to qualitative research instead, for now
Session replay and structured user interviews surface real problems without needing a statistically valid sample — see heatmaps and session replay for how that works as a substitute while traffic grows.
Checkout flows are a common place this floor gets tested in practice, because the traffic reaching the final step is always a fraction of the traffic reaching the site — see checkout CRO for how that plays out and what to test instead when a full multivariate checkout test is not realistic yet.
There is a temptation, when traffic is thin, to lower the significance bar instead of admitting the test cannot run yet — accepting an 80% or 90% confidence level instead of 95% so the calculator returns a smaller, more achievable number. Resist it. That is not a smaller version of a valid test; it is a test built to produce false positives more often, and shipping changes off it will cost more in wrong decisions than the qualitative alternative would have cost in time.
What tools actually run this kind of test?
Most dedicated AB testing platforms handle the split, the significance calculation and the reporting for you, which is worth paying for once your traffic clears the floor — building this by hand in a spreadsheet is possible but error-prone, and a platform purpose-built for it removes an entire category of mistake around how the traffic split and the statistics are implemented. What no platform does for you is the judgment calls: which page to test, what the hypothesis actually is, and whether the sample size math even makes sense for your traffic before you commit budget and weeks to a test.
That judgment is exactly what a CRO agency does as its core job, and it is worth being clear-eyed about the split: the tool runs the test, but the tool cannot tell you whether the test was worth running in the first place.
How does this connect to a landing page test specifically?
The same math applies, but landing pages usually have an advantage: they often carry more total traffic than a checkout page deep in the funnel, so they clear the sample size floor faster. That makes them a reasonable place to build the habit of calculating sample size properly before you have a checkout flow with enough volume to test — see landing page optimization for what to test first once the numbers say you are ready.
Once you have run a few tests with sample size calculated properly in advance, the process stops feeling like statistics homework and starts feeling like a normal part of shipping a page change — which is the point: the math is there to keep you from fooling yourself, not to slow you down.
Questions people also ask
What sample size do I need for an AB test?
It depends on your baseline conversion rate and the smallest lift you want to reliably detect — there is no single number. A calculator using your own baseline rate and minimum detectable effect will give you the real figure for your page.
Can I stop a test early if it looks like a clear winner?
Not without cost. Checking results repeatedly and stopping the moment one variant looks ahead inflates your false-positive rate well beyond the significance level you set, because random noise regularly produces a temporary lead that reverses if the test keeps running.
What is statistical significance in plain terms?
It is a measure of how likely your result is to be real rather than random noise. A 95% significance level means that if there truly were no difference between variants, you would still see a result this large about 5% of the time by chance alone.
How long should an AB test run at minimum?
At least one full business cycle, usually two full weeks, even if the sample size is reached sooner — behavior differs by day of week, and a shorter window can catch an unrepresentative slice of traffic and call it a result.
What if my site doesn't have enough traffic to reach the sample size?
Test on your highest-traffic page only, widen the change so the expected lift is larger and therefore easier to detect, or run the test longer than usual. Below a certain traffic floor, qualitative research is a better use of the same time than an underpowered test.
Sources
- Sample Size Calculator — Evan Miller (checked 22 September 2026)
- A/B Testing 101 — Nielsen Norman Group (checked 22 September 2026)
- A/B testing definition — Optimizely (checked 22 September 2026)
- A/B testing guide — VWO (checked 22 September 2026)
Check whether your traffic can support a real test
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Jeremy ZavaCo-founder
Leads growth and client partnerships.
Works on: CRO, SEO
Reviewed before publishing by Ibrahem Helal, Co-founder.
- CRO vs SEO: Which Should Come First?SEO earns visits and CRO decides what they're worth. Here's how the two disciplines differ, where they overlap, and which to fund first with a limited budget.
- The Landing Page Optimization ChecklistA practical checklist for optimizing a landing page: what to fix before you test, what to test first, and the mistakes that quietly cap your conversion rate.
- Checkout Optimization: A Practical GuideA practical checkout optimization guide covering the fixes that reduce cart abandonment most reliably, from guest checkout to shipping cost placement.
- Heatmaps and Session Replay for CROHeatmaps show where attention goes on average; session replay shows what one visitor actually did. Use them together, and watch recordings, not just maps.