A/B Testing: The Complete Guide
What A/B testing is, when your traffic can support it, how to run a split test that holds up, and how to read the result without fooling yourself.
A/B testing is splitting live traffic at random between the current version of a page and one changed version, then measuring which one produces more of a single goal, such as sign-ups or orders. It works when you decide the goal, the change and the sample size before you start, and it fails when you peek and stop early.
This guide covers what a split test is, when your traffic can support one, how to run it step by step, how to read the numbers, and which pages to test first. If you want a team to run the programme for you, our A/B testing service does exactly that, and the plans are on the pricing page.
On this page
What is A/B testing, and what is it not?
A/B testing, also called split testing, is a controlled experiment on a live page. Version A is what you have now. Version B has one change. Each new visitor is sent to one version at random, and you count how often each group does the thing you want.
The random split is the whole point. If you change a headline on Monday and sales go up on Tuesday, you do not know if the headline did it, or a newsletter went out, or a competitor ran out of stock. When both versions run at the same time for random halves of the same audience, all of those outside forces hit both groups equally. What is left is the effect of your change, plus some noise that statistics can measure.
It helps to be clear about what A/B testing is not. It is not a before-and-after comparison, which is what most "we changed the page and conversion went up" stories really are. It is not a survey, where people tell you what they think they would do. And it is not a way to find out what to change. A test can only judge an idea you bring to it. The ideas come from research: analytics, session recordings, user interviews and a structured CRO audit of the pages that matter.
There are close cousins. A split URL test sends visitors to two different URLs rather than swapping content on one page, which is useful for full redesigns. A multivariate test changes several elements at once and measures every combination. That needs far more traffic, and the trade-offs are laid out in A/B testing vs multivariate testing. For almost every site we see, a plain two-version A/B test is the right default.
| Method | What it compares | What it can tell you | Traffic needed |
|---|---|---|---|
| A/B test | One control vs one variant, at the same time | Whether one change caused a lift | Moderate |
| Split URL test | Two different URLs, at the same time | Whether a redesign beats the original | Moderate |
| Multivariate test | Every combination of several changes | Which elements matter and how they interact | High |
| Before-and-after | The same page in two time periods | Very little; time effects swamp the change | Any |
| User research | What people say and do in sessions | Why they hesitate, and what to test next | Low |
When is A/B testing worth it, and when is it a waste?
A/B testing is worth it when a page gets enough conversions to reach a clear answer in a few weeks, and when the change is uncertain enough that you genuinely do not know which version wins. If either of those is missing, testing burns time you could spend shipping improvements.
Traffic is the first filter. The number that matters is not visits but conversions on the goal you are measuring. A pricing page with fifty thousand visits and forty sign-ups a month cannot tell a small improvement from noise in any reasonable time. A checkout with two thousand orders a month can. Put your numbers into a sample-size calculator before you do anything else [3]; if the answer is three months or more, testing that page is not the right tool yet.
Uncertainty is the second filter. Some changes do not need a test. A broken form on mobile, a price that does not match the cart, a button that is hidden below a cookie banner: fix those and move on. Testing is for the choices where smart people disagree, like a shorter form versus a longer one that qualifies leads, or a pricing page with three plans versus four.
The third filter is what you can act on. A test is only useful if the result will change a decision. If the team will ship the new design whatever the numbers say, the test is theatre. Say up front what you will do if B wins, if A wins, and if they tie.
When testing is not viable, the work does not stop. You run research, rank the problems by how much money they touch, and ship the fixes that are clearly right. Our view of how that trade-off plays out between hiring help and building a team is in CRO agency vs in-house.
How do you run an A/B test, step by step?
You run a test by writing a hypothesis, choosing one primary metric, calculating the sample size, building and checking the variant, launching to a random split, and then waiting until the sample is complete before you read the result. The order matters, because each step stops a specific way of fooling yourself.
The hypothesis comes first because it forces you to say why you expect a change. "Blue button" is not a hypothesis. "Visitors on mobile do not see the add-to-cart button without scrolling, so moving it above the fold will raise add-to-cart rate" is. It names the problem, the change and the metric. We keep a template for this in how to write a CRO hypothesis, and it is the single habit that most improves the quality of a testing programme.
The primary metric is the one number that decides the test. Pick one, write it down, and pick it before launch. You can watch secondary metrics, such as revenue per visitor or bounce rate, but they do not get a vote on the winner. If you let yourself choose the metric after the data arrives, one of twenty metrics will always look like a win.
Write the hypothesis
Name the problem you saw in research, the change you will make, and the metric you expect it to move. One change per test.
Pick one primary metric
Choose the action closest to revenue that still gets enough volume, such as completed checkouts or trial starts. Write it down before launch.
Calculate the sample size
Enter your baseline conversion rate and the smallest lift worth detecting into a calculator. That gives the visitors needed per version and, from your traffic, the run time.
Build and QA the variant
Check both versions on every major browser and device, confirm the tracking fires once per conversion, and make sure the page does not flicker the old version first.
Launch on a random 50/50 split
Assign new visitors at random and keep each visitor in the same version on return visits. Run for whole weeks so weekdays and weekends are both covered.
Wait for the full sample
Do not stop because a dashboard shows a winner early. Read the result only once the planned sample is complete.
Decide and record
Ship the winner, keep the control on a tie, and write down what you learned, including losses, so the next test starts smarter.
Quality assurance is where most tests quietly break. A variant that loads half a second slower, a tracking tag that fires twice, or a layout that breaks on one phone will all bias the result, and none of them will show up as an error. Check them before launch, not after a surprising result.
How much traffic and time does an A/B test need?
A test needs enough visitors per version to detect the smallest lift you care about, and that number depends on your current conversion rate and how small a lift you want to see. Smaller lifts and lower baseline rates both need much more traffic. In practice, most sites should plan for two to six weeks per test.
Three inputs set the sample size. The baseline conversion rate is what the page converts at today. The minimum detectable effect is the smallest relative change worth knowing about, such as a 10% lift. Statistical power and significance level are the error rates you accept, usually 80% power and 5% significance. A calculator turns those into visitors per version [3].
The relationship is harsh. Halving the lift you want to detect roughly quadruples the sample you need. That is why low-traffic sites should test bold changes, not button colours: a bold change has a chance of producing a lift big enough to see.
Time has its own rules on top of sample size. Run for at least one full week, and preferably two, even if you hit the sample sooner, because people behave differently on a Monday morning and a Saturday night. Avoid launching during a sale or a holiday unless the test is about that period. And do not run for months to squeeze out significance; the longer a test runs, the more cookies get cleared and the more returning visitors get counted twice.
| Input | If it goes up | What to do about it |
|---|---|---|
| Baseline conversion rate | Sample needed goes down | Test on steps with higher rates, such as add-to-cart instead of purchase |
| Minimum detectable effect | Sample needed goes down sharply | Test bolder changes on low-traffic pages |
| Power (for example 80% to 90%) | Sample needed goes up | Keep 80% unless a wrong call would be very costly |
| Significance (stricter, for example 5% to 1%) | Sample needed goes up | Keep 5% for most page tests |
| Number of variants | Sample needed goes up | Test one variant against control unless traffic is high |
If the numbers do not work on your key page, move the metric up the funnel. Clicks to checkout happen far more often than completed orders, so a test on that step finishes sooner. Just be honest that a lift in clicks is not proof of a lift in revenue.
How do you read A/B test results without fooling yourself?
Read a result only when the planned sample is complete, look at the primary metric first, and ask how big the effect is and how sure you can be, not just whether a tool painted it green. A result is a range of plausible effects, not a single number, and the range is usually wider than people expect.
The most common mistake has a name: peeking. If you check a test every day and stop the first time it crosses significance, your real error rate is far higher than the 5% you think you signed up for. Evan Miller's write-up of this problem shows how repeated checking inflates false positives, and his fix is simple: set the sample size in advance and do not stop early [2]. Some tools now offer sequential methods that are built to allow early looks, but you have to switch them on and understand them.
The second mistake is slicing. After a test ends, it is tempting to split the result by device, country, new versus returning visitors and traffic source. With enough slices, something will always look like a big win. Treat a slice result as an idea for the next test, never as a finding.
The third is ignoring the size of the effect. A statistically significant lift of 0.2% on a low-value page may not pay for the engineering time to keep the variant. Look at the confidence interval. If it runs from "slightly worse" to "much better", you have not learned much yet.
You will also meet two schools of statistics. Frequentist tests give you a p-value and a confidence interval; Bayesian tests give you a probability that B beats A and an expected loss if you are wrong. Many platforms support both engines [4]. Neither makes a small sample big. The practical differences, and when each one suits a team, are covered in Bayesian vs frequentist A/B testing.
Does A/B testing hurt SEO?
No, as long as you follow Google's published rules for website testing [1]. Those rules exist because a badly set-up test can look like cloaking or create duplicate pages. Set it up properly and search engines treat it as normal site behaviour.
Google's guidance comes down to four points. Do not cloak: show Googlebot the same content any other visitor might see, and do not single out its user agent to always get one version. If your variants live on separate URLs, put a rel=canonical on each variant pointing to the original. If you redirect visitors to a variant URL, use a temporary 302 redirect, not a permanent 301, so search engines keep the original in the index. And run the test only as long as it needs to; once you have an answer, update the page and remove the test code [1].
There is also a speed angle. Client-side testing scripts can delay rendering or cause the page to flash the old content before swapping in the new one. That hurts users and can hurt your Core Web Vitals, the loading, responsiveness and layout-stability measures Google reports on [5]. Where you can, run tests server-side or at the edge, and keep the client-side snippet small. How load time links to conversion is its own topic, and we cover it in page speed and conversion rate.
What should you A/B test first?
Test first on the page with the most money flowing through it and the clearest problem in your research. For an online store that is usually the product page or checkout. For a SaaS company it is usually the pricing page or sign-up flow. For a service business it is often the homepage and the enquiry form.
Pick the page by money, then pick the change by evidence. Look at where people drop off in analytics, watch recordings of the sessions that drop off, and read what support and sales hear most. The idea that comes out of that research is a far better first test than anything from a list of generic best practices. For a longer list of starting points ranked by likely impact, see our A/B test ideas.
Different pages fail in different ways, so each has its own playbook:
- Product pages usually lose sales on missing information, weak images, unclear delivery costs and a buy box that sinks below the fold on phones. Product page optimization walks through what to fix and test.
- Pricing pages lose buyers to confusing plan names, too many options and no clear answer to "which plan is for me". Pricing page optimization covers plan layout, anchoring and the tests worth running.
- Homepages lose people who cannot tell in five seconds what you do and who it is for. Homepage conversion optimization is about the first screen and the paths out of it.
- Buttons and calls to action lose clicks on vague wording and weak placement. Call-to-action optimization covers copy, placement and contrast.
- Phones lose conversions on long forms, small tap targets and slow loads. Mobile conversion optimization deals with the device where most of your traffic now arrives.
- SaaS sign-ups lose people on long forms, forced credit cards and empty first sessions. SaaS signup optimization covers the path from landing page to activated account.
A good first test also has a short path to revenue. A change on the checkout step pays back faster than a change to the blog sidebar, because every extra conversion there is money. Save the upper-funnel tests for when the programme is running and trusted.
Which A/B testing tools do you need?
You need three things: a tool that splits traffic and records conversions, an analytics setup you trust, and a way to watch real sessions. You do not need an expensive platform to start. The tool matters less than the process, and a cheap tool used well beats an expensive one used carelessly.
Testing platforms fall into two broad groups. Hosted visual editors let marketers build variants without code, at the cost of a client-side script and a monthly fee that usually scales with traffic. Developer-first and open-source tools run tests in your own code or at the edge, which is faster and cheaper to run but needs engineering time. Since Google Optimize was retired, most teams have picked one of those two routes. We compare the options by budget and stack in the best A/B testing tools.
Around the testing tool sits the rest of the stack: analytics for the funnel, heatmaps and session recordings for the "why", surveys for the words customers use, and a place to log every test and its result. Our wider list, with what each category is for and what it costs, is in the best CRO tools.
| Job | What the tool does | Can you start free? |
|---|---|---|
| Split traffic and measure | Assigns visitors, serves variants, calculates results | Yes, with open-source tools |
| Funnel analytics | Shows where people drop off | Yes |
| Session recordings and heatmaps | Shows how people use the page | Yes, on limited plans |
| On-page surveys | Asks people why they hesitate | Yes, on limited plans |
| Test log | Records every hypothesis, result and decision | Yes, a shared sheet works |
Whatever you choose, check one thing before you commit: that the tool can keep a visitor in the same version across visits and devices where it matters, and that its conversion counts match your analytics within a few percent. If they do not match, fix the tracking before you run anything.
What results should you expect from A/B testing?
Expect most tests to lose or tie. That is normal and not a sign that testing is failing. The value of a programme comes from the few clear winners, which you keep, and from the losers, which stop you from shipping changes that would have quietly cost money.
Be wary of anyone who promises a fixed uplift. Nobody can know in advance how your visitors will respond to a change, and a promise of "30% more conversions" is a sales line, not a forecast. What an honest team can promise is a cadence: how many tests per month, how each is documented, and how results are reported.
Wins also shrink. A variant that shows a big lift in the test often settles lower once it is live for everyone, partly because the first result was on the lucky side of the noise and partly because novelty fades. Plan on the lower end of the confidence interval, not the headline number.
The honest way to judge a programme is over a quarter or two, not per test. Count how many tests reached a clear answer, how many winners shipped, and what the combined change in the primary metric was against a stable baseline. We show how we report that, with a clearly labelled worked example, in what CRO results look like.
Should you run A/B tests in-house or hire help?
Run tests in-house if you have steady traffic, a developer who can build variants, and someone who owns the research and the statistics. Hire help if any of those is missing, or if tests keep ending "inconclusive" and nobody knows why. Many teams do both: an outside partner sets up the process and the first tests, then hands over.
The skills are the hard part, not the tools. A testing programme needs research, copy and design, front-end development, analytics and statistics. A single hire rarely covers all of them, which is why small in-house programmes often stall at two or three tests a year.
We sell this service, so weigh our view with that in mind. The fair test for any partner is the same one we would apply: can they show you a documented hypothesis for each test, a sample-size plan before launch, and a written result after it, including the tests that lost? The full comparison of costs and trade-offs is in CRO agency vs in-house, and if you would like us to look at your site, the A/B testing service page explains how our engagements work.
What this guide covers next
Each part of this guide has a deeper post of its own. Here is where to go for each one:
- The best A/B testing tools compares testing platforms by budget, stack and team size.
- A/B test ideas lists tests worth running, ranked by likely impact.
- The best CRO tools covers the analytics, recording and survey tools around a test.
- A/B testing vs multivariate testing explains when to test one change and when to test many.
- CRO agency vs in-house weighs cost, speed and skills for each route.
- Bayesian vs frequentist A/B testing explains the two ways of reading a result.
- How to run a CRO audit is where good test ideas come from.
- How to write a CRO hypothesis gives you a template for every test.
- Page speed and conversion rate explains why load time belongs in every testing plan.
- What CRO results look like shows how to report a programme honestly.
Questions people also ask
What is A/B testing in simple terms?
It is showing two versions of the same page to two random groups of visitors at the same time, then counting which version gets more of the action you care about. Because the groups are random and run side by side, the difference can be pinned on the change rather than on luck or the season.
How much traffic do I need to run an A/B test?
Enough to reach the sample size your calculator gives you in two to six weeks. As a rough guide, a page with a few hundred conversions a month can detect only large changes. If the calculator says months, run research and ship obvious fixes instead of testing.
How long should an A/B test run?
Until it reaches the sample size you fixed before launch, and for at least one full week, ideally two, so every weekday is covered. Do not stop early because the dashboard shows a winner; stopping on a peek is the most common way to ship a false result.
Does A/B testing hurt SEO?
Not if you follow Google's guidance: do not show search crawlers a different page than users, use rel=canonical on variant URLs, use temporary 302 redirects rather than permanent ones, and end the test once you have an answer.
Is A/B testing worth it for a small business?
Sometimes. If you have steady traffic and one page that drives most revenue, yes. If you have little traffic, you will learn more from a conversion audit, user interviews and session recordings, and you should ship the fixes they reveal without waiting for a test.
Sources
- Minimize A/B testing impact in Google Search — Google Search Central (checked 22 September 2026)
- How Not To Run an A/B Test — Evan Miller (checked 22 September 2026)
- Sample Size Calculator — Evan Miller (checked 22 September 2026)
- Statistics overview — GrowthBook (checked 22 September 2026)
- Web Vitals — web.dev (Google) (checked 22 September 2026)
Find out if your site is ready for A/B testing
Book a free 30-minute consultation with Ibrahem and Jeremy. We look at your site while we talk, and you leave with a prioritised plan.
Who wrote this
Ibrahem HelalCo-founder
Leads strategy and delivery.
Works on: SEO, GEO
Reviewed before publishing by Jeremy Zava, Co-founder.
- Best A/B Testing Tools, ComparedA working list of A/B testing tools by team size and budget, with what each one actually changes about how you test and what it costs to run.
- 37 A/B Test Ideas Worth Actually RunningA practical list of A/B test ideas across headlines, pricing, forms and trust signals, grouped by where they tend to pay off and how to prioritize them.
- What Is a CRO Hypothesis? A Working TemplateA CRO hypothesis names the change, the reason it should work, and the metric that proves it — anything looser is a guess wearing a hypothesis's clothes.
- How to Run a CRO Audit (Free Checklist)A CRO audit is a structured review of analytics, heuristics and real user sessions that finds the specific reasons visitors leave before converting.