How to Run A/B Tests That Give You Answers, Not Noise

A/B testing is the backbone of evidence-based conversion rate optimisation. But most businesses run tests badly. They stop tests too early, test too many things at once, or draw conclusions from data that is not statistically significant. The result is not optimisation. It is guesswork with extra steps.

This guide is for marketers who want to run tests that produce reliable answers. Not impressive-looking dashboards. Real answers.

What A/B Testing Actually Is (and Is Not)

An A/B test compares two versions of something, a page, a headline, a CTA button, to see which one performs better against a specific metric. Version A is the control, your current design. Version B is the variant, the change you want to test. Traffic is split randomly between the two, and you measure the difference in outcomes.

What A/B testing is not: a way to validate ideas you have already decided on. If you run a test hoping for a specific result, you will find ways to justify ending it early or reading the data selectively. Go in with genuine curiosity about what works.

Statistical Significance: The Concept You Cannot Skip

Statistical significance tells you the probability that the difference you observed is real, not just random noise. In most A/B testing, the standard threshold is 95% confidence. That means there is only a 5% chance that the observed difference occurred by luck.

Why does this matter? Because small sample sizes produce volatile results. If you flip a coin 10 times and get 7 heads, that does not mean the coin is biased. You just need more flips. The same applies to A/B tests. A variant that is "winning" after 50 visitors might be losing after 500.

Never call a test based on how the numbers look today. Call it when you have reached statistical significance.

How to Calculate Significance

You do not need to do the maths by hand. Tools like Google Optimize (now sunset but its methodology lives on in alternatives), VWO, Optimizely, and free online calculators will compute significance for you. Feed in your sample size, conversion count for each variant, and the tool tells you whether the result is significant.

If you are using a tool that does not calculate significance automatically, use a Bayesian or frequentist calculator. There are good free ones from Evan Miller and AB Testguide.

Sample Size: Know the Number Before You Start

Before running any test, calculate the minimum sample size you need. This depends on three factors: your baseline conversion rate, the minimum detectable effect (the smallest improvement you care about), and the significance level (usually 95%).

If your landing page converts at 3% and you want to detect a 20% relative improvement (going from 3% to 3.6%), you will need roughly 12,000 visitors per variant. If your page only gets 500 visitors a week, that test will take nearly six months. Knowing this upfront tells you whether the test is worth running at all, or whether you should test a bigger change that requires a smaller sample to detect.

What If I Do Not Have Enough Traffic?

Low-traffic sites can still run A/B tests. You just need to adjust your approach. Test bigger changes, not subtle tweaks. A completely rewritten headline vs the original will produce a detectable difference faster than changing one word. Alternatively, focus your testing on higher-traffic pages where you can reach significance faster.

Another option is to consolidate traffic. If you are running ads to five different landing pages, consider running your test on just one page and sending all traffic there temporarily.

Test Duration: Why You Cannot End Early

A/B tests need to run for a minimum duration regardless of sample size. The reason is cyclical variation. Visitor behaviour changes by day of week. Monday traffic behaves differently from Saturday traffic. A test that runs only three days captures a biased slice of your audience.

Run every test for at least one full business cycle, which for most businesses means a minimum of one week. Two weeks is better. If you reach your required sample size in four days, keep the test running until the end of the week anyway.

One Variable at a Time: The Iron Rule

If you change the headline, the CTA text, and the hero image simultaneously, and the variant wins, what won? You have no idea. Was it the headline? The image? The CTA? Some combination? You cannot tell, which means you cannot learn.

Test one variable at a time. If you want to test your headline, keep everything else identical. Once you have a winner, lock that in and test the next variable. This is slower. It is also the only way to build genuine understanding of what drives conversions on your pages.

The exception is multivariate testing, which tests multiple variables simultaneously using combinatorial analysis. But multivariate testing requires very high traffic volumes. For most businesses, sequential A/B testing is the practical path.

Prioritisation Frameworks: What to Test First

You have a list of 20 things you could test. You cannot test them all. Which ones do you start with?

The ICE Framework

Score each test idea on three dimensions from 1 to 10:

• Impact: If this test wins, how big an improvement do you expect?
• Confidence: How sure are you that this change will produce a positive result?
• Ease: How easy is it to implement and run this test?

Average the three scores and rank your test ideas from highest to lowest. Run the highest-scoring tests first.

The PIE Framework

Similar to ICE, but uses different dimensions:

• Potential: How much room for improvement does this page or element have?
• Importance: How valuable is the traffic on this page?
• Ease: How easy is it to run the test?

Both frameworks work. The point is to have a system, any system, that stops you from testing random ideas and starts you testing the ideas most likely to deliver results.

Common A/B Testing Mistakes

Peeking at Results Daily

Looking at test results every day is tempting. It is also dangerous. Early in a test, the data is noisy. You will see swings that look meaningful but are not. The temptation to "call it early" when one variant looks like it is winning is the single most common cause of false conclusions in A/B testing.

Set a calendar reminder for when your test should end. Check the results then. Not before.

Testing Trivial Changes

Changing your button colour from blue to green is unlikely to meaningfully change your conversion rate unless your button was virtually invisible before. Test things that affect the decision-making process: the headline, the offer, the form structure, the pricing display. Cosmetic tweaks rarely move the needle.

Ignoring Segmentation

An A/B test might show no overall winner. But when you segment by device, mobile visitors strongly preferred variant B while desktop visitors preferred variant A. Always check your results by device, traffic source, and any other meaningful segment. The aggregate can hide important patterns.

Not Documenting Results

Every test, winner or loser, teaches you something. Keep a testing log that records the hypothesis, the test setup, the result, and what you learned. Over time, this log becomes the most valuable asset your CRO programme produces. It tells you what your audience responds to and what they do not.

What to Do With Losing Tests

A test that does not win is not a failure. It is information. It tells you that the change you made either does not matter to your visitors, or was not the right change. Both are useful things to know.

Review the losing variant. Did it address the right objection? Was the change big enough to be noticed? Was the test run long enough? Sometimes a losing test reveals that the problem is elsewhere on the page, not in the element you tested.

Good testing is patient, disciplined, and honest. It does not always feel exciting. But the compound effect of running 20 well-designed tests over a year will transform your conversion rate in ways that no amount of guessing can match. For more on improving specific page elements, check our landing page optimisation guide.

Frequently Asked Questions

How long should an A/B test run?

At minimum, one full week to account for day-of-week variation. Ideally two weeks. Beyond duration, the test also needs to reach your calculated minimum sample size before you can draw conclusions. Do not end a test early even if the results look decisive.

Can I run multiple A/B tests on the same page at the same time?

Technically yes, but it introduces interaction effects that can pollute your data. If you are testing a new headline and a new CTA simultaneously but as separate tests, changes in one might influence the other. Unless you are doing proper multivariate testing with sufficient traffic, run one test per page at a time.

What is the difference between A/B testing and multivariate testing?

A/B testing compares two complete versions of a page or element. Multivariate testing tests multiple variables simultaneously in different combinations to find the best overall combination. Multivariate testing requires significantly more traffic to reach significance because you are testing many combinations at once.

What tools do I need to run A/B tests?

At minimum, you need a testing platform (VWO, Optimizely, or Google Tag Manager with some configuration), analytics to track results (Google Analytics 4), and a sample size calculator to plan tests in advance. Some platforms bundle all of this together.