A/B Testing and Statistical Significance: How Much Traffic Do You Really Need?

Statistical significance means the difference between your variants is probably real and not down to chance. The honest answer for most B2B sites: you need at least hundreds of conversions per variant before an A/B test becomes reliable, and plenty of sites never reach that volume. Low on traffic? Then steer on other signals rather than on a dashboard full of noise. In this article you will read how much traffic and how many conversions you really need, how long to run a test, and when a different method serves you better.

We would rather look at this honestly than optimistically. A test that never reaches significance brings in no customers and no revenue. It only delivers false certainty. That is why our CRO approach always starts with one question: is your traffic large enough to test at all?

Test it yourself: check whether your result is statistically significant with our free A/B test calculator.

What does statistical significance mean in an A/B test?

In an A/B test you show two versions of a page to comparable visitors and measure which version converts better. Statistical significance is the threshold that tells you how confident you can be that the measured difference is not down to chance.

In practice, most tools work with a 95% confidence level. That means there is at most a 5% chance you are seeing a “winner” that is not actually a winner. It sounds strict, but that is exactly the point. Without that threshold you roll out improvements that deliver nothing, or worse: that quietly lower your conversion rate.

The problem is that significance only emerges with enough data. And “enough” is almost always more than people think. If you want to understand exactly how a conversion rate is built up, first read how to calculate conversion rate. That number is the basis you use to calculate sample size.

How many conversions do you need for a reliable A/B test?

Here comes the number many agencies would rather keep quiet. Peep Laja of CXL (ConversionXL) argues that A/B testing only becomes genuinely worthwhile from roughly 1,000 conversions per month. Not 1,000 visitors, but 1,000 conversions. For most B2B sites with a few dozen leads a month, that is a confronting figure.

Per individual test you want at least 250 conversions per variant before you can trust the outcome. Collect fewer and there is a good chance your “winner” turns out to be a loser next month. That is not optimisation, that is gambling with a chart next to you.

Run the numbers for yourself:

• Your site gets 2,000 visitors per month.
• Your conversion rate is 2%, so 40 conversions per month.
• Split those across two variants and you land on 20 conversions per variant.

With 20 conversions per variant you are nowhere near significance. You would have to keep testing for months, sometimes years, on a single change. By then your market, your offer or your audience will have shifted again.

How long should you run an A/B test?

A good rule of thumb: at least two full weeks, so you cover every day of the week and weekend patterns. But runtime alone tells you nothing. Your test has only run long enough once you have hit the required sample size and have seen a few full business cycles.

Never stop a test the moment it looks “barely” significant. That is called peeking, and it is one of the most common mistakes in CRO. Stopping early massively increases your chance of a false positive. Set your sample size upfront, and stop only once you have reached it.

Also mind the size of what you are testing. Very small changes, such as a button colour or a comma in your headline, rarely produce an effect large enough to reach significance with limited traffic. Test bigger, structural changes that touch the whole conversion funnel instead. Those stand a better chance of a measurable difference and are worth the testing time.

What do you do if your site has too little traffic to test?

This is the honest core of this article. Does your B2B site fall short of 1,000 conversions per month? Then classic A/B testing is not your best tool for now. And that is fine. It does not mean you cannot optimise, it means you deploy other, often better methods.

At low volumes these approaches deliver usable insight faster:

• Customer interviews and sales conversations. Five good conversations with customers often tell you more about doubts and objections than a test that never reaches significance. Also read conversion research.
• Usability testing. Have a handful of people use your site while thinking out loud. You see friction immediately, without needing hundreds of conversions.
• Heatmaps and session recordings. These show where visitors drop off or get stuck, qualitatively and right away.
• Site-wide improvements. With little traffic you are better off optimising the whole flow at once than testing separate elements individually. See B2B conversion optimisation for the approach.

The difference between an honest agency and a dashboard agency sits exactly here. We say what a big agency sometimes does not dare to: your site may be too small to test reliably, and in that case you pick a method that does work instead of pretending.

How do you calculate the sample size for your A/B test?

You do not need to be a statistician. Use a free sample size calculator (from Evan Miller and VWO) and fill in three things:

1. Your current conversion rate. For example 2%. Do not know it? Work it out first via how to calculate conversion rate.
2. The minimum effect you want to detect. If you want to be able to measure a 20% improvement, you need less data than for a 5% improvement.
3. Your confidence level. Usually 95%.

The calculator then gives you the number of visitors per variant. Do not be alarmed if that number runs into the tens of thousands: that is not a bug in the tool, that is the reality of statistics. The smaller the effect you want to measure, the more traffic you need. That is why it pays to test big, promising hypotheses first.

Frequently asked questions about A/B testing and significance

Is 95% significance always enough?

For most B2B decisions, yes. 95% means a 5% chance of a false positive. For decisions with major financial impact you can hold out for 99%, but then you need even more data. More important than raising the percentage is not stopping the test prematurely.

Can I test with very little traffic?

Technically a tool can always show a result, but it is not reliable. Below 250 conversions per variant the chance of a false positive is too great. At low volumes, choose customer interviews, usability testing or site-wide improvements.

What is the difference with multivariate testing?

In A/B testing you compare two versions; in multivariate testing you test multiple elements at once. Multivariate testing demands far more traffic still to reach significance. For most B2B sites that is not an option. More on this in A/B test vs multivariate test.

How long does an average test run?

At least two full weeks, and only close it once your predetermined sample size has been reached. With little traffic that can take months, and that is precisely why A/B testing is not the right tool for every site.

Does every conversion count, or only the final conversion?

That depends on your goal. If you test an intermediate step such as a form step, you measure that microconversion. If you test your whole offer, you measure the eventual lead or enquiry. Look at your entire funnel first via conversion research before you choose what to measure.

Ready to look honestly at your testing potential?

Statistical significance is not a button you switch on, it is a threshold your traffic has to clear. Do you have that volume? Then we help you test in a structured way on what delivers customers and revenue. Do you not have it? Then together we pick a method that does work, instead of a test that measures noise. Honest advice, a small team that moves fast, and focus on growth rather than vanity numbers.

Book your free intake