A/B Testing and Statistical Significance: How Much Traffic Do You Really Need?

Statistical significance means the difference between your variants probably isn’t down to chance.

For most B2B sites, that takes hundreds of conversions per variant.

Plenty of sites never reach that volume.

The verdict: below roughly 1,000 conversions a month, an A/B test mostly measures noise. Steer on other signals instead.

A test that never reaches significance brings in no customers and no revenue.

It only delivers false certainty.

That’s why our CRO approach always starts with one question: is your traffic large enough to test at all?

Test it yourself: check whether your result is significant with our free A/B test calculator.

What does statistical significance mean in an A/B test?

It’s the threshold that tells you how confident you can be that a measured difference is real.

In an A/B test you show two versions of a page to comparable visitors.

Then you measure which version converts better.

The 95% convention

Most tools work with a 95% confidence level.

That means at most a 5% chance you’re seeing a “winner” that isn’t one.

It sounds strict. That’s the point.

Without that threshold you roll out changes that deliver nothing.

Or worse: changes that quietly lower your conversion rate.

Significance needs data

Significance only emerges with enough data.

And “enough” is almost always more than people think.

Your conversion rate is the starting point for that sum. Work it out first with how to calculate conversion rate.

How many conversions do you need for a reliable A/B test?

At least 250 conversions per variant, and ideally 1,000 conversions a month across the site.

The 1,000-a-month rule

Here’s the number many agencies would rather keep quiet.

Peep Laja of CXL (ConversionXL) argues that A/B testing only becomes genuinely worthwhile from roughly 1,000 conversions per month.

Not 1,000 visitors. 1,000 conversions.

For most B2B sites with a few dozen leads a month, that’s a confronting figure.

250 per variant per test

Per test you want at least 250 conversions per variant before you trust the outcome.

Collect fewer and your “winner” may well turn out to be a loser next month.

That isn’t optimisation. That’s gambling with a chart next to you.

Run the numbers for yourself

Here’s an example with typical B2B figures:

• Your site gets 2,000 visitors per month.
• Your conversion rate is 2%, so 40 conversions per month.
• Split across two variants, that’s 20 conversions per variant.

With 20 conversions per variant you’re nowhere near significance. Here’s how far off that is:

You’d have to test for months, sometimes years, on a single change.

By then your market, your offer or your audience will have shifted again.

How long should you run an A/B test?

At least two full weeks, so you cover every weekday and the weekend pattern.

But runtime alone tells you nothing.

A test has only run long enough once it hits the required sample size and has seen a few full business cycles.

Don’t stop at “barely significant”

Stopping the moment a test looks significant is called peeking.

It’s one of the most common mistakes in CRO.

Stopping early massively increases your chance of a false positive. Our piece on A/B test peeking shows why.

Rule: set your sample size upfront, and stop only once you’ve reached it.

Test big changes, not commas

Mind the size of what you’re testing.

A button colour or a comma in your headline rarely produces an effect big enough to measure with limited traffic.

Test bigger, structural changes that touch the whole conversion funnel instead.

Those are far likelier to show a difference.

What do you do if your site has too little traffic to test?

Use other methods.

Below 1,000 conversions a month, classic A/B testing isn’t your best tool for now.

And that’s fine. It means you use methods that work at your volume.

Here’s the decision in one picture:

Four methods that work at low volume

At low volumes these approaches deliver usable insight faster:

• Customer interviews and sales conversations: five good conversations reveal doubts and objections. Also read conversion research.
• Usability testing: a handful of people use your site while thinking out loud.
• Heatmaps and session recordings: see where visitors drop off or get stuck.
• Site-wide improvements: optimise the whole flow at once. See B2B conversion optimisation.

Usability tests show friction immediately, without hundreds of conversions.

Heatmaps and recordings do the same, qualitatively and right away.

A free tool like Microsoft Clarity covers both. Look at the scroll depth figure on its demo dashboard:

Microsoft Clarity’s homepage, October 2026. The demo dashboard shows scroll depth, the number that tells you how many visitors leave the first screen.

That’s insight you get from day one, with no significance threshold to clear. More in our guide to website heatmaps.

Say it out loud

This is where an honest agency and a dashboard agency differ.

Your site may be too small to test reliably.

Then you pick a method that works instead of pretending.

How do you calculate the sample size for your A/B test?

With a free sample size calculator and three inputs. You don’t need to be a statistician.

Use the one from Evan Miller or Wingify, the company behind VWO. Fill in:

1. Your current conversion rate. For example 2%. Don’t know it? Work it out via how to calculate conversion rate.
2. The minimum effect you want to detect. Measuring a 20% improvement needs less data than a 5% improvement.
3. Your confidence level. Usually 95%.

Here’s what that looks like with the example above, 2% and a 20% effect:

Don’t be alarmed by the result

The calculator gives you the number of visitors per variant.

It often runs into the tens of thousands.

That isn’t a bug. That’s statistics.

The smaller the effect you want to measure, the more traffic you need.

So test big, promising hypotheses first.

Frequently asked questions about A/B testing and significance

Is 95% significance always enough?

For most B2B decisions, yes. 95% means a 5% chance of a false positive. For decisions with major financial impact you can hold out for 99%, but then you need even more data. More important than raising the percentage is not stopping the test prematurely.

Can I test with very little traffic?

Technically a tool can always show a result, but it isn’t reliable. Below 250 conversions per variant the chance of a false positive is too great. At low volumes, choose customer interviews, usability testing or site-wide improvements. Our guide to A/B testing with low traffic goes deeper.

What is the difference with multivariate testing?

In A/B testing you compare two versions; in multivariate testing you test multiple elements at once. Multivariate testing demands far more traffic still to reach significance. For most B2B sites that’s not an option. More on this in A/B test vs multivariate test.

How long does an average test run?

At least two full weeks, and only close it once your predetermined sample size has been reached. With little traffic that can take months, and that’s precisely why A/B testing isn’t the right tool for every site.

Does every conversion count, or only the final conversion?

That depends on your goal. If you test an intermediate step such as a form step, you measure that microconversion. If you test your whole offer, you measure the eventual lead or enquiry. Look at your entire funnel first via conversion research before you choose what to measure.

Ready to check your testing potential?

Significance is a threshold your traffic has to clear, not a switch.

Have the volume? We’ll help you test what brings customers and revenue. Don’t have it? We’ll pick a method that works instead of one that measures noise.

Contact us and we’ll tell you within 24 hours whether your site can carry an A/B test.