A/B Testing: What to Do When You Have Too Little Traffic

A/B testing means showing two versions of a page side by side to see which one converts better. It is the gold standard of conversion optimization. But there is a problem almost no one names honestly: most B2B sites have too little traffic to test reliably. In this article you will read how A/B testing works, how to set up a good test, and, more importantly, what to do when your traffic is too low.

Test it yourself: check the significance of your variants with our free A/B test calculator.

What is A/B testing?

In an A/B test you show half of your visitors version A (the original) and the other half version B (your new idea). You then measure which version delivers more conversions, a method Optimizely describes as the basis of data-driven optimization. If B clearly wins, you roll it out.

The power: you do not guess, you measure. You replace opinions with evidence. Instead of a debate about what “feels better”, you let your visitors vote with their behavior. That is why testing is the heart of a good CRO approach.

How do you set up a good A/B test?

A test is only as good as its design. Four elements decide whether your result is worth anything.

These four elements do not follow one another in isolation, they form a single cycle together: the insight from one test feeds the hypothesis of the next.

Start with a hypothesis

Never test something at random. First formulate a hypothesis: “if we do [change], then [conversion] rises because [reason]”. In practice we see teams without a hypothesis move buttons around haphazardly and afterwards have no idea why something worked. A hypothesis forces you to think about the reason, and that reason is reusable later. The common mistake: testing on gut feeling instead of on an observation from conversion research.

Change one variable per test

If you want to know what made the difference, change one thing at a time: only the headline, or only the call to action, not both. If you change several things together, you know that B won, but not because of what. If you do want to test several elements at once, you need multivariate testing, and that demands even more traffic.

Set the sample size and duration in advance

Before you start, calculate how many visitors and conversions you need per variant, and how long the test will then run. Let a test run for at least one to two full weeks so you capture all days of the week: behavior on Monday differs from Sunday. The common mistake is stopping the test as soon as it “wins”. That is called peeking, and it almost always produces a false result.

Wait for statistical significance

Only stop when the result is statistically significant (see below). A difference that looks large can be pure chance when the numbers are small.

What can you test?

Not every element is worth the effort. The components that move the most:

• Headlines and message. The headline is the first thing people read and often has the biggest impact. A sharper promise beats a vague title time and again.
• Call to action. The text, placement and shape of your button steer conversion directly. See call to action for what works.
• Forms. Every extra field costs conversion. Removing or combining fields is one of the most reliable wins in B2B.
• Layout and order. The order in which you show proof, offer and form determines whether people reach the button.

Button colors and micro-copy deliberately do not top the list: they rarely deliver enough to be worth a test.

Statistical significance in brief

Statistical significance is the probability that your result is real and did not arise by chance. A common threshold is 95% confidence, a core concept that VWO works out in its guide to A/B testing. Below that threshold you do not know whether B is really better or you simply got lucky.

The catch: significance depends on numbers. With ten conversions per variant almost nothing is significant, however large the difference looks. A webshop with millions of visitors reaches 95% easily. A B2B site with a few hundred visitors and a handful of leads per month does not. That brings us to the core of this article.

The problem: A/B testing demands a lot of traffic

A test is only reliable when enough people see both versions and deliver enough conversions.

Testing with too little traffic gives you the feeling of certainty without the certainty itself. You see “winners” that arise by chance and in reality mean nothing.

This is the silent pitfall in B2B CRO. A test that has to run for months to say anything, or that gives a false result, costs you more than it delivers. To get a feel for the numbers you need, run your situation through the A/B test calculator or read our deeper treatment of A/B testing on low traffic.

Good test versus bad test

What do you do on low traffic? (the B2B approach)

No or little traffic does not mean you have to guess. You simply shift from testing to researching and deciding with evidence:

• Do qualitative research. With few visitors, five to ten customer conversations often teach you more than a thousand data points. See conversion research.
• Look at behavior. Heatmaps and session recordings show where people get stuck, even without a test.
• Measure micro-conversions. Form starts, downloads and scroll depth occur more often than leads, and give a signal faster.
• Make bigger, evidence-based changes. Instead of testing a button, revise the whole message or page based on research. See value proposition.
• Stack evidence over time. Decide on a combination of signals, not on one unreliable test.

When can you A/B test?

If you have a lot of traffic on a page, for example a busy landing page from ads, then testing is perfectly possible there. In that case choose the pages with the most traffic and the highest stakes. For the rest of your site you work research-driven. You can read more about the reverse situation in when you had better not A/B test.

Test the right thing, not the easy thing

Even with enough traffic the rule holds: test what has impact. Button colors rarely deliver anything. The message, the offer and the call to action determine far more. This is exactly why B2B CRO works differently than webshop CRO.

Which tools do you use?

For tests on sites with enough traffic, teams usually use a testing tool with a visual editor and built-in statistics. When choosing, watch for three things: does the tool not slow your page down, does it calculate significance itself, and does it work with your analytics. To test without harming your findability, follow the guidelines from Google Search Central on website testing: do not use cloaking and do not leave variants in place permanently.

To work out whether a result is significant, you do not need to buy an expensive tool. Our A/B test calculator does that for free, and calculating your conversion rate gets your baseline in order before you begin.

Common mistakes

• Stopping too early. Stopping the test as soon as a variant is ahead is the most common mistake. Small samples fluctuate strongly, and today’s “winner” loses tomorrow.
• Testing without enough traffic. A test that never becomes significant gives you a random outcome with a reliable appearance.
• Changing several things at once. You then know that something won, but not what, so you learn nothing for next time.
• Optimizing the wrong metric. More clicks on a button is no win if the number of leads stays the same. Steer on the conversion that makes money.
• Detail tests over substance. Spending months on button colors while the message itself is off.

Frequently asked questions

How much traffic do I need to A/B test?

There is no fixed number, because it depends on your current conversion rate and the difference you want to be able to measure. A rule of thumb: if you have fewer than a few hundred conversions per variant per month, reliable testing becomes hard. Run your own situation through the calculator and only then choose between testing and research.

How long should an A/B test run?

At least one to two full weeks, so you include all days of the week, and always until you have reached the number you calculated in advance. Do not stop earlier because a variant is ahead: that is exactly when chance misleads you.

Is A/B testing the same as CRO?

No. A/B testing is one method within conversion optimization. CRO also includes research, analysis and evidence-based changes. On low traffic you do CRO without much testing, and in practice that often works faster.

From testing to more customers

A/B testing is not a goal in itself. The goal is more leads from your traffic, with certainty instead of guesswork. We steer on leads, not on pretty test reports. The right method, tuned to your traffic, delivers that.

That is how our approach for Get Driven delivered 400% more conversion.

Not sure whether you can test?

Tell us how much traffic you have, and we will tell you honestly whether testing makes sense or whether research delivers results faster.

We are a small team, so we move fast and do more than you expect. Plan your free intake and you will hear within 24 hours where your opportunities lie.