A/B test or multivariate testing: what does your B2B website really need?

Short answer: for almost every B2B website with normal traffic an A/B test is the right choice, and a multivariate test (MVT) is wasted time. Multivariate testing sounds more advanced because you test several elements at once, but you need an enormous amount of traffic to get reliable results. If you do not have that traffic, you run a test for months that never delivers anything. Below you get an honest account of when each type does or does not make sense, and why we opt for the simple A/B test in most cases.

Test it yourself: check whether the difference between your variations is statistically significant with our free A/B test calculator.

What is the difference between an A/B test and a multivariate test?

In an A/B test you pit two (or more) versions of the same page against each other, testing exactly one clear difference each time. Version A has a blue button, version B a green one. You send traffic to both and see which version produces more conversions. Simple, clear, and a fast verdict.

A multivariate test goes further: you test multiple elements at once, in every possible combination. Test three headlines, two buttons and two images and you end up with 3 x 2 x 2 = 12 variations. The advantage sounds appealing: you see not only which version wins, but also which specific element has the biggest effect and how elements interact.

The problem lies in the maths. Every variation needs enough visitors and conversions to support a reliable judgement. With 12 variations you split your traffic across 12 buckets instead of 2. That means you need twelve times as much traffic to reach a verdict you can trust.

When does multivariate testing make sense?

Multivariate testing is a beautiful instrument if you have the traffic to feed it. The rule of thumb from practice: count on at least 100,000 unique visitors per month on the page you test before a genuine multivariate test delivers reliable results.

Let us be honest: how many B2B companies do you know that reach that traffic on a single landing page? Virtually none. A typical B2B website runs on hundreds to a few thousand visitors per month on its most important pages. At that volume a multivariate test runs for months without ever pointing to a reliable winner. You get noise, not direction.

Multivariate testing is therefore something for large webshops, media sites and consumer brands with massive traffic. For a B2B company with limited traffic it is an expensive approach that sounds good in a proposal but rarely delivers. We would rather not sell you that.

Does an A/B test actually deliver enough?

Good question, and the honest answer surprises a lot of people: A/B tests do not always win either, far from it. A sizeable share of A/B tests do not improve conversion significantly, something that also becomes clear from the difference between A/B and multivariate testing. Most tests show no clear difference, or the “winner” turns out to fall within the margin of error.

Testing more variations can increase the chance that one of them really performs better. But that same traffic reality comes around the corner again: more variations also means more traffic needed per variation. For most B2B sites two to three variations is the achievable maximum.

This is exactly why we do not steer on the number of tests you run, but on what it delivers in conversion rate optimization. A dashboard full of “we ran 40 tests” is a vanity number. The question is: how many extra enquiries, demos or quotes did it produce? Also read why we steer on conversion instead of visitor numbers.

Can you still improve several elements at once?

Yes, and there is a nuance here that the simple “A/B versus MVT” story misses. In an A/B test you can also put up a fully redesigned variation in which several things have changed at once. That is called a radical or “redesign” test. You then do not know exactly which element made the difference, but you do know which version wins, and that with far less traffic than a genuine multivariate test.

Sometimes one small element makes a surprisingly big difference, by the way. In some tests a seemingly minor change, such as adding or moving a link, turns out to have a striking effect on how many pages visitors view. You only find that kind of effect by testing, but you do not need to set up a 12-variation monster for it.

For a typical B2B company our approach is therefore: start with a few targeted A/B tests on your most important pages, or with a radical redesign test if the current page is really below par. If you later have structurally high traffic, you can always look at multivariate testing then. Want to know which method suits your situation? Then read how we approach conversion optimization for B2B. Not sure whether you have enough traffic to test at all? Our explanation of A/B testing in practice will help.

Frequently asked questions about A/B testing and multivariate testing

What is faster, an A/B test or a multivariate test?

An A/B test, by far. You split your traffic across two variations instead of ten or more, so you reach a reliable verdict much faster. For most B2B sites with limited traffic that is the difference between weeks and months (or never).

How much traffic do I need for a multivariate test?

Count on at least 100,000 unique visitors per month on the page you test. If you do not reach that, a multivariate test is not worthwhile and you are better off with an A/B test or a radical redesign test.

Does an A/B test always work?

No. A large share of A/B tests do not improve conversion significantly. That is why we focus on tests that come out of genuine customer research, not on randomly tinkering with button colours.

Is multivariate testing throwing money away for B2B?

In many cases yes, because the traffic simply is not there. We only advise it if you structurally have very high traffic on the page to be tested. Otherwise you invest your time and budget better in targeted A/B tests.

Want to know what fits your site?

Which testing method makes sense depends entirely on your traffic, your goals and the state of your current pages. We give honest advice: sometimes that is a few smart A/B tests, sometimes it is better to get your whole conversion flow in order before you start testing at all. What we do not do is sell you an expensive multivariate approach your traffic cannot support. Small team, sharp choices, and steering on customers instead of vanity numbers.

Book your free intake