Multivariate testing (MVT) is an A/B testing methodology that lets you test multiple variations of an element (or more than one element) on your website at the same time during an experiment.
While multivariate testing is a very effective way to test and therefore to improve the experience and increase conversion rates on your website, you need to follow best practice to get significant, reliable and actionable results.
This blog explains what multivariate testing is, and outlines the methodology you need to follow to better engage with visitors on your website.
1 What is Multivariate Testing (MVT)?
A multivariate test is a form of A/B testing where each variation includes several changes at the same time.
The idea is to simultaneously test all of the possible combinations of multiple elements that are running on the same page, showing you which combination has the greatest impact on your conversion rates.
Here’s a simple example:
Imagine you want to change both the background color and wording of a CTA.
For the color, you want to test yellow, blue and red.
For the wording, you want to test two phrasings: “Start” and “Buy”.
Through a multivariate test you generate six versions of your page, each showing one of the six possible combinations:
A mini glossary of multivariate test terms
Multivariate test: a multivariate test enables the conversion of one or several combinations of several elements to be tested.
MVT : MVT is the acronym for Multivariate Test.
Section : a section groups together several variations of an element. For example, the “color” section, or the “wording” section.
Variation : a variation is a proposal to change an element. It varies compared to the original version. In an MVT, we can create several variations that we group together in sections. For example, the variations of the “color” section would be “blue”, “red” and “yellow”. The variations of the “wording” section would be “Buy” and “Start”.
Combination : the combination designates several variations of different sections that we combine and display at the same time to the exposed visitors. A combination would be “Buy” in blue or “Start” in red, for example.
2 What is the value of multivariate testing for your optimization strategy?
MULTIVARIATE TESTING CAN CLEARLY IDENTIFY THE ELEMENTS THAT INFLUENCE YOUR CONVERSION
Compared to a classic A/B test a multivariate test allows a greater understanding of the reasons behind performance differences between different variations.
By systematically breaking down potential changes into combinations, we can isolate the key elements on the page being tested and measure the influence of each of them.
A multivariate test lets you understand how the elements of the page interact with each other and which ones are instrumental to improving performance
Multivariate testing saves considerable time
Automatic creating and testing all potential combinations of a change will save you precious time. This is especially true when you are evaluating a large number of different factors.
If we add a change to the shape of the CTA (square or rectangle) to our previous example of three colors and two wordings, we get to 12 different variations (2 x 2 x 3) that we need to test in order to identify the variation that converts best.
In cases such as these, being able to automate creating combinations becomes a must.
3 When should you use multivariate testing?
When you don’t have a clear-cut opinion on the hypotheses to be initially tested
The first stage of designing your A/B test is to generate the different changes you want to test.
However, sometimes it can be hard to pinpoint the individual element that will make the key difference. For example, if your CTA is not performing as expected is the problem the wording, the color, or the shape? It’s not easy to be sure enough to create a strong hypothesis right from the start.
In this case, a multivariate test is essential. You can test several elements at the same time, and let your testing platform generate all the possible combinations for you. This means you don’t rule out any possibilities, and you’ll save considerable time thanks to automation.
When you’re working from an analytical perspective
As we have said, a multivariate test can refine the analysis of your results by isolating the different components within the changes being tested.
If you’re working from an analytical perspective where the interpretation and understanding of your A/B test results are key, then multivariate testing will be a good ally.
The question of minimum traffic
The amount of traffic on the page being tested needs to be considered before you start testing. Essentially, the higher the number of combinations, the more traffic you will need.
Generally speaking, you’ll obtain an acceptable reliability rate in an A/B test from:
- 1,000 visits per variation if the page being tested has a high conversion rate (such as engagement on a landing page or click-through rate for a CTA button), and where the expected potential gain is high (for example, +10% or 20%)
- 7,500 visits/variations if the conversion rate is low (<5%), for example in forms.
Reaching 7,500 visits per variation takes time and can be complicated. And of course the more we multiply the number of variations (which happens quickly with multivariate testing), the higher we need traffic to be in order to reach the critical threshold of number of conversions by variation.
Essentially, as the nature of a multivariate test is combinatorial, it is highly recommended that you select a finite and relatively low number of variables to change. There’s no point in using multiple variations if it generates results that are meaningless because the visitor sample size was not large enough to make them valid.
4 How does multivariate testing work?
1. Create your variations
As with a classic A/B test, the first stage is to create the variations you want to test.
Let’s return to our example in which you want to change both the background color and wording of a CTA.
It’s very simple:
In your A/B testing platform, such as Kameleoon, you create the sections of your test; here these are “Color” and “Wording”.
Then, for each section, you build the variations you want. For example, in the “Color” section, you’d create a “yellow” variation, a “blue variation and a “red” variation. In the “Wording” section, a “Start” variation and a “Buy” variation.
The platform will then automatically generate all the possible combinations of your different section variations.
2. ALLOCATE TRAFFIC FOR YOUR MULTIVARIATE TEST
Now you have to decide how to allocate website traffic to the different combinations for the duration of the test.
There are two approaches for allocating your traffic: by section or by combination.
Traffic allocation by section
In our example, we have the Color and Wording sections.
The Color variations are “yellow”, “blue” and “red”.
The Wording variations are “Buy” and “Start”.
By default, the traffic will be distributed equally among all these six variations. Allocation by section is the “simplest”, since you have fewer sections than combinations. Here, you will have to distribute 100% of your traffic between Yellow, Blue and Red, and 100% of your traffic between Buy and Start.
The A/B testing platform will then automatically allocate traffic to each of the combinations.
Traffic allocation by combination
In our example, six combinations are possible:
“Start” in blue
“Start” in yellow
“Start” in red
“Buy” in blue
“Buy” in yellow
“Buy” in red
You can therefore decide manually how to allocate your traffic to each of the combinations, or, simply split it equally between all of your variations.
This traffic allocation method can become difficult to manage if you have many different variations since the number of combinations increases exponentially.
However, it is useful in certain specific situations, particularly when you want to exclude certain combinations from your test.
Here’s an example: you are testing changing the background color and the text color in a CTA. You don’t want your visitors to see the combination of black background/black text (because they would see nothing). So if you are setting traffic allocation by combination you could manually allocate zero traffic to go to the black background/black text combination.
3. Set up the KPIs to monitor
Once you have set the traffic allocation, you can create the goals to be measured during your test.
Again, this follows the same approach as for a classic test. You will find all available information on setting up your reporting tools and your goals in our article on this subject.
4. SIMULATE THE MULTIVARIATE TEST BEFORE LAUNCHING IT
You can then view all the combinations you are testing in simulation mode. This enables you to check you are happy with your choices and they work seamlessly before putting the test live online.
5. Monitor the results
On your results page each combination is considered as a separate variation, in the same way as in a classic A/B test.
You can therefore analyze, in detail, the level of impact each element has on your main KPIs, and ensure that you are making the best informed data-driven decisions when it comes to changing your page.
In conclusion, multivariate testing is a very powerful experimentation method to use, provided you have a sufficient level of traffic. Be aware, though, that because of the number of variations involved it can take a relatively large amount of time to obtain reliable results.
So while the use of MVTs should not be systematic, they can prove very useful in certain situations, helping you test a wider number of elements and get systematic results.