What you need to know about A/B testing before you start

October 7, 2019

Marketing teams that have adopted a culture of experimentation know that to provide their visitors with a quality online experience they need to base their UX decisions on reliable data, not on subjective guesswork and hunches. Adopting A/B testing will make you more agile and able to achieve this, since you can identify what really works and make the best decisions for your business.

However, A/B testing is a practice that requires a precise process and it takes a little time to familiarize yourself with its workings. For example, what conditions are needed to run a reliable A/B test? And what are the different types of A/B test you can run?

In this article, we are going to look at the basics and main principles you need to understand before you can run relevant experiments and create a successful A/B testing strategy.

1 What is an A/B test?

An A/B test is an online experiment conducted on a website, mobile app or any other online channel, to test potential improvements to an element against a control (or original) version. The test will show you which version works best for your audience, based on a statistical analysis.

The main benefits of A/B testing

A/B testing is one of the most powerful tools available to marketing teams. It enables you to optimize the overall experience on your website to nurture the relationship with your visitors and encourage them to convert. Benefits include:

Gain a better understanding of your visitors by analyzing how the different elements of your pages impact on their behavior.
Eliminate subjectivity and base your decisions on reliable data by rigorously confirming all your optimization hypotheses.
Focus your investments on what works best for all of your audience thanks to what you learn from your tests.

How does it work?

Let’s look at an example: you’re wondering whether showing the number of items still in stock would influence visitors on your product page.

To identify the best version of your product page, you can run an A/B test for which you create two versions of it. All the traffic landing on this page will be randomly and evenly distributed between both versions, without visitors noticing anything.

Once the test reaches a suitable significance level , you can analyze its performance and see which version gave the best result. Finally, you can make the change permanent by redirecting all your traffic to the winning variation and... move on to the next test.

Different variations of A/B Testing

Identifying what can be optimized on your website

What can you test with an A/B test? The answer is simple: pretty much everything. Any element of your website can be tested: entire pages, forms, pictures, text, calls-to-action (CTAs), links, banners, etc.

You can investigate all optimization possibilities that are likely to improve the user experience:

positioning, size and design of CTAs
number and ideal size of steps in conversion funnel
format and size of reassurance messages, ...

2 What are the requirements for implementing an A/B testing strategy?

A/B testing must be based on a rigorous methodology to prioritize your ideas and obtain statistically significant results.

Draw up a testing roadmap

It’s essential to plan your tests to give your optimization strategy structure. Prioritizing your actions will allow you to get the very best out of your website and organization. To draw up a testing roadmap, you must:

Decide on the goal of your tests: whether they are to increase conversions, engagement or loyalty. Setting clear expectations is vital for measuring your performance.
Decide on the KPIs of your tests: average visit duration, click rate, number of page views, for example. KPIs must be carefully chosen based on your test goals.
Prioritize your roadmap tests: to give each of your A/B tests an order of priority, you can examine different criteria such as their potential, their importance and their ease of implementation.
Communicate the results: whether before or after the test, communication with your team and other stakeholders is essential to foster continuous development.

Allocate a significant traffic volume

One key factor for reliable testing is the number of traffic volumes. We can’t give you exact numbers, but as a rule of thumb bear this in mind:

With fewer than 10,000 monthly visitors, A/B testing will only work on very specific elements (CTAs, forms) and on the busiest landing pages. In addition, tests must be run over a long period of time to yield significant results.
With 10,000 to 200,000 monthly visitors, you can run more A/B tests, but these will often take time to yield results.
With 200,000 to 1 million monthly visitors, your traffic is high enough for A/B testing. You might still have some trouble with the pages that attract fewer visitors, such as the end of the conversion funnel.
With more than 1 million monthly visitors, traffic is rarely an issue.

3 What are the different types of A/B test you can run?

There are multiple types of A/B test available depending on your goals and needs. Here’s an introduction to the various methods used by marketers:

Multivariate testing (MVT)

Sometimes, it can be useful to test several changes on one element of your page. For example, the wording and color of a CTA or the image and wording of a banner. A multivariate test enables you to test all these optimizations with a single experiment.

MVT

When should you run a multivariate test (MVT)?

When multiple variations are possible for one element of your website, multivariate testing enables you to directly identify the combination that is most successful with your visitors.
Nonetheless, multivariate testing requires very large amounts of traffic to yield real statistical significance. You’ll have as many variations as there are combinations of tested elements on your page. For more details, we have an entire article about multivariate testing for you to read.

A/B/n testing

An A/B/n test enables you to test the optimization hypotheses of a page on your website. You can, for example, test six variations of a page by doing an A/B/C/D/E/F test.

ABn testing

When should you use an A/B/n test?

A/B/n testing is useful when you want to test several versions of a page, where each version includes significant differences (general layout, design, etc.)
The difference between this and an MVT test is that MVT testing just creates variations with all the combinations of the elements you want to change on the page. If you want to radically change the layout of a page or make extensive design changes, it’s more useful to run an A/B/n test.

Dynamic traffic allocation tests (or multi-armed bandit algorithms)

With dynamic traffic allocation, an algorithm automatically and gradually redirects your visitors toward the winning variation of the test.

In a standard test, the traffic is distributed equally between each variation until the test is completed. So if two types of CTA are being tested, 50% of the visitors will be exposed to variation A, and the rest to variation B. With dynamic traffic allocation, the distribution can change depending on the performance of the element being tested. If variation B shows a high conversion rate, then the traffic distribution will be adjusted automatically. For more details, you can read our article about dynamic traffic allocation.

Dynamic traffic allocation tests in A/B Testing

When should you use dynamic traffic allocation?

Dynamic traffic allocation is relevant in two specific cases:

For tests over a limited time period: you’ll more quickly send a greater portion of your traffic to the winning variation and maximize your conversion potential.
For tests whose results you’re not tracking closely: the algorithm enables you to configure the test at the beginning and then let it run. The algorithm learning time will be longer but the traffic will continuously be directed towards the best variation.

However, to use dynamic traffic allocation, you need to have a large volume of visitors to gain significant results. It should therefore be used at strategic moments and marketers must take this into account when analyzing their results.

As this article explains, A/B testing is not a process where you can improvise: to obtain reliable results you must prioritize your tests, allocate significant traffic to them and choose the optimal method for your goals. By following these steps you’ll be able to make informed decisions and optimize your audience’s conversion and engagement.

To find out more, read our article about analyzing and interpreting your A/B tests.

ACADEMY/Return to our A/B testing training course.