Skip to main content

What is A/B testing?

Everything you need to know about A/B Testing, all in one place
what-is-ab-testing

Introduction

We live in an era of data-driven marketing, far away from the times when marketers simply made decisions based on guesswork and intuition and hoped for favorable outcomes. The modern day marketer has a scientific approach and relies on data. And A/B testing is the best way to remove uncertainty and gut feel when making marketing or design decisions for websites, ads or other digital campaigns.

Embrace the experimentation mindset.

By basing your strategy on data and A/B tests, you’ll be agile but most importantly you’ll have guaranteed feedback on what works and what doesn’t. You’ll be better placed to make sound business decisions and to invest time and money in what your visitors actually want.

experimentation-mindset-ab-testing

A/B testing to drive marketing success

75% of websites with more than 1 million monthly visitors already run A/B testing programs.

But successful A/B testing requires preparation and education, and time and effort to put into practice. You’ll have to create a process, put a framework in place, learn about statistics, set up and learn a new tool and make sure you’re actually getting accurate results. But the effort and time is worth it, given the potential to achieve your marketing goals.

To help we’ve brought together in-depth content on A/B testing, from the best blogs and experts.

ab testing

What is A/B testing (or Experimentation/Split testing)?

Definition: A/B testing is an online experiment conducted on a website, mobile application or ad, to test potential improvements in comparison to a control (or original) version. Put simply, it allows you to see which variation (version) works better for your audience based on statistical analysis.

 

A/B testing is also known as split testing, which can be either exactly the same thing as A/B testing or mean split URL testing. For a classic A/B test, the two variations are on the same URL. Alternatively with split URL testing your changed variation is on a different URL (although this is hidden from your visitors).

ab-testing-definition

What about multivariate testing (MVT)?

Sometimes, you want to test several changes on a page, for example the banner, header, description and video. To test all of these elements at the same time, you use multivariate testing (or MVT).

In this case you have multiple variants generated to test all the different combinations of these changes to determine the best one.

The big downside of multivariate testing is that it requires an enormous amount of traffic to be statistically accurate. Before starting a multivariate testing project, you need to check that your audience is high enough to provide representative results.

multivariate-testing-mvt

Dynamic traffic allocation or multi-armed bandit testing

Multi-armed bandit testing (or dynamic traffic allocation) is when your algorithm automatically and gradually redirects your audience toward the winning variation.

multi-armed-bandit-testing

A/B/n testing

A/B/n testing is when testing more 2 variations of an element or page. You could test 6 versions of a page and do an A/B/C/D/E/F test.

ab-n-testing

What are the benefits of A/B Testing?

Why should YOU do A/B Testing? Or a better question is: Are you satisfied with the way you’re exploiting your hard-earned traffic? Once you have traffic, increasing your conversions is much less expensive with great potential ROI. And with A/B Testing, it’s even greater. But it’s not all, here is a couple of other benefits:

  • Learn deeply about your audience with every test: what they like, how they react, their needs and habits.
  • Remove gut decisions from your marketing strategy by adopting an experimentation culture and testing everything.
  • Focus your time and money on what your visitors respond to best, thanks to the learnings of your A/B Tests.
     

And to give you more operational examples of questions you’ll be able to answer with A/B Testing:

  • Which elements drive sales, conversions or impact user behavior
  • Should you have long or short forms
  • Should you implement this new feature
  • Which title for your article generates more shares
  • Which steps of your conversion funnels are underperforming

How does A/B Testing work

You compare the current version (control) of a page/element against a (or more) variation of it with the changes you want to test (website page, element in a page, a CTA, picture, …).

You divide your traffic in equal portions, then they are randomly exposed to one or the other variation during a given period of time. Then, their performances (conversions, sales, …) are compared and analyzed to determine if the change(s) are worth implementing.

Getting deeper in the inner workings of A/B testing

how-ab-testing-work

Conversion Rate Optimization is key for your success with A/B Testing

What’s Conversion Rate Optimization (or CRO)?

It’s usually either buy something or “convert” and give you their contact info. In other words, get your visitors through the buyer’s journey.

How to do A/B Testing: frameworks and methodology

The success of great A/B testing is the process. It’s a science experiment, so your process has to be rigorous, with strong prioritization to focus on the most valuable tests.

Each company has its unique process but it usually resembles something like this:

1

Measure, study, analyze you website data. Identify the problems and opportunities.

3

Prioritize your tests ideas: one of the most used prioritization framework is PIE first coined by Widerfunnel.

With this framework, you rank your tests ideas on 3 criteria to determine which ones you should run first:

Potential ./10: How much room for improvement is there on this(these) page(s)?

Impact ./10: How valuable is the traffic on this(these) page(s)?

Ease (of implementation) ./10: How easy will this test be to implement on your site?

You then average the 3 and you’ll know which tests to do first. There are of course several other frameworks, try them out and make them your own.

4

Test your highest priority hypothesis.

5

Analyze your test results and learn from them

6

Communicate

7

Repeat

What to A/B test: ideas by the dozen

You can basically test everything on your website:
 

ab-testing-ideas


But sometimes, you need inspiration. So here are dozens of A/B testing ideas for you.

Note : Before we let you dive in these listicles, a small disclaimer: what worked for others might not work for you. Don’t blindly apply what you read there, make sure to analyze thoroughly if it’s relevant for you and how (if) you can adapt for your business.

Dozens of A/B tests ideas for you to get inspired

A/B best practices and … mistakes

A/B testing can be hard … and easy to mess up at the same time. So it’s good to be aware of what could go wrong and have safeguards in place. Thankfully, people have gotten into lengths on both these topics.

Make sure you set up for success by studying best practices and possible mistakes. But like with tests ideas, don’t take everything at face value. Put things to the test, see if it applies for your business.

A/B testing reporting & results

A/B testing is about making data-driven decisions AND learning. Meaning your reporting and results have the outmost importance. Be it to extract the lessons, communicate with your colleagues or get ideas for your next tests.

How to handle A/B testing results and reporting

ab-testing-reporting-results

Dive deep in A/B Testing statistics

A/B testing is based on statistical methods. You don’t need to know all the maths behind, but a little brush up in statistics won’t hurt and certainly improve the chances of your success.

There are 2 main statistical methods behind A/B testing solutions. There aren’t one better than the other, they just have different use.

Frequentist approach

Allows a simple read on the results reliability thanks to a confidence level: with a level of 95% or more, you have a 95% chance of making the right decision. But this method has a downside: it has a « fixed horizon », meaning the confidence level has no value up until the end of the test.

Bayesian approach

Provides a result probability as soon as the test starts. No need to wait until the end of the test to spot a trend and interpret the data. But this method also has prerequisites: you need to know how to read the confidence interval given to the estimations during the test. With every additional conversion, the trust in the probability of a reliable winning variant improves.

A/B testing skills & management

To put all odds in your favor, there are a number of skills and management tips you can polish. Web analytics, UX design, communicating results, are some examples.

Sharpen your skills for better A/B tests

 
ab-testing-skills-management

A/B Testing tools to maximize your chances of success

Be it project management, sample size or duration calculator, or toolkits for your process, there are many tools to help you win.

Supplement your A/B test software with these tools

What A/B Testing blogs should you read?

There are some awesome blogs with stellar A/B testing or Conversion Rate Optimization related content you can follow to learn, get inspired, and do better A/B Testing.

blogs-ab-testing

86 A/B Testing experts to follow

You might want to keep yourself in the loop as the CRO world moves quite fast. Best way to do that is to follow the most prominent experts. Here are their handle on twitter, and a list so you can follow all of them (and us of course @kameleoonAI)Twitter list: Individual Twitter handles

Lance Jones > @userhue

Jason Kincaid > @jasonkincaid

Noah Kagan > @noahkagan

Hiten Shah > @hnshah

Dave McClure > @davemcclure

Avinash Kaushik > @avinash

Daniel Gonzalez > @HiDanielG

David Kirkpatrick > @davidkonline

Shanelle Mullin > @shanelle_mullin

Steve Blank > @sgblank

Matt McGee > @mattmcgee

Rand Fishkin > @randfish

Bart Schutz > @BartS

Rick Perreault > @rickperreault

Sean Ellis > @SeanEllis

Campaign Monitor > @CampaignMonitor

Moz > @Moz

Bryan Eisenberg > @TheGrok

Shopify > @Shopify

Scott Brinker > @chiefmartec

Chris Goward > @chrisgoward

Brian Massey > @bmassey

Jeffrey Eisenberg > @JeffreyGroks

Sherice Jacob > @sherice

Carlos del Rio > @inflatemouse

Pam Moore > @PamMktgNut

Angie Schottmuller > @aschottmuller

Ryan Deiss > @ryandeiss

Ian Lurie > @portentint

ashukairy > @ayat

Khalid Saleh > @khalidh

Anne Holland > @AnneHolland55

Lincoln Murphy > @lincolnmurphy

Amy Africa > @amyafrica

Unbounce > @unbounce

Raven Tools > @RavenTools

Roger Dooley > @rogerdooley

Neil Patel > @neilpatel

Nichole Elizabeth > @NikkiElizDemere

Craig Sullivan > @OptimiseOrDie

Peep Laja > @peeplaja

Jon Henshaw > @RavenJon

Marketing Nutz > @MktgNutz

Dan Siroker > @dsiroker

Tommy Walker > @tommyismyname

Joanna Wiebe > @copyhackers

Rich Page > @richpage

Tiffany Da Silva > @bellastone

Jason Quey > @jdquey

Ton Wesseling > @tonw

Michael Aagaard > @ContentVerve

Matt Gershoff > @mgershoff

Andy Johns > @ibringtraffic

Brian Balfour > @bbalfour

Oli Gardner > @oligardner

Tim Ash > @tim_ash

Paul Rouke > @paulrouke

Linda Bustos > @edgacentlinda

Theo van der Zee > @theovdzee

MAA1 > @MAA1

Talia Wolf > @TaliaGw

Justin Rondeau > @Jtrondeau

Tyson Quick > @TysonQuick

KlientBoost > @KlientBoost

Andre Morys > @morys

Conversion.com > @conversion_com

Anna Talerico > @annatalerico

Kelly Cutler > @kfcutler

Brooks Bell > @brooksbell

Andrew Youderian > @youderian

Alhan Keser > @AlhanKeser

Conversion Sciences > @ConversionSci

Alex Birkett > @iamalexbirkett

Kaitlyn Nelson > @kaitlynelson

Kevin Hillstrom > @minethatdata

Dan Wang > @danwwang

Malachi Leopold > @livethetreplife

Aaron Orendorff > @iconiContent

Chief Conversionista > @Conversionista

Joel Harvey > @JoelJHarvey

Outsource your A/B testing with these agencies

Outsourcing your A/B testing can be a great way to still do it without the necessary resources. Here are some of the best ones out there.

Note: we sell an A/B testing tool but we also can handle your entire testing, check out our customer success page.

Let’s discuss your challenges

Talk to our experts to learn how we can help you drive greater success.