Skip to main content
7 steps to start A/B testing your Shopify Plus store

7 steps to start A/B testing your Shopify Plus store

January 12, 2022
Reading time: 
15 mins

The biggest mistake Shopify stores make is assuming people will react to their website the way they imagine.

In reality, the best e-commerce experimentation teams often stumble on huge uplift-inducing changes by accident while the majority of their A/B tests "fail". However, these accidents are by design; the failures are valuable.

Top A/B testing teams use frameworks for testing often and testing strategically. They learn from all tests, even those where variables don't improve metrics. They understand there's a lot they don't know and the only way to understand customers and achieve conversion rate optimization are with a high volume of testing across the whole customer experience.

A robust A/B test framework and methodology can help your Shopify store:

  • Reduce shopping cart abandonment
  • Maximize the value of your acquisition campaigns
  • Increase sales and the average order value (AOV) per customer
  • Maximize seasonal/holiday revenue using promotions
  • Align company messaging with customers’ values


Shopify recognizes the value of A/B testing and has native integrations with dedicated A/B testing solutions like Kameleoon to make experimentation more accessible and impactful.

After reading this article, you will be ready to run your first Shopify Plus A/B test.

If you already do A/B testing, it will also show you how to increase the variety and complexity of your testing—the two most significant factors in generating high ROI on CRO—using Kameleoon.

Kameleoon is an advanced client and full-stack A/B testing and personalization tool with a native integration with Shopify Plus. Its unlimited, flicker-free A/B/n testing, AI personalization and real-time data reporting help mid-size and enterprise brands create world-class online shopping experiences for their customers. Find out how you can increase revenue and loyalty for your Shopify store by booking a demo today.


How to Run Your First Shopify A/B test

Running your first Shopify test requires a testing process you can refine and scale over time. It starts with understanding the resources you have to work with (mainly, sample size) then determining the problem and the solutions you plan to experiment with.

1 Understand your top-level data. Specifically, calculate your sample size

The biggest hurdle to running an A/B testing program in your Shopify store is your monthly traffic.

Therefore, calculating the sample size you need to run split tests is the first step. You can use the sample size calculator in the Kameleoon App, pictured below under Part 6 of this guide, or this calculator from Evan Miller, to determine how many visitors you need for both variations in your test.

While your website visitor numbers may cross the 10k+ monthly threshold, the traffic on the specific page you want to A/B test matters. A lack of sufficient traffic to a page may increase the duration of your A/B test. It may be more sensible not to run an A/B test on a low traffic page and instead focus on surveys and focus groups.

2 Get granular. Determine your business problem and create hypotheses

Once you’ve determined a page is getting enough traffic to run a test within your goals and time constraints, define the business problem your test intends to solve.

Maybe you found cart abandonment issues on a high-traffic product page. Maybe your customers are not following the recommendations on the recommended products section of your homepage.

Study your store's data to identify issues you could solve with A/B testing on your Shopify website.

However, don't rely entirely on data and ignore qualitative findings.

Matt Abbott, Head of Growth at Swanky says that, sometimes, “you can get clouded by the data and, because Shopify stores have a reasonably consistent architecture, it’s easy to underestimate the impact clarity of messaging and design has on the user experience. If your products are right and your customers have a high determination to purchase, a customer will struggle through a terrible user journey."

The biggest gains will be secured through helping those customers that are unsure to be confident and progress through the journey as unencumbered as possible through finding the best user experience from a series of test iterations on copy tone, imagery and page layout.
matt abbott
Matt Abbot
Head of Growth, Swanky

After identifying the issue, confirm what visitors are doing when they incounter that part of their customer journey and form a hypothesis.

Combine user experience platforms like Hotjar and analytics tools like Google Analytics to investigate user behavior. These tools will show you how customers behave when navigating your site. 

Create several hypotheses based on the user behavior you observe. What we mean by hypothesis is a reason for the cause of that issue. Read our article on optimizing customer experience with user data for examples of user insights that can inform hypotheses. 

For example, “Our customers aren’t adding products to their basket because the CTA is indistinguishable from the rest of the page." Or, "Our customers aren’t adding products to their basket because there isn’t enough delivery information.”

How to determine your business problem and create hypotheses for your A/B tests

Furthermore, data analysis might uncover additional issues that impact your conversion rate. For example, after analyzing visitor session recordings, you may realize that 8 in 10 visitors ignore your product recommendation section entirely.

Further analysis may rule out other factors, like device type, influencing customers’ actions. You can now make a hypothesis that a lack of personalization for each customer is why they’re likely ignoring the product recommendation section in your Shopify store. Maybe personalizing product recommendations based on prior visits or geography will increase the number of customers clicking and buying these items. Or maybe, changing the price. 

3 Decide which type of test to run

Your hypothesis will determine the type of test you need to run.

Take the second example above, where your hypothesis is that a more personalized experience will increase the number of purchases made from the product recommendation section. The Shopify A/B test you run will involve designing a variation where your product and personalization algorithms work together to deliver a better customer experience.

In another case, your hypothesis may be that removing the “add to cart” button would create a faster shopping experience. This could be because your customers typically purchase 1 item per visit. In A/B testing in your Shopify store, you can create a variation where the “add to cart” button is absent. Test it against your control where the button is present to see if it makes a difference to customer experience.

How to decide which type of test to run

“From experience, don’t design the alternative variation with a fixed mindset on what the solution will look like. Set the challenge to your design team and if they come back with three alternatives, test all three against the control (if traffic levels allow)," said Matt Abbot.

Challenging brand guidelines can always be tricky, but with A/B testing, the customers are deciding what they prefer, not the company’s brand team.
Matt Abbott
Head of Growth, Swanky

4 Determine the segments, triggers and data you will use

Different experiments will target different segments of users, e.g. returning visitors or users who saw your main product page. If you have users from throughout the Anglosphere, you can target British customers with UK English, similar to what Matt Abbott dis in an experiment described below. Creating segments of your audience enables you to show your experiment to only visitors who fit your criteria and exclude those who do not.

Determining which segment of your audience is right for your hypothesis is important before you run A/B tests in your store. Segmentation ensures you show the right experiment to the right customers.

How to determine the segments, triggers and data you will use


You can target your segments further by including/excluding them based on pages they visit. For example, you can target only the French-speaking segment of your audience who visit your product pages in an experiment. Visitors from the French-speaking segment of your audience who do not visit any product page will not see the experiment. Using segments and targeting conditions will give you data that is invaluable in helping you to make decisions for your business.

For Shopify, as a platform that facilitates easy expansion into new markets, getting the language right is going to become increasingly popular across our split testing strategies for international merchants.
Matt Abbott
Head of Growth, Swanky

"Not only will it be important to ensure you are getting the language and tone right for your international stores, you might be surprised about what different markets react to," said Matt.

"I’ll never forget the results of a British English vs. American English test on an American storefront for a British company. The (sensible) hypothesis was that American spellings would be received better by the American market, but we were wrong, British spellings generated 13% more conversions. Proving that it’s good to test everything.”

5 Finalize your experiment

If using Kameleoon, there are two ways to finalize and run your A/B test.

You can login via your Shopify store, where you have installed the Kameleoon script, or you can login via the Kameleoon App. Both ways lead you to a page where you can create your experiment.

On the homepage, click on “New Experiment”. Then, click “Create New Classic A/B” experiment. The Kameleoon App automatically creates two variations when you create an A/B experiment—the control and your variation.

You can change elements in your variation to make it different from the control page. These changes could be things like removing the “add to cart” button, incorporating better personalization, or something more.

Now you can finalize your experiment. Add the variation you want to include in the experiment in the traffic allocation section and choose which segment of your customers you want to target with your experiment. By default, Kameleoon splits the traffic on the testing page between your control and variation, but you can be more specific by choosing segments of your visitors for your experiment.

Simulate your experiment to check that all parameters are working correctly. Then launch your experiment.

6 Set the duration of your test

Ideally, your A/B test should run for 2-3 weeks.

Depending on your traffic, the duration of your experiment may be shorter or longer than this ideal.

How to set the duration of your test

Leaving your experiment to run for longer will account for customers’ evolving behaviors. There will be days where your traffic is normal and a good representation of your customers, then there will be other days where external factors like payday, holiday, weather changes, and major world events may spike or tank your traffic.

If you were running an experiment in March 2020 when the World Health Organization (WHO) announced the pandemic, you may have seen a traffic spike if your store-sold groceries or medical essentials. Conversely, your traffic may have tanked if your store sold luxury items, as customers focused on essential products.

Letting your A/B test in your Shopify site run for a proper amount of time will help balance out the highs and lows of traffic.

Having said that, running your experiment for too long will contaminate the data. A/B tests that run past 30 days will pollute your data as there is no way for your Shopify A/B testing app to determine if a visitor has seen your test in the past.

Tools like Kameleoon use cookies to divide your visitors into segments and track which variation of your experiment they saw. These cookies expire after 30 days.

7 Learn from your insights

The major benefit of a well structured A/B testing programme in your Shopify store is that every experiment can achieve something.

While it will be amazing for all your hypotheses to be true and your experiments to improve conversions, this rarely happens.

Most A/B tests will not result in breakthroughs. In fact, as many as 7/10 A/B tests “lose” from a commercial point of view. This is why you test—to find what works for your customer base and vice versa. A better way to look at this is: imagine if you hadn’t tested your ideas in the first place? You would have implemented many new changes that hurt how customers use your store.

What to learn from your A/B tests insights

If your variation beats the control at the end of your experiment, you can start rolling out the changes to your audience.

In Kameleoon, you can gradually roll out your recent changes to your visitors. This way, you can track customer behavior to new features. And, if complex conditions arise, you can easily roll back new features.

If your control beats your variation, analyze the insights you gathered from your A/B tests. Store these insights and use them to inform future experiments.

If you believe a test is so successful that you wish to roll it out to 100% of traffic, you can do so with one click in Kameleoon. Often, this gives your dev teams time to code the changes in the backend.


How to scale your Shopify Plus A/B testing with Kameleoon

Kameleoon is the number one enterprise-level solution for Shopify Plus stores. It provides an omni-channel, contextual, profile-based, rules-based, and algorithmic targeting, plus product/content recommendation and testing capabilities for customers.

Many A/B testing tools falsely claim to have an integration with Shopify or Shopify Plus. What they actually have is a script for ecommerce stores to install on their Shopify site, but these scripts tend to be buggy and may pollute the data you get from Shopify A/B testing.

Kameleoon’s Shopify app is 100% approved and listed on Shopify Plus’s app marketplace. It makes scaling A/B testing and personalization for your customers easy, without the need to involve multiple teams or dedicating tons of development resources.

One of the biggest problems Shopify owners have with testing is that they’re not very technical and can’t do split URL tests. They are also too worried to delve into the liquid files on their store. The Shopify Plus Kameleoon App does the entire process for you. There’s no need for technical knowledge, and it enables you to start testing. Further on, you can also do advanced tests using Kameleoon and inject code.
Frederic de Todaro
Chief Technology Officer, Kameleoon

Set up Kameleoon with your Shopify Plus Store

You can set up Kameleoon directly from your Shopify Plus app.

  1. Click “Apps” in the left-hand corner and then customize your store.
  2. Search for “Kameleoon”
  3. Add your Kameleoon site code in the settings portion of the app.


How to set up Kameleoon with your Shopify Plus Store


The Kameleoon script will be automatically added to all pages of your Shopify store.

The tool will then automatically track the main e-commerce actions that take place in your store—e.g. transactions, product views, access to cart page, and collection views.

Use Kameleoon’s Visual Editor to build tests

Creating variations using the visual editor is quick, convenient, and flexible.

Kameleoon’s visual editor allows you to edit, delete and rearrange elements on a page without involving a developer. You can set up A/B tests with ease to find answers to pressing questions.

In the example below, we’ll perform a common checkout optimization.

You may have noticed that customers who use the “Add to Cart” button rarely buy the product. From that insight, you want to test the hypothesis that removing the “Add to Cart” button will increase purchases. Here’s how you can build this test.

  1. Log in to your Kameleoon App.
  2. Click on “New Experiment”.
  3. Choose the “Classic A/B” option to launch the visual editor.


How to use Kameleoon visual editor to build tests

You can start editing variations immediately as Kameleoon automatically creates two versions for you. Head over to the Add to Cart button on your product page, select the button and hide it by clicking the options on the left-hand side.

  1. Allocate traffic to your variations.
  2. Choose segments and goals for your experiment.
  3. Simulate your experiment to be sure that all parameters are correct.
  4. Launch your experiment.

Do experimentation with the Code Editor

Kameleoon’s code editor offers you the means to get granular with your experimentation.

You can roll out experiments by editing the JavaScript and CSS code directly in your variations. Kameleoon’s Code Editor lets you push out and test advanced modifications in your Shopify A/B tests.

To access the Code Editor, simply log into Kameleoon, and click on “New Experiment”.

How to do experimentation with the code editor

Then, select the “Developer A/B” option. This will take you into the Code Editor, where you can design variations and launch your experiments.

Identify elements that influence conversions with multivariate testing

With Kameleoon’s multivariate testing, you can narrow down which elements of your website influence conversions.

Testing different combinations of website elements will help you find elements driving conversions and work to improve those areas of your website that aren’t working so well. In Kameleoon, login and click on “New Experiment”. Then click on “Multivariate (MVT)”. Create your sections for different elements like button color and CTA wording. Then create the different variations for each section. Allocate your traffic, simulate, then launch your experiment.

Create personalized experiences based on users’ behavior, device, and location

Kameleoon’s platform allows you to create personalized experiences for different segments of your audience.

Let’s say your Shopify store has visitors from the UK, US, Canada and France. You can create personalized experiences for visitors from each of those countries.

For example, you can create a banner for American visitors offering discounts in celebration of holidays like 4th of July and Labor Day.

In the same vein, you can create personalized experiences based on users’ behaviors and devices.

Use AI to scale your Shopify A/B testing and personalization

Kameleoon’s platform uses predictive algorithms to identify visitors’ intent and interest in different products to enable you to personalize their experiences.

Kameleoon Conversion Score™ (KCS) uses a scale of 0 - 100 to show a visitor’s probability of converting.

How to use AI to scale your Shopify A/B testing and personalization

You can also personalize a visitor’s experience on the website after 15 seconds based on behavioral signals like product categories viewed, time of day, weather, or price sensitivity.

For example, you can reorganize the homepage based on a visitor’s KCS, allowing you to scale personalization for millions of visitors without involving different teams in your company or worrying about endless maintenance.

Example: How The Wine Collective increased revenue with Shopify A/B testing

The Wine Collective is an online store on the Shopify Plus platform that offers premium wines to consumers.

The company wanted to understand their visitors better and provide a world-class customer experience worthy of their high-end wines. Optimizing conversions for better revenue was another goal they wanted to achieve.

How The Wine Collective increased revenue with Shopify A/B testing


Larissa Enbridge, CRO Lead at The Wine Collective, turned to Kameleoon as her Shopify A/B testing app of choice to help drive these business goals.

Using Kameleoon, The Wine Collective ran experiments that increased their conversion rate by 5.3 percent. Understanding their customers and providing better experience through insights from testing with Kameleoon led to an 18 percent decrease in bounce rate.

To find out if your store can increase conversion rates and provide better customer experience with an advanced experimentation tool like Kameleoon, book a demo call

Kameleoon is an advanced client and full stack A/B testing and personalization tool with Shopify Plus integration to make customer experience optimization easy for eCommerce stores. Its unlimited, flicker-free A/B/n testing, AI personalization and real-time data reporting help mid-size and enterprise brands create world-class online shopping experiences.

Topics covered by this article