5 ways to increase ROI on your Magento A/B testing program

According to data from BuiltWith, Magento is used by 12% of all online retail stores.

Being more enterprise-focused than self-hosted alternatives like Shopify and WooCommerce, it attracts businesses who are more likely to understand the basics of A/B testing and know the value of experimentation in growing revenue.

However, the problem with many advanced teams is that they mistakenly believe that running occasional tests for front-end elements equates to having a robust experimentation program.

This off-and-on-approach causes a vicious cycle of deprioritizing A/B testing, where low ROI reinforces the infrequent testing and the organization's inability to go beyond the most basic conversion rate optimization techniques.

It’s no wonder that Shopify has several native integrations with enterprise A/B testing tools, including Kameleoon. While Magento has an extensions marketplace, stores on the platform may have trouble finding robust and advanced A/B testing extensions. This may prevent you from scaling your A/B testing and increasing their ROI.

You can mitigate this by installing experimentation tools like Kameleoon using a script and implementing some of the advanced A/B testing techniques and strategies listed below.

Chosen because they improve customer experience when implemented consistently, these strategies will make people want to shop at your eCommerce store.

Advanced ways to improve your Magento A/B testing

To improve your Magento A/B testing program and increase ROI, the way you run experiments needs to change.

You need practical processes that are the next step in your optimization journey. These techniques build on the foundation of your current program. They provide you with actionable steps to improve your A/B testing program and increase ROI with every experiment you run.

1Test more elements per experiment

A direct way to increase ROI in your Magento split testing program is to bundle more tested elements per experiment.

If your usual approach is to test a new element in an experiment. And then refine its placement, color, and other aspects with more experiments after it beats your control. You will run into an ROI wall as you’d be investing more per experiment for lifts.

Running a multivariate test where you test your new element, its placement and color in one experiment, is more effective, although resource intensive.

For example, if you wanted to test adding an “alert me” button to out-of-stock product pages in your Magento store. Instead of A/B testing color and placement of the new button in the future, you can bundle adding the button, its placement and color into one MVT to achieve results faster.

Many Magento store owners shy away from using MVTs because they:

• Believe they're too complex to run
• Do not understand how to incorporate MVTs into an overall testing program
• Think they're too resource and traffic intensive

Because most Magento stores receive hundreds of thousands to millions of visitors monthly, the traffic needed for running MVTs should not be a problem. Treating MVTs like A/B tests often leads to frustrations for many optimizers.

Use MVTs to see how different elements of a webpage interact with each other. And find the best combinations for maximum conversions.

Say, you’re running a campaign where you’re funneling visitors from your newsletter to a landing page in your store. An MVT will be the best way to test all elements of your landing page and inform design for future landing pages.

Once you’ve mastered testing more elements per experiment, the next stage is to group experiments based on the concepts behind them.

Grouping similar experiments help you increase your testing velocity and ROI. For example, you can group together experiments focused on product page improvements. This way, all your resources and time are focused on multiple experiments in a particular area that will bring results.

2 Move from iterative to innovative testing

In experimentation, “innovative testing” is not a buzzword, but a set of practices that combine in-depth user understanding with insights from prior experiments to produce and test bold design changes that drive significant lifts.

By contrast, iterative testing is what most people imagine when they think of A/B testing.

1. Run a test
2. Build on the results of Experiment 1
3. Build on the results of Experiment 2
4. Build on the results of Experiment 3 & so on.

For example, you notice that conversion rates for users who add to cart on your site is low. You could hypothesize that visitors are using the cart as a place to save products for the future. You could test adding a “wishlist” button to see if users will adopt that. If adoption is low, you can build on that in future experiments to improve adoption of your new feature.

Iterative testing is a backbone of a basic experimentation because it allows you to build on a foundation of data and insights from previous experiments.

In a typical iterative experiment, you use both quantitative and qualitative research to form a hypothesis. Then test that hypothesis. And use the results from that experiment to inform the next experiment.

However, iterative changes in your Magento store will lead to an increase in ROI. But this increase stalls as you reach the local maximum in your A/B testing program.

With a solid background in iterative testing, you move on to more complex experiments to grow your business.

Innovative testing takes the form of testing your value propositions, reducing the steps within a workflow or even a page redesign. It takes more technical know-how and time to do innovative tests. But they pay off in bigger ROI and better growth.

Innovative testing requires qualitative data and an accurate understanding of your customers. This ensures that your tests in your store pay off.

3 Adopt a sequential testing method in your experiments

In typical A/B testing, you only stop experiments when you reach statistical significance, with one of your treatments being the clear winner.

This practice is inefficient, as it exposes your customers to negative treatments which can cause conversions to fall. And lower your ROI in the long run as customers

To get the best out of your Magento A/B testing, you need to adopt sequential testing. This way, you can monitor the data from your experiments at various intervals and stop your experiments early.

Stopping experiments can improve your ROI. If your experiments are showing positive trends at your interval analysis, you can ship the treatment and increase your conversions. And then move onto designing a new experiment.

But if your treatment is tending towards negative trends, stopping early shields your customers from a negative experience. It saves you experiment costs. And you can divert resources to other promising or new experiments.

4 Run more multi-arm bandit experiments

An issue with A/B testing in your Magento store is the "Learn - Earn" problem.

First, you learn via gathering data and A/B testing. Then you earn by rolling out treatment to your customer base. The problem with this approach is that during the learning phase, you spend time and resources sending segments of your customers to less successful treatments. Once the experiment concludes, you then send all your customers to the best treatment.

The problem with this is that during the learning phase, you’re exposing some audience segments to unsuccessful treatments. Negative experience in your store will lower your conversion rate. And with resources spent on negative variants, your ROI will drop significantly.

A way to mitigate the ‘Learn - Earn’ problem in your store is to run multi-arm bandit (MAB) experiments, or dynamic traffic allocation.

Dynamic traffic allocation is useful when:

• You need to test seasonal or holiday campaigns. You can test messaging and other elements whilst sending traffic to the best performing treatment. Thus, learning and earning.
• You’re continuously testing multiple elements. It enables you to automate part of the optimization process. This is because you will not need to analyze experiments

MAB experiments allow you to mitigate losses from the ‘Learn -Earn’ problem. You send customers to the better performing treatment and shield them from negative variants. Thus, your conversion rate, and in turn revenue, doesn’t drop.

Since you send customers to better treatments, you can focus your resources there. With automation, this process becomes a lot easier. And you can get better ROI from every experiment you run in your Magento store.

5 Combine A/B testing with personalization

You can marry A/B testing and personalization to grow your revenue and boost ROI in your Magento split testing program.

Millions of customers who visit your site expect a personalized experience regardless of their location, device types and time of day. A/B testing elements and serving the same version of your website to every visitor will cause your conversions and ROI to stall.

To do personalization, you need to segment your users based on:

• Visitor characteristics like geographical location, browser language, device type, etc.
• Acquisition channels (include/exclude visitors based on referring websites)
• Behavior on your site like the total number of pages visited.

Using geographical location as an example, you can head into your analytics software to see if segmenting with this characteristic works for your store. If your store is receiving visitors from 3 different countries, you can segment by location.

Head into your A/B testing tool to create new segments. Name your segment and choose ‘Geolocation’ location as a condition. Narrow or expand your condition. And then hit ‘Create’.

Now you’re ready to run A/B tests on personalizations for this audience segment you created.

Run experiments to find the right headline, image, value proposition, and products for each of your audience segments and increase your conversions.

You can incorporate rules-based personalization into your A/B test to further refine your targeting. And you can scale A/B testing plus personalization using a predictive personalization solution, like Kameleoon’s personalization platform, to serve customers' unique experiences based on their behavior in real time.

This ensures that customers have the best experience every time they visit your Magento store. Thus increasing your revenue and ROI.

Using Kameleoon to increase ROI on your Magento A/B testing program

Kameleoon is an advanced client and full-stack A/B testing and personalization tool. Its unlimited, flicker-free A/B/n testing, AI personalization and real-time data reporting help mid-size and enterprise companies create world-class online shopping experiences for their customers.

It provides contextual, profile-based, omni-channel, rules-based and algorithmic targeting, plus content/product recommendation and testing capabilities for customers.

You can easily connect Kameleoon with your Magento store and start optimizing customer experience and increasing ROI from your A/B testing program.

Install the Kameleoon script in your Magento store

The first step to testing with Kameleoon is to install the Kameleoon script in your Magento store.

Log into your Kameleoon App, click on ‘Sites’ under the ‘Administrate’ button on the left-hand menu. It will show the URL of the website you entered when you signed up. Click on the website card.

Click on ‘Setup’. Follow the prompts on the page to complete your installation. Kameleoon has recommended settings for you to follow in installing your unique script:

• Adding the script to the <head> of your website
• Not using tag manager to install your script. Tag manager as it increases flickering
• Using the installation tag ‘Asynchronous loading with Anti-Flicker’

Run Tests with Kameleoon’s Code Editor and dedicated developer tools

To run your first experiment, head into your Kameleoon dashboard and click on ‘New Experiment’.

Choose "In the Code Editor" from the pop-up.

Choose "JS/CSS" to use your custom code to create an experiment.

Design your variation. You can also include or exclude the original page from your modifications using the button at the top right corner of your screen.

To help you design your variation, Kameleoon offers:

• Syntax error detection to avoid long delays and other errors
• CSS color selector to help you choose colors directly on the dashboard without using external tools
• JavaScript autocomplete to help you pick your desired parameter, variable or method

After designing your variation, you need to choose the traffic distribution either manually or let Kameleoon automatically allocate traffic for you. Then select your targeting either by webpage or previously designated segments and goals.

Now you can simulate your experiment to confirm that everything works the way you intended. You can now publish your experiment or schedule it to run at a later date.

Perform Experiments Server-Side

To create a server-side experiment, head into your dashboard. Select "New Experiment".

Choose "In the Code Editor" from the pop-up. Then click on "SDK" to create your experiment.

Enter the required information and hit “Create.” The code editor will launch and you can start creating your experiment. You can choose to include or exclude changes you made to your variation into your control. Simply check/uncheck the ‘Also add JSON data on the original’.

Finalize your experiment by choosing your traffic distribution. Kameleoon offers dynamic traffic allocation for your experiment. You can turn this on during the finalization stage.

Set your goals and targeting segments. You can choose to use a client-side segment in your server-side experiment by copying it and making it server-side. This keeps the segment both on the server- and client-sides.

Launch your experiment.

Use Kameleoon’s advanced targeting and segmentation capabilities

Kameleoon offers advanced targeting and segmentation to help you serve the right tests and personalization to the right audience.

The 3 advanced targeting Kameleoon offers are:

• Custom data
• Acquisition channel
• Key pages

To create new advanced targeting, click on ‘Configure’ on the left menu in your dashboard. Then click on ‘Advanced targeting tools’. And click on the ‘New’ button at the top right of your screen.

Then fill the fields in the menu that pop-ups up.

You can choose any of the 3 targeting options depending on your needs. Click on ‘Next’ and follow the prompts to complete your advanced targeting segments.

Run Multi-arm bandit test and dynamically allocate traffic

Create your experiment as outlined above. To turn your A/B test into a multi-arm bandit test, turn on dynamic traffic allocation by clicking the following button in the ‘Traffic distribution’ stage of finalization:

Kameleoon will dynamically distribute traffic to more successful variations, thus helping you limit the opportunity cost of lost conversions in low-performing treatments.

Request a demo to see this feature and the Kameleoon platform in action.

Questions? Curious? We'd love to hear from you. Please reach us at [email protected]