How to Avoid the Flicker Effect in AB Testing

A/B testing performance is important to improving user experience and boosting revenue on your site. 

One thing that can hurt your testing performance is the flicker effect. The flicker effect, also known as Flash of Original Content (FOOC), affects the experience of your visitors and your A/B testing data. 

In this article, you will learn:

• What the flicker effect is

• What causes it

• The impacts it has on your A/B testing program

• And how to overcome it in your experimentation

What is the Flicker Effect?

The flicker effect occurs when the original page appears for a second before the new variation is shown in an A/B test. Your visitors see a flash of your control page before the variation loads for them.

Image

Did you see the website change? That's the flicker effect. You caught a glimpse of the control page before the variation loaded. 

Why is flickering a big deal if it only happens for a second? 

That sliver of time is more than enough for the human brain to process what exactly it is seeing. 

In 2014, a group of MIT scientists found that it takes the human brain only 13 milliseconds to process images. Even though the flicker occurs for 1000 milliseconds (1 second), your visitors already saw the control page before the new variation. 

The flicker effect impacts the user experience of your visitors and the results of your test. Some users may leave the site while the response of those who stay may not be trusted. This is because of the Hawthorne Effect as people behave differently when they know they’re being watched. 

Causes of the Flicker Effect

Essentially, there are 3 reasons why visitors may be experiencing the flicker effect on your website:

1 How your A/B testing tool script is installed

The way your A/B testing tool script is installed may be causing the flicker effect for your users.

The position of your testing script on a page can cause flickering. When your script is at the bottom of the page, it will only be rendered after the other scripts above it have been loaded. Your entire page will have loaded before the testing script renders which will cause your users to experience the flicker effect. 

Another way installing your A/B testing tool script can cause the flicker effect is indirect installation. If you do not have the technical skills to install the script directly in the header of the pages on your site, you may opt for indirect installation via a tag manager. 

While tag managers make it easier to handle installation, they may cause flickering as they load scripts asynchronously. When a visitor goes to your website, all the scripts on your site will load all at once. Some scripts will load faster than others. If your testing script doesn’t load first, your visitors will see a content flicker when it does. Kameleoon negates all these problems by having one of the fastest code snippets on the market that is 100% flicker free. 

THE TYPE OF A/B TESTING SCRIPT

A/B testing tools offer scripts that either load synchronously or asynchronously. The type of script your tool offers can cause flickering. 

When your A/B testing script loads synchronously, nothing else loads on the web page until the testing script has been executed. Your server treats the experiment script’s instruction as primary - it will be rendered first before any other elements on your web page. 

While synchronous A/B testing scripts prevent flickering, they have the problem of displaying a blank page until the script is fully rendered. This becomes a problem when your web server refuses to respond and the page your visitors see is permanently blank. But there are ways to prevent the blank page as we will discuss below. 

Asynchronous testing scripts load at the same time as the other elements on your web page. This allows them to load fast and prevents the blank page as different elements on the page appear. But asynchronous scripts are prone to flickering. 

A big issue with asynchronous loading is that there is no guarantee that your testing script will be executed before other page elements. These elements that may be part of your A/B test may be rendered before your testing script. After your testing script loads, the elements already on the page will flicker as they are replaced by the variant versions.

If your A/B testing tool offers asynchronous loading or asynchronous scripts, your visitors may experience content flickers during your experiments. 

THE TECHNOLOGY USED BY YOUR A/B TESTING TOOL PROVIDER

Even if you directly install your testing script at the top of each page on your site, your visitors may still experience the flicker effect. 

The technology used by your A/B testing software provider could cause flickering when you run experiments. When you use an A/B testing tool, it provides a script that you install at the top of your page. If that script is large, it can cause flickering when it loads your tests as it will take time to fully render. 

Some A/B testing tools may automatically include tests that have ended or tests that you are still working on in their script. This considerably increases the size of the script and leads to flickering as other elements are loaded before it. 

The way the testing software implements your modifications may contribute to the size of the script. Some tools add unnecessary lines of code to achieve your modifications especially if you’re using a visual editor. If you’re not very technical, you may not realize there are additional lines of unnecessary code that are adding to the size of the script. 

2 Impact of the Flicker Effect on A/B Testing
 

The flicker effect impacts the user experience for your visitors. 

When a visitor lands on a page on your site and is bucketed into an experiment, it can be jarring for them to see your control before the variation loads. Your visitor may think there is a problem with the page they’re viewing and leave.

Say you’re running an experiment where you’re removing the “add to cart” button and replacing it with a “buy now” button on your product pages. When a visitor, who is bucketed into your new variation group, lands on this page and sees the control before the variation loads.

It can be disconcerting for your visitors as they saw something different before the variation loaded. They may leave the site or keep refreshing the page, thinking that something is wrong. 

The flicker effect can hurt your conversion rate when you implement changes based on deceptive data. 

While visitors may leave after seeing a flicker, not all of them do. Some visitors may still go on to browse your site, make purchases and leave. The data for these visitors is included when you end your experiment. But you have no way of knowing if experiencing the flicker effect affected their behavior on your site.

Your A/B testing data is meant to help you make decisions that will boost your revenue. The risk of making decisions and implementing changes based on tainted data from visitors experiencing FOOC is high. You may implement changes that do not increase your conversions or changes that lower your conversions below your baseline. 

3 Ways to Minimize the Flicker Effect

While flickering occurs for a second, it greatly impacts your testing program and business. Fixing it is important to your bottom line. Here are ways to minimize the flicker effect:

INSTALLING THE A/B TESTING SCRIPT PROPERLY

1. Directly install the code snippet from your A/B testing tool into the header of your site. Do not use a tag manager as it will cause content flickering.

2. Install the script from your A/B testing tool inside the <head> element of the pages on your website. This will ensure that the script is loaded first and prevents flickers.

3. Opt for synchronous loading if your A/B testing tool offers it to prevent flickering. For example, Kameleoon offers an asynchronous blocking integration in its A/B testing script to remove flickering. 

4. Reduce the size of the code snippet by removing any inactive tests. This is important if your testing solution automatically adds this to your script. 

Optimize the loading time of your testing pages

1. Use a content delivery network (CDN) to make loading the testing script fast. 

2. Do not use JavaScript redirects unless absolutely necessary. Unlike server redirects, JavaScript redirects within an A/B testing tool will need to render the original page and all its elements before the variant. This makes them heavy which slows down the loading time of your page.

3. Optimize and compress the images on your page - use PNG format where possible.

DON’T INSERT A JQUERY LIBRARY

Many new sites are based on frameworks like Angular or React. This makes jQuery an unnecessary burden. 

If your website already uses jQuery elsewhere, it may cause flickering in your A/B tests. This is because even if you remove it from your A/B tests, it may still flicker if the jQuery scripts from your testing tool provider load later. 

Look for A/B testing tools that don't use external libraries in their snippet, like Kameleoon. This helps to reduce the size of the Kameleoon snippet, making it load fast to prevent flickering.

SWITCH TO SERVER-SIDE TESTING

The flicker effect is a client-side A/B testing problem, switching to server-side testing tool removes this problem entirely. 

In server-side testing, the server picks the correct page, whether the control or variant, and serves it directly to your user. This removes the need to use a JavaScript snippet and eliminates the flicker effect in its entirety. 

A/B testing tools like Kameleoon are full-stack experimentation solutions that offer personalization as well as experimentation on both client-side and server-side. This gives you the freedom to test wherever you want to. 

USE STYLE SHEETS

Use style sheets to make graphic changes in your variants. It is much easier to modify an aspect of an element on a page instead of using JavaScript to change the style of the element. This helps you save space in the tool snippet.

Depending on the tool, this is done for you automatically so you do not need to worry. In Kameleoon, for example, you can design whatever you need without worrying about the performance of your test snippet as Kameleoon does this automatically for you.

OPTIMIZE THE CODE IN YOUR A/B TESTS

Depending on the tool, you may need to optimize the code it outputs when you use its “What You See Is What You Get” (WYSIWYG) editor. 

Some testing solutions add unnecessary JavaScript instructions to achieve the changes you want to test. Toggle over to the code editor to review and optimize the code before you launch your experiment. You may need to pull technical resources to edit the code if you are not a developer and are not familiar with JavaScript.

Alternatively, you can use A/B testing tools that mitigate this. In Kameleoon, for example, you can create your variations in the graphic editor without worry. Kameleon uses an engine to apply modifications to a page, rather than JavaScript. No need to learn how to code or rope in your technical team.

MODIFY THE SETINTERVAL ON YOUR PAGE

The setInterval is used to change the targeted elements as quickly as possible. The idea is to test different values to find the ideal parameter that makes the flicker effect disappear without overly increasing the CPU your A/B testing solution uses.

Kameleoon’s solution eliminates the need for setInterval as modifications are applied to a given element as soon as it is injected into the page’s DOM.

The Kameleoon platform is 100% flicker-free

To enhance user experience and ensure accurate results for our customers, we made a commitment to ensure that Kameleoon’s solution is 100% flicker-free. To achieve this, we developed our “anti-flicker snippet.” 

Our snippet behaves in a similar way to the synchronous snippets, which block webpage content until it's fully loaded, resulting in a brief blank page display. However, our solution offers a few key advantages over synchronous snippets.

Our anti-flicker snippet leverages a "blocking CSS rule" that initially hides the entire page's content. Once Kameleoon’s application code loads, typically within an average of 50 milliseconds, this rule is swiftly removed and the web page displays. In case of any delays, sometimes due to a bad network connection, a configurable timeout is included. 

If the specified time elapses and Kameleoon’s application file hasn't loaded, the CSS rule is removed to prevent an indefinite blank page scenario.

This approach ensures that Kameleoon never disrupts a website under any circumstances. The timeout acts as a safeguard, limiting how long the installation tag can "block" the display of your site while waiting for the application file.

In the event of a timeout, how Kameleoon behaves can be configured in one of two ways. Users have the option to disable the solution entirely, meaning no experiments will be applied to that specific visitor if a timeout occurs. Alternatively, you override this and choose to run the code normally. However, with this option, you may run the risk of flickering due to the delayed execution.

Understanding how anti-flicker snippets work is essential for delivering a good user experience. By understanding how your A/B testing tool handles anti-flickering and implementing it correctly, you can enhance user experience and ensure accurate results.

Best practices for reducing the flicker effect in A/B testing

One of the best ways to avoid the flicker effect is to choose the right A/B testing solution. The right testing solution for your business works to remove the flicker effect right from its script before it makes it to your site.

Since you may already be using an A/B testing tool software, here are some best practices to implement to minimize the flicker effect during your A/B tests:

1. Avoid using a tag manager to implement the testing snippet. If you’re not confident installing it yourself, ask a developer or another technical member of your team for help.

2. Make sure to install the snippet at the top of the page of your site.

3. Use a CDN to ensure your testing pages and the script on it loads fast.

4. Optimize the pages you plan to test on - cache pages and compress images. Increasing its speed will make loading your testing snippet fast. 

5. Cross-check the changes you made in the WYSIWYG editor in the code editor. This ensures that the A/B testing solution is not injecting unnecessary JavaScript instructions to achieve the changes you want to test.

Learn more about how Kameleoon’s experimentation platform eliminates the flicker effect here.