Can marketers trust their A/B testing tool anymore?
That’s the key question given the introduction of Apple’s latest Intelligent Tracking Prevention (ITP) update, which is poised to have an enormous impact on CRO practices.
At Kameleoon, we’re surprised that ITP has not yet made the digital marketing headlines. While it is a very technical topic it can potentially have an enormous impact on marketers carrying out A/B testing experiments.
ITP is implemented within the Safari browser, and while this has a relatively low percentage of the desktop market, it is a completely different picture when it comes to mobile devices, where Apple has much higher market share (closer to 40% to 50%). This means that almost half of your web traffic could be impacted by ITP!
Here are the key points we captured from our conversation with Jean-Noël:
- What ITP 2.1 is
- How it impacts A/B testing tools
- How we overcome this issue at Kameleoon
1 What is Intelligent Tracking Prevention?
While previous ITP versions mainly targeted online advertising systems, 2.1 is much more aggressive and impacts web analytics software such as Google Analytics (GA) and A/B testing tools in a very serious way.
This identifier is randomly generated on the first pageview of the first visit of a user to your website. It is then reused on all subsequent page views and visits to identify this particular user. For example, this allows Google Analytics to know if a visitor is new (as a new identifier/cookie has been generated) or returning (as it finds the cookie that has been set previously).
Given that cookies now have a lifetime of seven days in Safari, this overrides the much longer default expiry date normally set by GA. So, a visitor making a first visit on Monday, then returning on Tuesday of the next week will be considered as a totally new visitor.
Their new visit won't be linked to the previous one. As a result your new visitor metric, as reported by Google Analytics, is completely false for any Safari traffic. Many other metrics (such as number of unique visitors, time to buy, etc) are also completely distorted. Essentially, it's hard to trust any number from cookie-based analytics solutions anymore!
2 What are the main impacts of ITP 2.1 on A/B testing tools?
The majority of testing and conversion optimization platforms also bundle web analytics capabilities within the tool. This means they’ll suffer from the exact same issues we just described, which is problematic in itself. However, for A/B experiments, an even more serious obstacle appears.
This is because each time a visitor is bucketed into an experiment, a variation is selected for them. This variation must be remembered, since if they return, they must see the same variation as before.
To achieve this all front-end testing solutions store this information (the association between experiment and variation) in a cookie. So, with ITP, a visitor coming back more than seven days after they initially triggered an experiment runs the risk of seeing a different variation. A/B testing in these conditions is useless and does not produce reliable results.
3 Why is Kameleoon the only conversion optimization platform that automatically and transparently handles Apple ITP?
In short, because our engineering team has implemented a cross-domain Local Storage (LS) for Kameleoon.
Local Storage, a standard Web technology supported by virtually all browsers in 2019, is basically a data store in the browser, so it can handle data in the same way that cookies do.
Given all of its characteristics, Local Storage seems to be an ideal candidate to replace cookies. And ITP 2.1 does not place restrictions on Local Storage.
Actually, several testing platforms already implement LS based storage instead of cookie based storage. However, there is a huge problem with Local Storage: it is limited to a single exact subdomain.
For this very reason, unless you're lucky enough to have a single site and a single subdomain throughout this entire site, you will still suffer from serious issues if you use a solution relying on a simple Local Storage implementation. These issues would appear similar to those caused by ITP, although the reason is totally different.
As an example, let's imagine that most of your e-commerce website is hosted on https://www.randomshop.com, but the conversion funnel is on https://transaction.randomshop.com. If your visitor ID is persisted on the LS, your analytics solution will report two different visitors when a complete purchase occurs, one seen on www.randomshop.com and the other on transaction.randomshop.com. Data does not transition consistently from a domain to another, and such a solution would see the as two completely separate sessions, not a single one of the same visitor.
So we went further and implemented a full cross-domain Local Storage implementation, as even a small percentage of pages on a different subdomain on your website can mean unreliable and even false results for A/B experiments.
When the first versions of ITP came out, we actually had to perform some minor changes to our system. Oddly, external Local Storage domains are actually treated as third-party context by all ITP versions, like cookies. We initially hosted the common iFrame file on our own (Kameleoon) domain, not on the customer domain (mostly to facilitate setup), so the common Local Storage data was considered to be third-party and was blocked by Safari. To fix this, we just changed the hosting of the iFrame so that it became hosted on the customer’s domain.
4 Two additional closing notes:
Although we can guarantee that our analytics core is unaffected by ITP, and that all our reported traffic data is reliable, of course we cannot guarantee the same for third party web analytics platforms. Our integration bridges with partner tools continue to work in the same way as before. We recommend that customers discuss the situation with these vendors directly.
Our server-side SDKs (and Mobile SDKs) are completely unaffected by ITP, as analytics tracking calls are made server-to-server, never on the front. If you use the Back-End/Front-End bridge, a cookie will be used to synchronize the kameleoonVisitorCode, but it will be set via the server (HTTP header), not via the front, and will be considered a first-party cookie, so is not affected by ITP.
The Kameleoon team is extremely proud to be able to provide our customers with this painless solution to the ITP 2.1 challenge. Thanks to it, our customers can continue to enjoy stable, reliable A/B testing on their mobile traffic, without having to spend precious time implementing custom hacks to mitigate ITP. As a software vendor, we strongly believe we should do the maximum to ease the lives of our customers and this is a perfect example of our philosophy.
We will continue to monitor the evolution of Apple's ITP technology, as well as other initiatives from other browser vendors (Firefox has a similar technology to ITP, but this is not yet as extreme). Any new changes will be swiftly analyzed and we’ll deliver solutions as soon as possible. We currently have a very sizable head start over competing platforms in this area, and we intend to keep it.
For a full technical understanding of ITP 2.1 and how Kameleoon overcomes it, visit our documentation section.