Skip to main content
Thumbnail

23 Experts share bold predictions for experimentation in 2024

December 7, 2023

We sought the opinions of 23 leaders in our industry to get their predictions on where the industry is heading and the biggest trends likely to impact experimentation in the coming year.

Among the foreseen changes is how experimentation will transcend the confines of a siloed team and surpass the bounds of web experimentation—a notable departure from the myopic Conversion Rate Optimization (CRO) lens. The move means experimentation will become an integral part of product and marketing teams, among others.

The impact of data legislation, especially in the EU, is expected to reshape the industry, driving companies to adapt and prioritize customer-centric data approaches. This shift aligns closely with AI-driven personalization, which will put even greater emphasis on data.

These predictions foreshadow a fundamental transition from CRO to decision science, ushering in a new era of experimentation with a broader and more impactful perspective.

Amidst these predictions, there is hope for greater support for the next generation of practitioners, especially as job roles and expected competencies change with the increased use of AI and machine learning tools. 2024 promises to be a year of dynamic shifts and profound evolution for the experimentation industry.

Experimentation will continue to mature 

1. Experimentation will extend across all teams

The Kameleoon 2023 Experimentation & Growth Survey found that businesses with well-aligned product-led and marketing-led growth strategies are expected to be 81% more likely to grow significantly in 2024. Experimentation breaking out of its silo will generate big wins for those able to follow this trend. But as Ruben discusses below, not all companies are ready for this shift;

“It is very different per country and market. Some markets still have to start with CRO. For more mature markets, there continues to be a higher demand for moving experimentation from its silo into product teams. We need to let go of our strict process and embed it in the process of the product team.”
Ruben de Boer
Ruben de Boer
Lead Conversion Manager at Online Dialogue

2. Self-serve models will drive more teams to experiment

What is driving experimentation into other teams (specifically product and marketing) across companies? Sai believes this transition is being fueled by the popularity of self-serve sales models, which require a digital version of the sales and onboarding process. To achieve this, marketing, sales, and product teams need to be involved in experimentation to create highly converting experiences.

“We've witnessed a significant shift in companies previously focused on lead generation moving toward a more self-serve model. With this change, we are observing CRO aligning more closely with Product Management.

In e-commerce companies, this fusion has already been present, but we are now noticing it in other industries as well. Whether it's a software company offering a self-serve signup or an insurance company enabling individuals to initiate their coverage online, both will necessitate CRO experts who understand that the website is the product itself.”
Sai Maddali
Sai Maddali
Digital Optimization Analyst at URBN
"My prediction is that more and more DTC's, or CPG's businesses will be experimenting on faster shipping times, testing new vendors for better UX, and testing improvements on firmware updates rather than blindly shipping.

The landscape of experimentation has surpassed what people are used to: testing on a website. Companies like Disney are constantly pushing the envelope of what is experimentation. They don't just test website changes, they're testing past the website.

Don't let experimentation learnings stop at your website. Technology has reached the point where full customer journeys can be produced all on the same device and other devices for known users. Engineers are already invested in the product and can be leveraged to help build experiments; they know the code best. Push the boundaries.

In 2024, teams should be trying to push innovation and insights outside of the four walls of a screen."
Eddie Aguilar
Eddie Aguilar
Founder, Blazing Growth

3. CRO will be seen as decision science

Kelly sees experimentation morphing into something different altogether: decision science.

“I believe we're going to shift from CRO to decision science, adding a more well-rounded understanding of what it means to help make better decisions using all available tools in our toolbox, including qualitative research and additional experimental design methods.”
Kelly Anne Wortham
Kelly Anne Wortham
Founder, Forward Digital & Test & Learn Community

Similar to Kelly’s prediction, Michiel suggests that as experimentation matures, teams will look at different methods to help make decisions, such as meta-analysis.

“2024 will be a breakthrough year for meta-analysis of A/B tests because more and more companies have had a CRO program for years. Meta-analysis will help more tests go into production because tests that individually would not make the cut CAN be called 'fit to push live’ using meta-analysis.”
Michiel Jansen
Michiel Jansen
Head of Global Conversion Centre of Excellence ING

Gary also shares some of the areas where testing can help with business decision-making outside of website optimization.

“The value focus will shift from the myopic CRO lens to a broader experimentation mindset that drives compounding growth, whether on conversion, process, system, or customer value optimization.”
Gary Viray
Gary Viray
Founder at Propelrr

4. Increasing industry maturity will change who is experimenting

The more mature experimentation is, the greater the impact our work has on the business. Interestingly, the recent Google Optimize sunset is having a lasting impact on the wider industry and its overall maturity, as Shiva explains.

“The maturity of experimentation will naturally skew up due to the Google Optimize sunset. Those who were serious about experimentation will actually start funding the program properly with more resources, while those who were passive (and probably bad at it, to be honest) will throw it away.

This will combat an economy where CROs are amongst many being laid off, unfortunately (which will ding our maturity), but overall experimentation maturity across the board will increase, which will only beget more folks adopting it.”
Shiva Manjunath
Shiva Manjunath
Senior Web Product Manager, CRO at Motive

It’s not only the absence of Google Optimize increasing industry maturity. Deborah examines the impact of a broader move to a pay-to-play model.

“The move from Google’s Universal Analytics to GA4 signals the shift towards paid platforms where companies will now need to pay to store their data in BigQuery, plus use other paid tools to analyze and visualize their data. With the increasingly complex setup required to track, collect, and store data, many experimenters now also need to pay outside analytics firms to get all the systems properly running.

Plus, as we shift away from being able to freely collect 3rd and 1st party data, the cost to gather and analyze user information is becoming increasingly high. Data itself is becoming a commodity that comes at a premium. In essence, experimentation is moving from an activity that anyone used to be able to do for free to something only available to the world’s biggest and most elite corporations who can afford to pay to play.”
Deborah O'Malley
Deborah O'Malley
Founder, GuessTheTest

However, Manuel believes we’re still far from full maturity and provides a list of areas businesses need to fix before we can escape the “CRO hamster wheel” including the narrative told to senior management.

“A lot of businesses are stuck on the same CRO hamster wheel—running tests but not really going anywhere. Call me cynical, but this industry needs a radical overhaul from the ground up if experimentation is to be important to a business beyond “We ran X more tests this year.

There is still a lack of understanding at the senior level about what experimentation can really do. Typically, they have bought into the narrative about big wins and revenue gains and ignore the true power of experimentation. Couple that with the fact there are still a lot of inexperienced CROs parading as senior experimentation leads or managers with no understanding of how to communicate well. There are a lot of bad practices festering at the practitioner levels that are kept well hidden from senior management. Selective reporting happens, too. So there's a lack of true oversight from the top down.”
Manuel Da Costa
Manuel Da Costa
Founder at Effective Experiments

It’s undeniable that experimentation has grown in popularity over recent years, but Sina has noticed a slightly less positive trend emerging.

“CRO is at risk of becoming just another buzzword. And the disparity between companies offering CRO and those executing it effectively is growing rapidly.

Similar to SEO 10 years ago, there is a pressing need to educate consumers on "black hat" vs. "white hat" CRO approaches and to share resources on how to vet expert service providers from pretenders. Otherwise, legitimate agencies providing data-driven and research-based CRO will continue to face increasing consumer skepticism, fuelled by disappointing experiences with agencies that don't understand statistics or customer research and are just spaghetti-testing.”
Sina Fak
Sina Fak
CEO of Conversion Advocates

AI & machine learning will disrupt all areas of the experimentation industry

5. The year of advanced customization

Many pundits have heralded previous years as “the year of personalization,” but to date, most businesses have only scratched the surface of what personalization can do. With the increasing sophistication of machine learning algorithms, online competition, and consumer demand for individualized content, will 2024 finally be the year of personalization?

Kelly has some reservations.

“I do not actually believe personalization is going to take off, at least not unauthenticated machine learning-driven personalization. It may be different beyond the authentication wall, where customers have different expectations, we have more information and privacy laws, and browsers give us more flexibility to work with, but time will tell.

On the other hand, customers tell us they like flexibility and customization, such as filtering and navigation. Helping them get where they want to go faster with those capabilities will be key.”
Kelly Anne Wortham
Kelly Anne Wortham
Founder, Forward Digital & Test & Learn Community

Are we actually looking at a year of advanced AI segmentation and customization rather than “true” 1-1 personalization? Koutheir's prediction suggests so.

“In my opinion, CRO will benefit from advanced segmentation based on AI in 2024. AI will play a big role in targeting and analyzing complex user segments in an automated way. Detecting specific user behavior and activating advanced personalizations in real-time.”
Koutheir Somai
Koutheir Somai
Analytics & Optimization Project Manager

6. Innovations will focus on privacy-preserving personalization techniques

Rohan is a little more optimistic about the impact of personalization next year, along with some critical predictions on how this will impact the wider industry.

“One significant industry trend poised to make waves in 2024 is the integration of AI-driven personalization at scale. As machine learning algorithms become more sophisticated, they will enable businesses to deliver highly personalized user experiences in real-time. This trend is fueled by the increasing demand for individualized content and the need for brands to stand out in a crowded digital space.

The impact will be twofold: firstly, it will lead to a more efficient and effective CRO process as AI can quickly analyze vast amounts of data to determine the optimal experience for each user. Secondly, it will shift the industry focus towards the ethical use of data and privacy compliance, as personalization relies heavily on user data. Companies will need to balance the drive for personalization with the growing public concern over data privacy, potentially leading to innovations in privacy-preserving personalization techniques.

This trend will likely elevate the importance of roles centered around data science and AI within CRO teams. It could lead to a new standard in user experience, where personalization is not just a feature but an expectation.”
Rohan Singh Rajput
Rohan Singh Rajput
Senior Data Scientist at Headspace

7. The future of the web as a prompt machine

The potential of AI is so broad it’s fascinating to hear how our experts think things might turn out. While James has reservations about when we might see big changes, there’s no doubt it will cause a significant shift in our industry and the world at large.

“It’s hard to know how AI will truly impact the world of experimentation, but it has the potential to completely change how we interact with the web at large. Maybe not in 2024; that's too quick—it still needs to go through the hype cycle. But we’re laying the foundation for what could be a dramatic shift in how we navigate the web.

The optimist in me believes every website could become a form of Google—a prompt machine to generate a single page of content for every individual person. Maybe personalization isn’t dead; it just needed a little AI boost?

The pessimist in me believes we’ll spend another couple of years frantically building AI features into every facet of our tech stack, only to have them collect dust years later.

How it plays out could determine whether we keep optimizing the front end of digital experiences or shift to optimizing prompts and what we return for prompts in the next few years.”
James Flory
James Flory
VP of Delivery at Conversion

8. Companies with solid data foundations will excel with AI

To deal with some of James' pessimism, Nils suggests how to avoid getting blinded by the new shiny AI. 

“The impact of AI will only go so far unless we get the basics right. Create the right data to train on. Research, hypotheses, documentation, statistics, and analysis.
If I were to make a prediction for 2024, it would be that the companies that excel at getting the basics right won’t be blinded by shiny objects (as easily) but instead use technology like AI to build on solid foundations and with that outcompete their competition.”
Nils Koppelmann
Nils Koppelmann
Founder at 3tech Gmbh

Experimenters' relationship with data will get tougher

9. No more cookies means upping the ante on zero and first-party data

While the cookie phase-out has been on the cards for a while (and doesn’t directly impact A/B testing), it does affect the broader digital landscape and what data companies can collect and use about customers. As SIobodan explains,

“Google Chrome will start phasing out third-party cookies in 2024 as part of its Privacy Sandbox initiative. This doesn't mean the CRO industry will completely change how it operates, but the tools will have to. More importantly, because of how much this will affect the ad industry and digital marketing in general, it will make privacy a hot topic in 2024, and I expect this to push more people from our industry to collect first-and zero-party data. That shift towards a more customer-centric future would be a win for everyone.”
Slobodan Manic
Slobodan Manic
Host, Nohacks.show & Growth Strategist, Maksimer

10. New legislation will impact our relationship with data and tools

It’s not just cookies that will shake up how we use data; new legislation and geopolitical events will also affect our industry, as Lucas explains; 

“We need to adapt or prepare for changing legislation. In the past few years, we have seen the EU and Privacy Authorities being very critical of Google Analytics and its compliance with GDPR, plus the EU ordering Meta to stop personalized advertising. Also, because of events at a geopolitical level, the gathering and use of data is likely to be put under a magnifying glass, especially next year. The fact that both the EU and the US have major elections scheduled for 2024 will drag this topic into the political campaigns.

I expect more stringent legislation to be developed in 2024. Potentially, a far tougher ruling over Google Analytics restricting or even prohibiting its use, browser standards to be more privacy friendly, making it harder to do personalization and experimentation might require explicit consent. Due to the geopolitical climate, the country a tool comes from will become more relevant, with preferences rising for EU-based tools for businesses based in Europe. Outside of Europe, that could lead to a mirroring effect.”
Lucas Vos
Lucas Vos
Senior Conversion Specialist at RTL Netherlands

Experimentation teams will have to focus on their human advantage

11. Employers will look for AI & machine learning tool competency

With all this talk of AI and machine learning, how many of us actually know how to use the plethora of tools that are emerging? The AI trend is predicted to introduce demand for a whole new set of skills, which need to be administered in compliance with regulations and measured for impact, as Tony suggests.

“Upskilling AI tech in the workspace, responsibly. This means deeper knowledge of the skills actually involved in using the TOOLS. Processes put in place, including data cleansing, audits, and regulation. It will also mean new metrics—do companies see greater efficiency (less time spent on ‘mundane tasks') enabling focus on internal and external relationships, creativity, and burnout?”
Tony Grant
Tony Grant
Marketing Engagement Lead - EMEA at IIRIS

12. Designers will need to rethink their USP

While the “robots are coming for your jobs” panic isn’t exactly true, new technology will undoubtedly change how and what we work on. And Brian suggests designers might be impacted heavily.

“In 2024, after some very public web design failures, businesses will hit a crossroads. Do we continue to let creative ‘experts’ develop our designs?

Generative AI will prove to be as good or better than human designers, will allow for unlimited iterations, will deliver changes instantly, and will never get grumpy about criticism. Human designers will turn to behavioral research, data analysis, and experimentation to regain their ‘human’ advantage.”
Brian Massey
Brian Massey
Conversion Scientist at Conversion Sciences

13. Diversity improves experimentation, and more companies will finally catch on.

Diversity in the workforce is something every business should be implementing without incentive, but there are also measurable benefits, too, as Lucia has found. 

“Five out of six experimentation programs I analyzed had higher win rates if more (diverse) people came up with a hypothesis from their unique knowledge and experience. I call this the Collaboration Bonus.”
Lucia van den Brink
Lucia van den Brink
Lead Consultant & Founder,Increase Conversion Rate

As more companies and hiring managers push for greater diversity and inclusion across experimentation teams, we can expect digital experiences that cater to a broader spectrum of people.

Sadly, the trend of “synthetic data” might counter this positive development, as Jason Patterson, Founder of Jewel Content Marketing Agency, mentioned. Synthetic data is artificially generated data that could be used in place of real user research. Such data is trained on an initial data set, which could have inherent biases and not be truly representative.

14. We will need to offer greater support for the next generation of practitioners

It’s so easy to get caught up in the tools, tech, data, and processes of experimentation. Despite all the predictions for AI and machine learning in the coming year, we need to ensure we don’t forget the people who make it all happen.

“I hope and believe we'll see a shift toward content and communities that are more empowering and empathetic toward junior experimenters. Let's not forget we were all new to our craft at some point.”
Tracy Laranjo
Tracy Laranjo
Fractional Head of CRO & Experimentation

Content (optimization) is king

15. Prioritizing genuine customer feedback & messaging will be top priority

In response to the emergence of synthetic data and AI-generated digital experiences, Loredana expects a renewed focus on customer-focused copy.

“There is the growing issue of AI (when not used strategically) to generate jargon-led or fluffy copy.

Companies ahead of the game will recognize the need to experiment with messaging that resonates most effectively with their customers and is based on real feedback.

This shift will place importance on customer-focused copy that is benefit-driven rather than feature-driven. There will be a strategic focus on aligning more closely with customer preferences and testing buyer psychology in copywriting on landing pages and websites.”
Loredana Principessa
Loredana Principessa
Founder at MAR-CO

Similar to Loredana’s prediction, Nima sees content optimization as a big trend for 2024.

“I think with the growth of content on the web, there will be a shift in understanding what content drives what behavior. There is a heavy focus right now on optimizing the UX and journey, but I think we will balance this by also looking at what content is important. 2024 will have a greater focus on content experience optimization.”
Nima Yassini
Nima Yassini
Director & Co-Founder at White Maven

Top 5 predictions for experimentation in 2024

2024 is going to be anything but quiet. Here’s a summary of the biggest trends likely to impact our industry in 2024 according to 23 industry leaders;

Trend 1: Experimentation will continue to mature

  • Experimentation will extend across all teams.
  • Self-serve models will drive more teams to experiment.
  • CRO will be seen as decision science.
  • Increasing industry maturity will change who is experimenting.

 

Trend 2: AI & machine learning will disrupt all areas of the experimentation industry

  • 2024 will be the year of advanced customization.
  • Innovations will focus on privacy-preserving personalization techniques.
  • The future of the web as a prompt machine.
  • Companies with solid data foundations will excel with AI.

 

Trend 3: Experimenters' relationship with data will get tougher

  • No more cookies means upping the ante on zero and first-party data.
  • New legislation will impact our relationship with data and tools.

 

Trend 4: Experimentation teams will need to focus on their human advantage

  • Employers will look for AI & ML tool competency.
  • Designers will need to rethink their USP.
  • Diversity improves experimentation, and more companies will finally catch on.
  • We will need to offer greater support for the next generation of practitioners.

 

Trend 5: Content (optimization) is king

  • Prioritizing genuine customer feedback & messaging is top priority.

 

A massive thanks to all of the experts who shared their predictions with us.

1. Ruben de Boer, Lead Conversion Manager at Online Dialogue 

2. Sai Maddali, Digital Optimization Analyst at URBN

3. Kelly Anne Wortham, Founder of Forward Digital and Test & Learn Community (TLC)

4. Michiel Jansen, Head of Global Conversion Centre of Excellence at ING Bank

5. Gary Viray, Founder at Propelrr

6. Shiva Manjunath, Senior Web Product Manager, CRO at Motive 

7. Deborah O'Malley, Founder GuessTheTest

8. Manuel Da Costa, Founder at Effective Experiments

9. Sina Fak, CEO of Conversion Advocates

10. Koutheir Somai, Analytics & Optimization Project Manager

11. Rohan Singh Rajput, Senior Data Scientist at Headspace

12. James Flory, VP of Delivery at Conversion

13. Nils Koppelmann, Founder at 3tech Gmbh

14. Slobodan Manic, Host at nohacks.show and growth strategist at Maksimer

15. Lucas Vos, Senior Conversion Specialist at RTL Netherlands

16. Tony Grant, Head of Marketing - Asia & Performance at INFINOX Global

17. Brian Massey, Conversion Scientist at Conversion Sciences

18. Lucia van den Brink, Lead Consultant & Founder at Increase Conversion Rate.com

19. Jason Patterson, Founder of Jewel Content Marketing Agency

20. Loredana Principessa, Founder at MAR-CO

21. Nima Yassini, Director & Co-Founder at White Maven

22. Tracy Laranjo, Fractional Head of CRO & Experimentation at Aftergrow

23. Eddie Aguilar, Founder, Blazing Growth
 

If you’d like to keep up to date with all the industry's top experimentation-related news, sign up for our newsletter or find industry leaders to follow in our Expert Directory.

Topics covered by this article