Skip to main content
Group of people sitting around table

Six Groundbreakers Give Practical Advice on Using AI in Experimentation

November 10, 2023

The hype around AI has hit a feverish crescendo. According to Gartner, we’ve reached the ‘peak of inflated expectations,’ which explains why many people feel overwhelmed and unsure about how to really use AI in experimentation.

But it’s worth considering that within the next 2-5 years, technologies such as generative AI and decision intelligence will reach mass adoption. Today’s early adopters will get a head start on their competition.

With that in mind, and to ensure we stay firmly in the realms of what’s practical, we spoke to six AI groundbreakers about where and how AI can help in day-to-day optimization work, including a particular focus on how to balance human creativity alongside AI capabilities.

Here’s what they shared.

Delegating data analysis

A fully staffed experimentation team is the dream for many working in the industry. But even if you are lucky enough to have plenty of resources, you can still find yourself constantly busy. Johann suggests what tasks to delegate to AI;

AI gives you time to think. Do more thinking, less doing. I now outsource CRO analysis tasks to AI routinely. It saves time and, frankly, does a better job. That frees up mental space to think strategically. Work on the "why" of a test, and let AI do the "what."
Johann Van Tonder
Johann Van Tonder
CEO at AWA Digital

While AI is known for lightning-speed data analysis on incomprehensibly large data sets, (right now) it can only take you so far. What’s needed is some of our uniquely human qualities, as Abhi explains;

Use AI tools to analyze large sets of data quickly and identify trends and patterns. AI can help you uncover insights that might be difficult for humans to discern in massive datasets, providing a data-driven foundation for experimentation.

Encourage your team to brainstorm and generate creative ideas. Humans can offer unique insights, empathy, and creativity that AI lacks. These ideas can be used to formulate hypotheses for testing.

Combine AI's data analysis capabilities with human-generated hypotheses. AI can help prioritize which hypotheses to test based on data, while humans can provide critical thinking and creativity in crafting those hypotheses.
Abhi Agarwal
Abhi Agarwal
Leader - Digital Transformation at IBM

Automate away the mundane tasks

Imagine a world where you could click your fingers, and all of those niggly little jobs were wiped clean off your to-do list.

Okay, automation doesn’t sound sexy, but you can’t deny taking busywork off your plate is an attractive prospect. Especially when you consider there’s no hiring process, training, or costs involved in getting your new AI intern up to speed, as Jim says;

Mundane tasks that require a modicum of “understanding” can now be automated. These tasks can be performed better than an intern, without any experience, can hope to accomplish.
Jim Sterne
Jim Sterne
President at Target Marketing of Santa Barbara

A brainstorming sounding board

One area that several individuals mentioned was AI acting as a creative assistant to support you with the brainstorming and ideation process. The pertinent word being “assistant,” as Jim stresses;

Generative AI makes an astoundingly good creative assistant. The important thing is to remember that everything that gets published must, must, must go through human review first. With that in mind, we have yet to plumb the depths of the value of using generative AI as a brainstorming colleague. It is exhibiting new mind-blowing capabilities every day. Things are about to get very exciting.
Jim Sterne
Jim Sterne
President at Target Marketing of Santa Barbara

So, how can you practically use AI to help you be more creative, whether designing variants to tests or coming up with new ways to solve user problems? Sim provides some ideas on which tools and prompts to use;

As individual practitioners, it is important for us to understand our own creative process and find ways to enhance it using AI. I like to do this by seeking new perspectives on problems, as it enhances my creativity and allows me to connect new ideas.

However, obtaining 10 different perspectives from different experts would be impractical and would result in discarding numerous brilliant ideas before they’ve had the time to develop.

Instead, I use ChatGPT to describe my problem and ask it to use the Six Thinking Hats framework to give different perspectives, or I consult expert perspectives on Delphi (an AI tool that “clones” the minds of experts).

Ultimately, creativity is limitless, and in order to leverage AI to enhance it, we must commit to continually evolving our understanding of it.
Sim Lenz
Sim Lenz
Director of Experimentation at

Creating good problems

Finding surface-level issues that users don’t care about won't move the needle. But complex problems that change user behavior or innovative ideas outside your local maximum can deliver massive improvements. However, as you provide advanced, personalized experiences, future problems become more complex. But some problems are good to have. As German’s recent experience shows,

Since the advent of AI in both text prompts, conversational tools like ChatGPT, and more sophisticated tools specializing in website analysis, my days are vastly different compared to the start of my career.

By treating AI like a colleague in a whiteboard meeting, I’ve been able to put out more A/B testing charters, drastically increase the pace of making presentations to stakeholders, and create much more sophisticated problems for myself.

Problems akin to using click behavior and user profile information to augment website experiences in real time or having so many experiments in the pipeline the conversation for Level of Effort vs. Level of Impact is the most common weekly conversation.
German Botello
German Botello
Global Site Personalization Lead at Bose Corp.

Your AI assistant

It’s worth realizing the current limitations of AI to avoid getting swept up in exaggerated claims. No, you can’t ply it with realms of data and testing tool access and let it do all your work.

While there is a lot to get excited about as the technology continues to develop, it’s important to realize that AI currently requires human input;

AI, in its current state, is powerful. However, it is still bounds away from being hands-off. Especially when we are talking about creativity. AI is in a great spot to help you come up with ideas, build structure, debug, and is sometimes better than Google.

It still needs a human touch to bring it across the finish line. Let's be honest, without this balance, the majority of people can spot what has been generated by AI. AI should be used as a means of brainstorming. Not your deliverable.
Ian Klosowicz
Ian Klosowicz
Data Analyst at

German put it wonderfully, “Don’t solely rely on AI but also don’t not use it. Let it be your colleague, the J.A.R.V.I.S to your Iron Man. Ask questions but question its responses, and brainstorm your own ideas.”

The future is just beginning

The current level of AI hype can be exhausting. But rather than disengaging, start focusing on small, practical ways you can use AI in your daily work, as the six groundbreakers above suggested.

5 ways to start using AI in experimentation

  1. Delegate data analysis tasks: let AI uncover the ‘what’ while you focus on the why.
  2. Automate away mundane tasks: from putting the next test live to updating the prioritization of new ideas based on the latest test results.
  3. Use AI as a brainstorming sounding board: Try using the Six Thinking Hats framework to get different perspectives or consulting expert perspectives on Delphi (an AI tool that “clones” the minds of experts).
  4. Create good problems, such as hitting test velocity thanks to AI-driven efficiencies.
  5. Have an AI sidekick: remember, while AI can help you come up with ideas, build structure, or debug problems, it still requires humans to review and check the output.

As you begin to understand where and how AI can assist you, it will be easier to stay up-to-date with this fast-moving technology. After all, things will only get more impressive from here on in.

Thanks to all the humans who contributed their thoughts and time: Ian Klosowicz, Data Analyst at, German Botello, Global Site Personalization Lead at Bose Corporation, Abhi Agarwal, Leader - Digital Transformation at IBM, Sim Lenz, Director of Experimentation at, Jim Sterne, President at Target Marketing of Santa Barbara, and Johann Van Tonder, CEO at AWA Digital.

If you’d like to keep up-to-date with all the industry's top experimentation-related news, sign up for our newsletter or find industry leaders to follow in our Expert Directory.

Topics covered by this article