AI will make experimenters indispensable

AI is still a huge talking point in tech, SaaS, and experimentation; it continues to reshape how companies optimize their growth. 

When we talked to leading experimentation experts—the Hall of Fame honorees from the Who’s Who in Experimentation 2025—they shared their own perspectives on AI and how it is interacting with their worlds of experimentation.

Many of them told us the same thing: that experimentation is a human discipline. AI may enhance it, but it cannot replace it.

AI scales insights from human intuition

Taking human insights and scaling with AI is a natural application of the AI toolset. In her work at GuessTheTest, founder Deborah O’Malley adopts tools that help her identify predictive signals from hundreds of experiments. 

This saves her time and energy, but the actual insights still come from her. “I’m not replacing human creativity or a data-driven process,” she explains. “Just augmenting it with intelligence that scales.” 

This is a key point: AI-powered insights still need human interpretation, and often human intuition too. 

Or, as AGConsult co-founder Els Aerts puts it, “the increasing value AI brings to the table has made the human side of experimentation more important than ever.”

Without a deep understanding of user behavior, teams risk optimizing for the wrong outcomes. Aerts’s insights are a reminder that while AI can help teams move faster, you can only make progress from asking the right questions.

Alignment through artificial intelligence 

Often, the real challenges of experimentation come in the form of alignment: breaking silos, securing buy-in, and more. It’s why Speero CEO Ben Labay proposes using a standardized scorecard before scaling experimentation. Otherwise, he says, “teams operate in silos, interpreting data differently, prioritizing inconsistent goals.” 

While AI is certainly capable of supporting governance, it cannot create alignment on its own. Erin Weigel, author of “Design for Impact: Your Guide to Designing Effective Product Experiments,” highlights this with a simple insight: “people learn best by doing—and by making mistakes.” 

These perspectives represent a shift happening across leading organizations: a move towards all-team experimentation, where marketers, developers, product teams, and designers contribute to a culture of experimentation-led growth.

AI supports the shift, but it doesn’t drive it.

What skills do experimenters need in an AI age?

AI can’t augment skills that don’t exist. Experimenters still need their fundamentals to succeed and to drive their AI agents and experimenters in the right direction.

Critical thinking, instinct, and the ability to learn, adapt, and work across teams in an organization are all still critical skills for every experimenter to have. 

As Collin Crowell, VP for North America at Kameleoon puts it, “because it makes creation so easy, experimenters need to work harder at knowing WHICH test to run.

“This means rolling up your sleeves and taking a long hard look at the data that feeds your AI. If that's not clean, AI is a gimmick.”

Of course, working with AI means experimenters need new skills as well. Using AI agents or experimenting with prompts means experimenters need to know how to work with AI: how to write effective prompts and how to set AI agents to automate the right tasks in the right way. 

For example:

• Evaluating AI-generated insights for bias and false positives
• Integrating AI into standard experimentation flows
• Validating agentic output 

These new skills are strategic and cross-functional, and will enable experimenters to get the most out of the AI tools that are redefining the industry. 

A practical insight: Prompt-based experimentation

Prompt-based experiments are a great example of how AI supports existing experimentation skills. It empowers anyone to configure, build, and deploy experiments in minutes.

But if you’re not testing the right elements, or don’t have a strategy for how to react to your findings, your AI will happily run your tests anyway. Prompt-based experimentation makes the testing easy and fast, but the fundamentals still matter. 

The top experimenters in 2025 approach AI differently than a lot of their peers; they want to know how to design systems where AI supports their teams to learn faster and act smarter. 

The outcome is acceleration, built on automation, and driven by pure human insights.

Want to learn more about what the industry leaders have to say? Explore the Who’s Who in Experimentation 2025 today.

Or, if you want to try prompt-based experimentation for yourself, start your free trial now!