Be careful what you wish for: cheaper testing and the traffic problem

AI variation builders blow open the bottleneck that every experimentation program has worked around for the last decade. 

For as long as A/B testing has existed, programs have had the same friction point: developer time. Roadmaps, hypotheses, and campaigns all have to compete for the same developer time that keeps the platform running, fixes critical bugs, and builds the updates that keep competitors at bay.

In Speero’s recent research with senior practitioners, a prompt-based experimentation tool (PBX) promised to remove that bottleneck. 

It’s a familiar promise, but in this case, the tool delivered, and the appeal was obvious. Ben Young, CRO at HubSpot, put it simply: “with AI,” he observed, “I don’t have to justify the resources anymore. The bottleneck is largely removed.” 

The result is that teams that adopt these tools clear their backlogs fast… and then discover there’s more to it than just developer resources.

{{blue-block-1}}

The bottleneck moves

This is something we’ve discussed before: when anyone can build a variation in minutes, the bottleneck clears.

But one thing that cannot be prompted into existence is an audience, which leads to a new bottleneck: the primary limiting factor on the number of tests you can run is your own imagination, which means you can generate test ideas much faster than the traffic to run them against. 

Suddenly, there are more experiments ready to launch than there are visitors to expose them to.

And suddenly, the question changes: “Can I build this?” was a resource question with a yes-or-no answer. “Should I test this?” is, on the other hand, a question of prioritization with an opportunity cost attached.

Every test you run is traffic you didn’t spend on a different test. Most programs have never needed to think this way, because developers handled prioritization by way of the build queue. 

Attrition is not a good way to optimize

A notable issue with testing based on dev queues is that testing ideas often live or die by the amount of effort required to build them unless the tester could prove their value to a higher-up (difficult to do without running the test). 

Deliberate prioritization is a better system. It’s a harder one too, and for the same reason: it requires a muscle most teams rarely exercise. Teams running AI-powered tests need to be able to rank ideas by their expected value before “spending” traffic on them.

In this sense, “we’re waiting on dev” is a comfortable restraint for a lot of teams. It contains within it a built-in excuse for why programs aren’t moving faster and acts as an “out” for teams that now don’t have to make hard calls on which tests actually need to run.

The harder discipline of deliberate choosing, however, will lead to better results. Building has never been the “point” of experimentation, after all: now, teams are much more empowered to chase what that actually is for them.

{{cta-block}}