Using Prompt-Based Experimentation: the context vs. the model

Most assume that better AI models build better variations. Recent research from Speero, however, points to a different variable, one that explains why a specialist tool holds its ground even against rapidly improving general models.

Some developers, upon seeing AI variation builders for the first time, ask themselves if it’s worth it to use when they already have Claude Code, Cursor, or Copilot open all day anyway. If AI writes the code, the tool is just the wrapper.

But Speero’s recent research, performed with senior practitioners and Kameleoon’s Prompt-Based Experimentation (PBX) tool, found that the intuitive answer—that the better model produces the better output—is not necessarily correct.

The variable that decided output quality in their research was the context the model worked from.

Where do the errors come from?

A general-purpose coding tool starts from a blank slate. Before it can build variations, a developer needs to load it with a page structure, the CSS, the framework, selectors, and the brand rules. At any of these stages, the model may hallucinate or make a faulty decision by choosing the wrong class or acting against a constraint that got missed in its initial instructions.

These are some of the common fears developers have around using LLMs for this level of work. So understandably, the developers who used PBX as part of Speero’s study were skeptical. The main thing that sets it apart, after all, is that it is a part of a broader experimentation platform, rather than the model itself.

But being a part of an experimentation platform means PBX has access to context that general-purpose models simply don’t. That includes:

• Master prompts that admins can use to encode a design system, accessibility rules, brand voice, and site-specific handling for things like custom components and naming conventions across the organization.
• Connected Figma integration that allows users to convert Figma mockups into production-ready variants that respect the original designer’s intent and guidelines. 

Chris Strobl, who has run PBX in production for most of a year, described what that means for his team:

{{quote}}

‍

This is a reflection of how a model working with more of the right context can outperform a “smarter” model from another source.

Another practitioner with a development background, working from a major airline website, made a similar observation:

{{quote1}}

‍

PBX is a browser extension, which allows it to read the actual rendered page, including the DOM and CSS in the state the users would see themselves. This allows the developer to skip the part of the workflow that demands the most attention and produces the most errors.

AI that understands business problems

Andrew Presnell, a CRO manager at Universal Technical Institute, commented in the research that, if AI could “consume background research, understand my problem, and then maybe come up with a solution and have an opinion. That would be powerful to me. But instead, I find myself having to write everything from scratch. It knows nothing about my business.”

Speero describes Presnell’s observation as “a forward-looking one about where the category is heading.” The ceiling imposed by AI code builders is based on the context they have.

So when you're using AI to build your variations, either choose a platform that supports your context, or train one carefully to know everything it needs to know. Faster output is good. When the output is faster and higher-quality? That’s huge.

{{cta-block}}