Velocity reveals more about your experimentation program than rigor

Most experimentation programs will never find out if they're any good.

It's not because they lack rigor or because they produce “winners.” It's because they never run enough tests to hit the problems that actually kill programs.

Velocity is a more revealing metric of program health than rigor.

You can't have a quality problem at a volume you never reach

The pushback I hear is that velocity is a vanity input. Chase it, and you get a sausage factory of experiments, defects shipped faster, and a count that looks healthy while the program rots.

Manuel Da Costa correctly calls this “experimentation theater.”

To be clear, pumping out tests for the sake of high-velocity numbers is dumb. But guess what? 

Almost none are. 

The "sausage factory" is an outlier problem. 

Most teams are nowhere near the volume where sloppiness becomes the risk. 

The average number of A/B tests teams produce each month is between 1 and 2, based on Kameleoon’s Experimentation-led Growth research. 

The average amount their traffic could be supporting? It’s 3-4x higher than that.   

Most are sitting on traffic they never use, telling themselves that fewer, more rigorous tests is the disciplined choice.

Except it isn't discipline. It's avoidance.

Velocity builds the muscle

You don't develop an experimentation program by being careful. You develop it with reps.

Reps are how you find out what's actually broken. And what's broken is almost never the statistics.

It's stakeholder management. It's the exec who nods in the meeting and kills the roadmap after. It's the PM who wants the win without the test.

You only hit those obstacles when you try to move real volume. A team running two tests a month never gets there. They never learn whether anyone wanted experimentation in the first place.

That's the thing velocity exposes. When a team finally pushes to scale, and the organization pushes back, you learn the truth. The program was built on faulty ground. No one believed in it. Rigor was never going to save it.

The opportunity cost nobody prices

Every experimentation program has a ceiling. Your traffic sets it. There's a maximum number of tests your site can support at the statistical power you need, and it's almost always far higher than what you're running.

Not using that capacity is a cost. You're leaving statistical power on the table. You're leaving decisions unmade. You're paying for traffic and optimizing a sliver of it.

So the question I care about isn't "how rigorous is your program." It's "can you even hit your max?"

If you can, congratulations. You're in the top 1%, and now rigor and per-capita experimentation (where you judge how many colleagues are producing experiments) is exactly where your attention should go. Turn to it. If you’re only running a/b tests, consider openening up to feature flagging and the whole spectrum of experimentation.

If you can't, adding rigor won't fix it. You have a different problem, and you need to find out what it is before it finds you.

Velocity is not the goal. Learning is.

To be clear, I'm not saying run tests to pad a number. A test nobody reads or decides on is theater, and Manuel's right to call it out. But you don't stumble into theater at two tests a month. You earn it much later, by scaling volume without ever learning from it. Almost no one is close. Chase velocity to find your obstacles, not a scorecard.

Find out where you stand

I built a simple calculator to answer one question: how many tests can your site actually support?

It doesn't measure your rigor. It measures your ceiling. Then you can ask the honest follow-up. Are you running that many? What is it costing you not to?

{{cta-block}}