What years of experimentation have taught me: five lessons every team should know
In my role, I get to work with testing teams across industries: marketers, PMs, CRO strategists, engineers, data analysts. Some are just getting started. Others have run thousands of experiments. Most are somewhere in between. When you spend enough time troubleshooting programs and helping teams grow, patterns emerge.
You start to see why some teams scale and others stall. You see where testing gets blocked. Sometimes it's the tech. Sometimes it's trust. Sometimes it's turf wars. And you see what the best teams do differently.
Here are five truths I’ve learned from that work. They’ve helped teams move faster, collaborate better, and rethink what it really means to be experiment-driven.
1. Solve stakeholder problems, not yours
The biggest mistake experimenters make when working with stakeholders is focusing on their own problems—or even the customer’s problem—without first understanding the stakeholder’s.
If you want buy-in, start by figuring out what’s blocking your stakeholder. What metric are they accountable for? What decision are they struggling to make? What outcome are they under pressure to deliver? Don't call them disparaging names like, HiPPO.
Then treat that like you would a customer problem. Find it. Define it. Measure it. Test your way toward a solution.
When you apply the same experimentation mindset to internal alignment as you do to customer experience, things start to click. You stop pitching experiments for their own sake and start offering real help.
Bonus: There is nothing an executive likes more than seeing teams work together to solve shared problems. Do you know how your product work is affecting marketing and vice versa?
2. Quantity beats quality, at the beginning and the end
In the early days of a program, quality doesn’t matter as much as you think. You need reps. You need feedback loops. You need to learn how to test by testing often.
And when your program matures? You’ll still need volume. But the purpose changes.
The best teams stop seeing experimentation as a UX tool and start seeing it as a business discipline. They don’t just test button colors. They test shipping logic, sales scripts, support workflows, even recruiting emails.
Everything becomes an experiment because everything can be improved. That mindset shift turns experimentation from a tactic into a company-wide habit.
3. Not all experiments are A/B tests
Most people equate experimentation with A/B testing. But A/B is just one method. Progressive rollouts, targeted feature releases, server-side flags—these are all experiments too.
When companies say they run thousands of experiments a year, they’re not running thousands of A/B tests. They’re counting every rollout, every flag, every targeted feature. And they should. Those are experiments.
The best companies are test-type agnostic. They let teams experiment in the way that makes the most sense for their skillset and tech stack. What they don’t allow is isolation. Every team is expected to understand how their tests affect other teams’ outcomes. That’s how they move fast without stepping on each other.
4. Tell stories, not just stats
Yes, the science matters. But if you can’t explain how your test changed user behavior in plain language, no one cares.
Executives don’t remember p-values. They remember people, you, their customers, and their competitors.
Tell the story of what the user was trying to do, what you thought would help, what you changed, and what happened. If you want support, you need to make your results memorable, not just measurable.
Bonus: Explain how your competitors address the opportunity/challenge you’re discussing. That keeps them listening.
5. The future of testing is not vibe coding; it’s “vibe experimentation”
AI isn’t a magic wand emoji in a visual editor. It’s “vibe experimentation” or “prompt-based experimentation.” You describe the experiment. The system builds it. You refine. It launches.
Prompting AI is faster than coding from scratch, more flexible than dragging UI elements, and more aligned with how modern teams actually work.
You don’t need to wait. You just need to start prompting.
If you’re building or scaling an experimentation program, keep this in mind. Experimentation is not one tool. It is not one team’s job.
It is a way of working. One that helps all teams learn faster and grow together.
