Quick links

What separates good vs. great experimentation programs?

Published on

May 31, 2022

Article

Do you belong to a good experimentation program but, in quiet desperation, dream of greatness?

Both good and great programs know how to test well.

But instead of optimizing the process of experimentation itself, great programs ensure that tests are aligned to senior leader business goals and are actionable—according to our panel.

On May 28, Kameleoon spoke to 3 experimentation experts—Shagun Aulakh, Manuel Da Costa, and Ben Labay—to find the secret sauce that allows some programs rise above the rest.

Main takeaway: A great program goes beyond experimentation—it’s aligned with, and visible to, the organization’s key decision makers

Whereas a good team focuses on the details of their tests, a great one is able to take a step back and see the program through the eyes of the organization’s stakeholders.

To do this, they go beyond data, using emotional hooks, to communicate insights that are actionable and fit into broader strategic goals.

They don’t try to be management consultants and remake the organization to fit around experimentation from the start, but adapt their program to realities on the ground.

Finally, they have a North Star metric to keep everybody focused, and an orchestrator who allows them to scale without getting sloppy.

To watch the replay, click here.

How great programs influence change

1. Good programs communicate important insights. Great programs communicate insights in a way that ties back to C-suite goals.

The C-suite has their own goals, their own initiatives, their own KPIs, their own things that keep them up at night. While they’ll sit in a meeting and hear the insights your experiments generated, at the end of the day, it’s just another PowerPoint deck, spreadsheet or dashboard out of the barrage they’re already getting.

Practitioners should think about what C-level people want out of the results, and position their experiments to help solve real problems within the business.

The C-suite doesn't care about the tests. They're not even going to look at them. They're not going to look at the variations and P value and statistical significance. They're not going to care about that. What they care about is how it is going to help them.

Manuel Da Costa

Founder, Effective Experiments

2. Good programs talk about findings. Great programs make findings actionable.

As an experimenter, you have to understand the company and be where your stakeholders are. You need to find out the medium that they prefer to best communicate the insights and actions that need to be taken from your experiments.

Great programs and companies go even further than bringing insights to the leadership team. These product owners and practitioners should not only be bringing insights, they should be bringing the actions that they took with regards to those insights.

Ben Labay

Managing Director, Speero

Note that every company is different, and understanding the best way to reach the organization with your findings is crucial. For example, some companies want insights packaged up in the form of newsletters while others want formal documents. Some like the charts and the graphs and others want simple one pagers.

How great programs deal with silos

3. Good programs complain about silos. Great programs understand that silos will always exist and work around them.

There are silos in every company. Often, changing management and organizational structure is far bigger than the job of an experimentation program, or lone experimenter.

While experimenters have to affect change in a business, great programs know how to pick their battles and work within the silos and reality of the organization. This doesn’t mean throwing in the towel in all cases, but carefully prioritizing which silos absolutely need to be tackled, rather than becoming a management consultant.

I think about how to open better communication and better visibility. And there are some silos that I really do need to break because it's blocking me from moving forward. So, I'll try to figure out ways to do that, but some silos are going to be there and I'm just going to try to move and navigate around them.

Shagun Aulakh

Director Web Experience & Experimentation, Gympass

4. But, some great programs break down silos too.

Some experimenters, like Manuel, lean away from calling it the job of an experimenter to break down silos. He believes that trying to get silos working together or even broken down is an organizational design challenge.

“It needs to be done from the top down,” he said. “It cannot be done from the bottom up, no matter how much you'd love to do that.

On the other hand, Ben believes experimenters, by definition, break down silos.

If you're a program manager, it's your job to make change. And it's your job to increase the CRO case, for example, and increase the speed of the flywheel. How do you do that? By finding the things dragging it down. The data, automatic reporting, it's over in the data team. It's not being automated as well as we could. Let's go to that team. Let's talk to them. Let's automate it. Boom, silo broken.

Ben Labay

Managing Director, Speero

How great programs structure their teams and KPIs

5. Good programs have goals. Great programs have a North Star metric.

Goals, KPIs, objectives, and a well-aligned business strategy is what motivates people. Motivation is defined by having a clear understanding of that object in the future, which you can see. And you have a clear vision of the path to get there. Those are the two ingredients that a great experimentation program needs.Good programs have KPIs to judge the performance of their tests, but the test pipeline isn’t pointed toward a blindingly clear star. Great programs move like self-propelling machines to one well-defined big destination.

“Great programs have a metrics strategy that aligns to what leadership thinks is the North Star” said Ben. “What the leadership thinks is the values that they want to bring to the organization.”

6. Good experimentation teams have managers. Great teams have orchestrators.

Every robust experimentation program has contributors—those providing the inputs and outputs, like designers and development—and managers. The job of the manager is essentially oversight. While, at most teams, this role is a CRO experimentation specialist tasked with management, great teams understand that this manager needs to do more than keep the gears turning.

Manuel says that the orchestrator is more like a generalist who understands CRO and experimentation well, but has the distance to be on the outside looking in, and coaching the team to maintain best practices.

An orchestrator is not a manager. An orchestrator is a different level that understands experimentation, the inputs and the outputs. But this person is able to systematically grow that program by monitoring every single person, every single team and understands how well they are managing the inputs and the outputs.