What to do when the experiment doesn’t change the decision

How often does this happen? An executive offers an idea they’re positive about. You, being the responsible specialist you are, test it every possible way and it just doesn’t work. You share the results. “Sorry boss.”

Then you come in next Monday and learn the decision went ahead anyway. 

Zach Flynn, who rebuilt Udemy’s experiment analysis process from scratch and is now a pricing and monetization scientist at Fivetran, is familiar with this friction:

{{quote}}

‍

So, when an experiment wins (or loses) and no one’s mind is changed, there are three questions worth asking.

1. Is this a hit you’ve already decided to take?

{{quote1}}

‍

This is an important reframing that Flynn offers: make it about “how to make that pivot good,” rather than whether to make it at all. If your executive has already set the direction, it may be better to start running experiments within that paradigm. 

2. What are you optimizing for that this metric fails to capture?

It’s not uncommon for testing specialists and management to define core metrics differently. For example, most A/B tests measure weeks, but many executives measure quarters and years. Looking at time through two separate lenses can lead to different perspectives on the same result.

Flynn’s suggestion is simple: don’t be afraid of running long experiments. “If the impact is large enough,” he argues, “and it’s a significant business direction, you can justify it.” 

3. Do you object to this specific variation, or the direction it indicates?

Occasionally, a losing variation means having tested a plausible guess over the root cause. So check with your team: where does the objection lie?

{{quote3}}

Decide how you will decide early

The fix is straightforward. Pre-commitment, in the form of one discussion, fixes a lot of these issues. Simply say “if X metric goes up and Y doesn’t go down, we launch.” 

The important thing is that everyone is on the same page about the decision being made, the primary and guardrail metrics used, the smallest effect worth shipping, and what might justify ignoring all of that.

“Otherwise," Flynn points out, "you end up in an ex-post position where everyone looks at the metrics after the fact, squints, cuts them every which way, reads the tea leaves, and lands on a story that usually just confirms their prior beliefs.”

As Katie Green, who hosted the episode, put it to Flynn: “what is our decision-making criteria and, if we’re divorcing from it, what’s the reason why?”

Knowing that beforehand makes all the difference.

Find the assumption you didn’t test for

It’s also important to remember that disagreements do not mean adversarial relationships. Not every override is a failure; often, acknowledging other points of view or arguing your own is a critical learning opportunity. 

An experiment tells you whether your variation worked, but it can’t say that a company should change direction. When the result fails to change the decision, try to figure out what assumption you didn’t test. Find out where you and your leadership are disagreeing. Sometimes, the disagreement itself is where the real learning happens.

{{cta-block}}