What you should never do in your experimentation program, according to these 8 experts

In today's data-driven world, experimentation is essential for businesses looking to optimize their products and services. However, not all programs are created equal, and while the internet is full of how-to guides, few cover what not to do. 

We asked eight experts to reveal what you should never do in your experimentation program. These experts have seen it all and there is a lot to learn from their wisdom - here are the top five things you should never do.

1 Never get blindsided by test outcomes

Companies may have different objectives when it comes to experimentation. Ultimately, however, testing can provide direction and insights into important business questions.

But, to use experimentation in this way, you need the correct focus.
 

,

Don't focus on the final outcome or obsess over finding answers. The role of a well-defined experimentation program is to find the right question. Only our users have the answers. If we don't ask the right question, we won't get the right answer.

Marcello Pasqualucci

Head of Experimentation at Sky

,

Annemarie Dooling, VP of News Experimentation at McClatchy explains how to find the right questions.

“Never use the word failure. Having an experiment end as a "failure" means there was an assumption prior to experimentation that you were trying to achieve rather than leading with questions," says Annemarie.

,

Starting with a hypothesis leaves room for a project completion you did not expect that could still lead to many outcomes, including the successful shuttering of an idea that isn't so useful anymore. And sunsetting is just as important as launching. Every experiment has a lesson to share, even if it's not the one you thought you'd see.

Annemarie Dooling

VP of News Experimentation at McClatchy

,

2  Never test without data

Early-stage experimentation programs often start by testing the team's own ideas. For example, changing the button color to blue will increase conversions.

But this falls into the trap of not asking the right questions (see above). After all, knowing if customers convert better with a blue button isn’t as valuable as knowing what features your customers are willing to pay for.

So how should you identify the right questions? One word; data. Mohtashim Hashimi, Lead Product Manager, Growth at Zenefits explains what you're looking for in the data.

,

You should never run experiments unless you're aware of the friction points in the customer journey.

Mohtashim Hashmi

Lead Product Manager, Growth at Zenefits

,

3   Never test without the right goals in place

A common theme mentioned by many experts was the need to have clear goals. Without goals, it's impossible to know if you've been successful. 

Aditi Gupta, Senior Product Manager at Thomson Reuters, shares how you should set your goals. 
 

,

Never start your experimentation program without clear objectives. The assumptions which you test might be ambiguous and unknown, but the outcomes should be aligned to the company goals.

Aditi Gupta

Senior Product Manager at Thomson Reuters

,

Mohtashim goes on to share how you should define important metrics connected to any goals or objectives used in your program.

,

You should not define output metrics unless the input metrics that will be impacted by the experiment are clear.

Mohtashim Hashmi

Lead Product Manager, Growth at Zenefits

,

4   Never experiment without a rigorous process  

What’s the point of testing if you can’t trust the results? Sandip Amlani, Global Optimisation Manager at Cisco says you should never run experiments unless you can execute them rigorously. 

“Experimentation is a scientific methodology that, when used correctly, provides a very effective way to make decisions," says Sandip.

,

Being a scientific method, it requires a high level of rigor and expertise around the accuracy and use of data. Otherwise, you might end up running confidently in the wrong direction. The strength of experimentation can then become a weakness, with misplaced confidence in a scientific method that is let down by poor execution.

Sandip Amlani

Global Optimisation Manager at Cisco

,

Anjali Arora Mehra, Senior Director - Analytics, Data Science, Experimentation & Instrumentation at DocuSign, shares what a rigorous process looks like. 

,

One thing you should never do in your experimentation program is run a test without any planning. You should never launch a test without a Test Plan/Charter.

Anjali Arora Mehra

Senior Director at Docusign

,

"A strong hypothesis and definition of a primary KPI are absolute essentials to a successful experimentation program," says Anjali.

"You may think it would slow you down, but in reality, it takes much longer to drive any meaningful insights and learning without a good understanding of why you think an idea/feature may work and how you will measure the impact.”

Diving further into what elements of your experimentation program needs to have a rigorous process, we find ourselves at test statistics.

Bartosz Mozyrko, Senior Product Manager at GoDaddy, explains what you need.

,

It is not uncommon for experiments to show a low p-value in their early stages, only to become statistically insignificant as more data is collected. This is why, in most cases, "P-hacking" is not advised.

Bartosz Mozyrko

Senior Product Manager at GoDaddy

,

"When the p-value is checked during an experiment, and the experiment is concluded right after it falls below alpha (e.g., 0.05), it violates the underlying assumptions of the controlled A/B experiment," says Bartosz. "As a result, the p-value becomes unreliable, leading to results that cannot be trusted." 

"To prevent this, it's best to estimate the required sample size before starting your experiment. Let the experiment run until that sample has been reached. And don't forget to write down your estimate somewhere (e.g., Jira ticket or Confluence page) as a reference for following experiments!"

5   Never slow down decision-making by doing too much at once

Once you have all of the right inputs and processes in place, pressure can mount to run more and more tests. While a rigorous process can help you run multiple tests simultaneously, you must be mindful of running too many at any one time. 

Pius Binder, Product Manager at Canva, explains why.
 

,

My biggest learning after being part of 1000 experiments is don't do too much at the same time. Never go beyond too many variants and parallel streams of experiments to make sure decision-making can be fast.

Pius Binder

Product Manager at Canva

,

The five things you should never do

Here’s a neat little summary to remind you of what not to do when it comes to experimentation. 

1. Never get blindsided by test outcomes. Focus on asking the right questions by crafting solid hypotheses. 
2. Never test without data. Find the friction points in the user journey. 
3. Never test without the right goals in place. Experimentation goals should align with company goals. 
4. Never experiment without a rigorous process. For example, before starting any test, ensure you have a test plan which sets out the sample size. 
5. Never slow down decision-making by doing too much at once. Ensure you don’t run too many tests simultaneously, making analyzing the results challenging and time-consuming.

These five points provide an excellent cheat sheet for anyone running a testing program who wants to avoid common pitfalls. Follow the above advice, and you can’t go far wrong.

A massive thank you to the experimentation experts for sharing their wisdom!