Keys to a world-class experimentation program

This is the 3rd of 4 articles covering Conversion Jam 2021, the largest conversion and growth event in the Nordics.

We all know that organizations that experiment grow much faster than those that don’t.

However, there is an enormous gap in ROI between experimentation leaders, who see parabolic growth in their revenue as they optimize, and those who “just try it” and get frustrated with their results.

While the premise of experimentation—constantly be testing—is opposed to fixating on best practices, we can point to things that make some teams and organizations world-class experimenters.

On Nov. 23, Conversion Jam held the second of four online sessions with experts in growth experimentation, entitled Keys to a world-class experimentation program.

The session featured:

• Nir Eyal, Bestselling Author of "Hooked" and Behavioral Design Consultant
• Hazjier Pourkhalkhali, Chief of Staff, Global VP, Strategy & Value at Optimizely
• Martin Richardsson, Head of E-commerce at NetOnNet

Test with more variants and complexity to get breakthroughs

Break out of your patterns. A lot of people are afraid of diminishing returns in experimentation, believing the low-hanging fruit will be plucked soon after they start testing. However, this belief is false—there will always be tests to pursue. Instead, the uplift that organizations get from their experimentation tends to stay the same over time—it doesn’t diminish or increase.

To get better over time, change the way you work. Most people stick to running simple A/B tests with low complexity of code. E.g. Changing text, optimizing copy, replacing images, moving buttons, etc. When you stay stuck in that mode, you stay stuck in that performance.

Experiment with more variants, take more risks. The more variations people test, the more psychological safety they feel to take risks. The fact experimentation is generally called “A/B testing” has a deep effect. “Fewer than a quarter of all experiments run around the world run 3 or more variations,” noted Hazjier. However, the more variants people test, the more likely they are to succeed and they are more likely to achieve breakthrough insights.

Development resources are crucial for increasing test complexity. Data shows that the more complex a test is, the more likely it is to drive a significant uplift. “When starting out, use your initial successes to build a business case to get more engineering capacity to tackle larger experiments,” said Hazjier. “Build the confidence, track record, and psychological safety to go after bigger and bolder ambitions."

Use your wins to build a case for more experimentation. “We’re in a position where most of the company understands and has seen for themselves the results of a good A/B testing program,” said Martin. “In the last year, it’s really been resultful, and that really makes it easier to ask for the resources you need.”

Be less distractable for more efficient experimentation

Only 10% of our distractions are caused by external triggers. Ninety percent of the time you get distracted is not because of what’s happening outside of you, but because of what’s happening inside your head. I.e. an internal trigger—an uncomfortable emotional state we seek to escape from. E.g. loneliness, fatigue, anxiety. Recognizing these triggers is the first step to stop being distracted.

Anything that gets in the way of what you plan to do is a distraction. In the conversion rate business, we’re busy with many things we need to get done, a mile-long to-do list. “We have to make sure we recognize that anything that is not what you plan to do is a form of distraction—even the work-related stuff,” said Nir. Checking your email or a Slack channel is no better than going off to play video games. In fact, it’s worse because you don’t even realize you got distracted.

Blaming the distractions outside of you is pointless. “You can’t go back to some mythical age before these things existed—distraction has always been with us.” Being a “blamer” doesn’t make much sense.

Don’t identify yourself with “being busy”. By identifying yourself as busy all the time, you’re driving yourself crazy—you never get to enjoy leisure. You always feel like you’re running behind a never-ending to-do list.

“Indistractable” people recognize the problem. Distractable people constantly get distracted by the same things again and again. Instead, say, “Ah, I see what you did there. I got distracted once, but I’m going to do something today to prevent getting distracted tomorrow.”