Expert FAQs: 6 experimentation leaders share their best advice for 2027

{{blue-block-1}}

Brianna Warthan, Associate Director Product Management & Development at AT&T

“Don’t start with the solution; start with the decision you’re trying to influence. Before designing a test, define the specific customer behavior or outcome you want to change, then understand what’s preventing it today. Dig into the data, customer journey, friction points, and underlying problem until you can clearly articulate what you’re influencing and why. Only then develop hypotheses and solutions. Strong CRO isn’t about testing more ideas; it’s about testing purposeful changes tied to a clearly understood problem and a measurable decision or outcome. 

“For example, if customers abandon checkout at shipping, don’t start by testing button colors for order submits. Define the outcome: increase checkout completion. Identify the decision: ‘Is this purchase still worth it?’ Investigate why customers hesitate; unexpected costs, delivery timing, or trust, and then design a test that directly addresses the identified barrier.” 

David Leese, Experimentation & Insights Senior Manager at Dell

“We call it A/B testing, but the real power in testing comes when we add ‘Recipe C.’ The question moves from ‘Yes or no?’ to ‘Which of these?’ and that’s helpful, but the real benefit comes from comparing B with C. B performs better than A; C performs better than A. We look at what we changed, and what the outcome was, relative to Control. 

“But if we compare B with C, we get an extra set of data points. How big should your landing page banner be? Small or large? Or maybe extra large? Two points are a line; three make a trend. Imagine playing Battleship and getting two guesses per turn instead of just one, and you’ll see how just one additional well-designed recipe will increase your optimization velocity by at least 50%.”

Ilya Izrailevsky, Senior Engineering Manager, Experimentation & Metrics at DoorDash

"We've been letting AI write our experiment readouts. What surprised me is where it fails. Not the math. What gets left out. For example, an underpowered test might come back as "no effect," when the honest answer is "we couldn't detect a lift smaller than 2%." Day-3 peeks can read as final. You'll hear about the 2 metrics that moved, not the 38 that didn't. Of 12 segments, only the winner reaches the recommendation. 

“My advice: have AI pull from whatever engine computes your results, put sample size and duration beside every lift, and set ship criteria before you see results." 

Megha Joshi, PhD, Lead Data Analyst, Progressive Insurance

“When running A/B tests and answering causal business questions, randomization is considered to be the gold standard. However, randomization will get you good causal answers only if it is implemented properly and if certain assumptions hold. Really pay attention to what exactly the units are doing after being randomly assigned to treatment or control condition. Does everyone experience the treatment? Do they not comply? Is everyone in the condition experiencing the same thing? Is there contamination? Really think about the basic assumptions of causal inference to see if your test is validly answering the right question. 

“Although randomization is the gold standard, it may not always be practical, feasible, legal, or ethical to randomize certain portion of the eligible population to one experience versus another. In such cases, there are many amazing and honestly fun quasi-experimental designs like difference-in-differences, synthetic control, propensity score matching that can help you answer your causal questions.” 

Nils Stotz, Head of Product Experimentation at Zalando

"The point of experimentation and conversion rate optimization is to give your organization fewer expensive mistakes. If experiments and AB Tests are not changing what your organization builds or funds, you're just doing theatre. Your product is not the experiment itself but a decision that you can defend. So if leaders are not asking for evidence before big decisions, then your experimentation program has not really landed." 

Carol Nogueira, former Senior Director, Ecommerce at Neiman Marcus., Founder, Bad Entrepreneur

"One of the most common mistakes in conversion optimization is testing too many variables at once. It feels efficient, but it makes results impossible to attribute. On a recent client project, a single change to the call-to-action copy on a product page outperformed a complete redesign.

When multiple elements are changed, their effects are conflated. A change that would have increased conversions can be offset by one that decreases them, producing a result that looks flat or insignificant, even when real shifts occurred in both directions.

“Isolating one variable at a time helps you understand what works, what doesn't." 

Start with the decision and keep your tests disciplined

Together, these expert tips paint a holistic picture of how we can approach experimentation going into 2027: that you should start with the decision you’re trying to influence, not the solution. Try moving out of “yes” or “no” answers by adding a C variant to make more detailed comparisons. Get test evidence before making big decisions. Finally, don’t skew a test by making too many changes that might cloud results.

Thank you to all participants for your valuable insights. As someone who jumped onto this CRO adventure back in 2003, when it was truly just beginning, it’s been an honor to see how much our field has grown and watch this new generation of leaders take it to the next level.

{{cta-block}}