How to say yes to a bad idea

At some point in the career of every testing specialist, there’s going to be an awkward moment: that bright Monday morning when an excited coworker offers a testing idea that you just know, with absolute certainty, is a bad one.

What to do? You can spend a testing slot on an idea you don’t believe in, or you can be “that person” who says no to an idea just because you don’t believe in it.

In a recent episode of Unite Voices, Linda Bustos, a UX expert who runs Ecom Ideas, offered a third option: lower the cost of being wrong until the argument no longer has weight. 

One scrolling ticker with two wins

Bustos’s perspective comes from a career filled with tests that performed both as and not as expected. 

“I think sometimes things that look like bad ideas actually win,” Bustos comments to Katie Green, the podcast’s host. “I remember pushing back on a team that wanted to put a scrolling ticker at the top of a page. I find those incredibly annoying, and I never would have thought it would win.

“But it did win the test, having a scrolling ribbon across the top.”

And then? “Then we did another round of testing shoving emoji into it, and for that particular brand and customer, that was a second round of lift.”

This is not an argument against expertise, of course. Bustos is a UX specialist with nearly twenty years of experience, so her objection had weight in the moment. And it goes both ways too; she notes herself that “the things you think are really great can also fail when you test them.”

Green said she knew the feeling. She herself has worked with experimenters who have been entirely confident in a result that she knew would not work because she has “thousands of tests in [her] backlog that they don’t have that context for.” That experience is evidence, but is not enough reason on its own to skip the test.

Three ways to lower the stakes

Bustos offered three ways to mitigate risk in a cost-effective way:

• Pick a page that can afford to lose when the result is uncertain. “You can test something on a PDP that gets medium traffic,” Bustos notes. “Not your best-converting page, not your hero product, but something with enough volume to get a test through without introducing too much risk.” You trade a little statistical power for less exposure. If the test wins, you can try a follow-up test on the hero page, rather than a site-wide rollout.
• Throttle exposure by sending only one in five visitors to the test. This will mean a longer runtime and much less risk. You could also use dynamic traffic allocation to automatically shift visitors away from a losing variation as the test is still running.
• Defer and deprioritize when necessary. Bustos suggested that “you can also say something is deprioritized because you’re busy with other things or defer the decision to later.” Green agreed too: “let’s think about that tomorrow” is her favorite approach to the same problem.

No can still be the right answer

This isn’t to say you should never say no. Not everything can be put into a test, and sometimes the build cost is too much. Many organizations depend on developer sprints for their A/B tests, so Bustos asks:

“Is this going to be enough of a lift to warrant putting it into a development sprint, possibly knocking off something else important or urgent?” Opportunity cost is always a consideration. 

Other times, the brand is set in stone and any test that contravenes those guidelines, even slightly, needs rejection. 

And according to Bustos, that can be the hardest part of the job: “letting a good idea go, or managing multiple expectations and maybe diluting the strength of a hypothesis in order to satisfy everybody.”

Still, she prefers to keep an open mind because there are good ways to test “bad” ideas that may surprise the tester. Transparent deferring, traffic throttling, and careful page selection mean she can keep that open mind without risk.

{{cta-block}}