About me

Ronny is a consultant who has held executive roles at Microsoft, Airbnb, and Amazon.

During his tenure at Microsoft, he launched the Experimentation Platform team, enabling testing across 15 Microsoft product groups. Ronny has co-authored several papers on A/B testing, which have received over 42,000 citations, and his three machine learning papers are ranked in the top 1,000 most-cited papers in Computer Science. He is also the developer of fifteen patents in Machine Learning, Visualization, and Experimentation. In June 2024, he released a custom ChatGPT tool, which is based on his previous research and book "Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing”. Ronny also teaches two interactive courses: Accelerating Innovation with A/B Testing and Advanced Topics in Practical A/B Testing.

Q&A

How are you keeping experimentation aligned with company-wide goals?‍

“I now recommend that development teams look for patterns that show strong evidence in the hierarchy of evidence: multiple positive A/B tests. Repositories including Evidoo, GoodUI, and GuessTheTest are great to review.  At Microsoft, we published Rules of Thumb after we ran thousands of experiments.

‍
At the same time, I encourage healthy skepticism with headline‑grabbing massive lifts. Through our community replication effort, we are replicating interesting patterns and would love others to join.  For example, a peer-reviewed article last year claimed that “rounded‑corner buttons” generated double‑digit CTR lifts, but that pattern failed to replicate in our much larger test.. Claims of large effects should trigger one to call Twyman's Law and raise a healthy skeptical view, so we can develop trustworthy and reliable patterns and understand when they work.”

‍