Why your experimentation program needs fewer tests

Transcript

Welcome to Unite Voices

Katie Green: Welcome to Unite Voices, featuring Ilya, Makram, and me, Katie Green, your host. I'm the Principal Advocate at Kameleoon, which means I'm responsible for our online community and creating a space for experimenters to learn from each other. I'm joined by Makram and Ilya, and I'll let you both introduce yourselves. Makram, why don't you go first?

Makram Mansour: Thank you for hosting me. My name is Makram, I'm Head of Marketplace right now at ID.me. Before that, I was managing experimentation at Intuit, the Intuit Experimentation Platform, and you're going to see a lot of common ground between me and Ilya on that. And before that, I was at LinkedIn managing T-Rex, LinkedIn's experimentation platform.

I'm very passionate about experimentation and personalization, so we're excited about this session today.

Katie Green: Ilya, tell us who you are.

Ilya Izrailevsky: Definitely. Thank you, Katie, for inviting me onto this podcast, and I'm really excited to share the panel with Makram.

I lead DoorDash's experimentation platform currently, but I've been on both sides of experimentation, both using it and building experimentation systems. I actually built the first version of Intuit's experimentation platform across TurboTax, QuickBooks, and Mint.com, and later led an effort to open source it, so that was one of the first experimentation platforms available on the market. Then I worked on later versions of that platform, led Robinhood's experimentation platform, and now DoorDash's.

But I've also been on the other side. I've used experimentation at companies like PayPal, Uber, and Amazon. What I've learned is that no matter what size of experimentation you're running, we all benefit. We should all be using experimentation, because it's just good for the growth of your business.

A Small but Mighty Community

Katie Green: A hundred percent. And you both know each other, that's why this is our first three-person Unite Voices episode. For anybody watching who's thinking, “I'd really like to do that, but I want to bring somebody,” we can have as many people as we want on the show, that's the beauty of it. The experimentation community is pretty small, but you two have worked together in the past, right? I want to make sure I'm understanding that correctly.

Makram Mansour: We haven't actually worked together, I don't think we were overlapping, but like you said, it's such a small community. We're connected on LinkedIn, we've chatted a lot of times. Ilya was at Intuit before me, managing the early version of experimentation there. People I worked with, like on Credit Karma's Darwin experimentation platform, even at Intuit there were two versions, Darwin for experimentation, and that person is now under Ilya's team. It's a small world, and that's the beauty of the experimentation community.

Ilya Izrailevsky: Exactly, to your point, we didn't directly work together, but it seems like we did. Throughout the years, we used each other's work, or worked with folks who were in the community. What I've found is experimentation is like the smallest big-tech function. The community is tiny, there are only so many experts in experimentation out there, but our work is used throughout companies, right? It's an essential part of big tech companies as well as small startups and mid-sized companies.

We all share our learnings about experimentation, and I'd say we agree on maybe 80% of things, p-values, variance reduction techniques, experimentation methodologies. Then we argue over the 10 to 20% of corner cases. But our goal, no matter where we go, is to build the best platform as well as the best culture of experimentation.

Untangling the Spaghetti of Multi-Platform Tech Stacks

Katie Green: Yes. I think that's what I want the audience to take away, that I didn't even realize you two never overlapped, but it's all in the spirit of experimentation to work together to make things better.

This question is for both of you, because you've both led such huge, high-stakes programs. A lot of people run into inheriting different tech stacks, Ilya, acquisitions happen all the time, and Makram, you've built programs from the ground up. I'd love to hear more about the challenges or mistakes you often see, related to what it looks like when you inherit a really difficult tech stack, and what your first step is in what I like to call “untangling the spaghetti” of multi-platform experimentation practices. I know so many people listening probably have a tool for ideation, a tool for build, a tool for analyze, and they're hosting everything in Airtable and then putting it all into Google Sheets. Tell us more about the technical side of what it looks like to untangle that spaghetti, whether you're inheriting a complicated tech stack or starting fresh.

Makram Mansour: Ilya, please go ahead.

Ilya Izrailevsky: Sure. At DoorDash, we have a pretty mature platform that we've been building for six to seven years now. But DoorDash has been acquiring companies around the world for our global growth, expanding both our marketplaces and domains, going beyond restaurants into grocery, electronics, retail, and clothing deliveries. For our global growth, we acquired a company called Wolt, based in Finland, and Deliveroo. They also had their own experimentation platforms.

What I found is that it's important not to just rip those out, because that destroys the local trust those companies have built by doing a great job in their own marketplaces. Wolt and Deliveroo, for example, are really great at localization, for local languages and custom features specific to those countries and regions. What we can all agree on to start with is the methodology and metrics that should be used across the board.

For a delivery marketplace, across consumers, dashers and couriers, and merchants, we all agree that for consumers we want the best quality, reliability, and order satisfaction. On the courier side, we all agree we need the best earnings for couriers, along with metrics like their utilization, making sure they're not sitting idle, and fair distribution of deliveries so everybody gets their share. Finally, on the merchant side, restaurants and store owners, we focus on their order volume, the unit economics of the specific items being delivered, and the longer-term longevity of their businesses, not just short-term wins.

We let local brands go in and iterate on those common metrics. Only now, after a few years, are we actually consolidating and building a unified platform, but we always start with the fundamental foundations first.

Katie Green: I love that, I want to make that part of the subtitle of what this episode is ultimately about. Because I often see, in my own career, that when you come into a new program it's really easy to blame it on the tools. It's very easy to say, “that's the problem.” But alignment on methodology is so important. Makram, in your experience, do you tackle the same problem first when you're inheriting complex tech stacks?

Makram Mansour: Yeah, exactly. Like I mentioned, Darwin. Intuit acquired Credit Karma, and Credit Karma had Darwin as their primary experimentation platform. We wanted to make sure we weren't disrupting Credit Karma's business, if it was working and the team was happy with that platform, great, we wanted to make sure it stayed that way.

Then we started to integrate and do more cross-org work between Credit Karma and TurboTax. You start to see a lot of connectivity there, experiments that run across both, either starting in Credit Karma and driving new acquisition to TurboTax, or vice versa. That requires the tools to handshake with one another, and that's where the discussion started between the Darwin platform team and the Intuit experimentation platform team, how do we make that happen so people aren't launching two experiments and flip-flopping between control and treatment across two different variants?

So you can see, each one has its own tech stack, and that's where the tricky situations start to happen. And obviously there are other factors at play too, when a company is combining technologies and tech stacks after an acquisition.

Building Trust and Culture Across Teams

Katie Green: You both have such specific experience in this part of the field. I'm really interested in something a lot of people struggle with, which is how you share this information and build trust. You're saying you align on methodology, but how do you build trust with such a large team when people are coming from such different points of view? In my experience, people feel very passionately about experimentation, which is incredible, but what do you do when somebody says, “I don't agree that's the right metric,” or, “I actually do think it's the tool”? I'm curious how you build trust and a culture of experimentation with your teams.

Ilya Izrailevsky: The way I like to approach it is, first of all, through learnings. It's not really about the tools, it's about the people and processes using the tool. It's really important to first enable experimentation and show, whenever somebody new joins, here are the experiments that have run in the past. We have a culture where all experiment readouts, whether wins or losses, are sent out across the company, and anybody who wants to understand what happened can read it and learn from it.

The second layer is using AI to surface what's called institutional knowledge. We're using AI to mine all our past experiment readouts, so you can type in a query like, “show me all the DashPass experiments that moved our global order volume by more than 0.2%,” and you'll see all those experiments and can dig further into the details.

And last but not least, having leadership and executive buy-in on reviewing experiments and replying to them. Our CEO at DoorDash replies to experiments and encourages people to look at alternative ways of approaching the problem. That builds trust. So no matter what culture of experimentation you came from, you should be able to understand the past, understand what's happening now, and provide your own input and feedback. Our goal, no matter what experience you're coming from, is to build the best-in-class experimentation that really works for DoorDash and our global businesses.

Makram Mansour: I'll give you a few examples too. I entered the world of experimentation at LinkedIn when I was a product manager in developer productivity, and I got the opportunity to manage T-Rex, LinkedIn's experimentation platform, working under Ya Xu, who co-authored the book “Trustworthy Online Controlled Experiments” with Ron Kohavi. Once I became PM for T-Rex, I started getting bombarded, left and right, with feature requests from different teams, and I started seeing how important experimentation was to all these various teams. AI engineers, data scientists, back-end engineers, all wanted in, back-end engineers needed it because of feature flags and cleanup work.

I had to develop a prioritization framework, because everybody wants their feature yesterday, and we were a small team. I actually did a Product School talk specifically about prioritization, because the demand on experimentation teams was abnormal.

Then I moved to Intuit, hoping for less demand, but it turned out to be even worse. I was dealing with B2B use cases for QuickBooks and B2C use cases for TurboTax. On top of that, you have web authors, CMS editors, designers, front-end developers, and back-end developers, all with different needs from the experimentation platform. Like you said, it's easy to blame the tool.

I was managing the experimentation platform, the personalization platform, and the CMS platform, so it was easy to start pointing fingers, saying these tools don't talk to each other. But experimentation is just one piece of the puzzle. If I'm running a pre-authenticated marketing website experiment, that touches the CMS platform and the experimentation platform behind it. So who owns that, the CMS team or the experimentation team?

To tackle all of this, I even organized a Go-To-Market Tech Summit, an annual event where we invited all the key stakeholders from Australia, from London, and got everyone together in Mountain View for a week to talk through their issues. We planned it ahead of time, so a week before, people came prepared with the positive things they wanted to say about us, so it wasn't just, “I hate this feature, I hate this feature.” Instead it was, “something is working, tell us what's working,” and then, “come back with the things you'd change, and if you have twenty, pick three, and let's see how we can tackle them.” That really relieved a lot of tension. Sometimes these human interactions are very important.

Katie Green: I think that's an interesting juxtaposition in your answers. Makram, you're talking about bringing people physically to the table, and Ilya, you brought up leveraging AI to democratize insights, something I talk about a lot. I think both pieces of the puzzle build trust.

Where AI Is Adding Value Today

Katie Green: I realize this is a bit tangential, but I want to make sure I ask it. Makram and I talk about AI all the time, Makram is a strategic advisor to Kameleoon, for anyone who didn't know. We're working on a PBX 2.0 series together, and have been for weeks now, so I know how Makram feels about where AI fits into the workflow, but I may have you repeat yourself on this podcast, Makram, because I think what you have to say is really important. Ilya, I'm curious, where are you leveraging AI, where is it providing value, and where do humans still fall in the loop?

Ilya Izrailevsky: Yeah, I'd say we're just starting on the AI journey right now. We've launched what we call an MCP server for experimentation, and it's been going viral, people have been building their own skills for the various stages of the experimentation lifecycle. We're trying to vet and contain it, to make sure people aren't cutting corners and are still using the established standards of experimentation. But we have a skill, for example, to create an experiment or a feature flag. We also recently built out an experiment summary and readout skill, which summarizes all the experiment analysis results, because it's really hard to understand what happened across experiments otherwise. We've also been building a background AI agent that scans your repositories and looks for stale feature flags or experiments you should remove, dead code, essentially.

What I'm realizing is we need to build a lot more skills throughout the experimentation lifecycle. At the design stage, for example, you no longer need to wait for an experimenter to create a Google Doc with their definition, they can design their experiment right there with the AI agent, working out the hypothesis and what metrics should be used. We're also using background agents to debug experiments that have gone wrong, sample ratio mismatch issues, or anything else that might need a restart. So we've been introducing AI throughout the experimentation lifecycle, but we still want a UI available where people can look at experiment analysis results, even as more and more people are just using the agent command line to interact with their experiments.

Katie Green: Before I let Makram go into the spiel I know he's so good at, isn't this funny? It sounds like Ilya was on our call this morning, in terms of looking at stale feature flags. You and I both went there this morning, because that's something we were talking about, since our PBX 2.0 is launching.

By the time this episode is out, it will have already launched, spoiler to everybody, this was pre-recorded, if you didn't realize. A huge part of PBX 2.0 is making sure you're acting on your test winnings in a reasonable time, so you don't end up with stale feature flags or stale learnings. I think it's hilarious, we're going to send you what Makram and I recorded this morning, and you'll hear a lot of the same terminology, because the whole thing was about AI. Makram, tell us a little more about where you think AI provides the most value in the experimentation workflow today.

The Three Phases of AI in Experimentation

Makram Mansour: I see three phases of AI hitting us right now. Phase one, just like Ilya described, is streamlining the end-to-end workflow from ideate to shipping, removing dependencies. If you think back six months or a year ago, we had two bottlenecks. One was developers, because every experiment needs developers to build and set it up, there's a lot of setup time required. And there was launch time after we had a winning experiment, getting it out the door, that's why we end up with all these forever-running experiments, because nobody has the bandwidth to turn them off, clean up the feature flags, and do it properly.

So there's a bottleneck on engineering, and another one on analysts doing the readouts, not just the readout side but also experiment design, because if an engineer doesn't set it up properly, it's garbage in, garbage out, and the experiment isn't valid. So how can we remove these bottlenecks with AI and democratize experimentation, so it's not only experts running experiments? Ideally, the tool should be intelligent enough that anybody, a designer, a marketer, a non-technical PM, can go run it. A designer coming from a Figma design should be able to just push that design into a prompt and get it up and running, because the whole point of experimentation is, let's test it out. I get into so many meetings where people say, “this is going to take a lot of energy,” and I have to say, “I'm not asking you to go to production yet, I just need to test it to see if it's working, and then we'll talk about that piece.” That's the beauty of removing the setup time, launching quickly, getting the analysis, and if it's winning, shipping it out with no dead time. That's phase one, in my opinion.

Phase two is the traffic bottleneck. Unless you're DoorDash and have the luxury of homepage traffic, we still need to be able to test everywhere, not just the homepage. The teams owning different, lower-traffic components won't have a lot of traffic to work with, so how do we enable them to do other flavors of A/B testing, maybe not online controlled testing with full causal rigor? One thing I was doing at Intuit was bringing them the toolkit for early testing, because early testing is better than no testing, at least it gives you a better data-driven decision.

The third phase is synthetic audiences. There's a lot of debate right now about whether synthetic audiences are scientifically proven, but if you're caught on time and only have a small window, take TurboTax as an example. All year long, once taxes are done, how much traffic is going to hit TurboTax.com? Not too much. Then from January through April, that's showtime. We don't want to be running a lot of bad tests during that slow season, so how can we leverage synthetic audiences then, to generate good ideas, so that when it's showtime in January, we're ready to run those tests. That's error number three, in my opinion, if you're not doing that.

Katie Green: Love this, “showtime,” I can imagine that really is showtime. We could spend another thirty minutes talking about AI and the future of experimentation. I'll leave us with a quote I saw on LinkedIn today, from Johnny Longden, who said A/B testing won't survive AI, but experimentation will know the difference. I loved that, probably true, so interesting, we could talk about this all day. I know there are people listening looking for your leadership on everything, but at least for now they've heard how to navigate really complex tech stacks, how to build trust, and where some of the leaders in the space see AI going, and where they're using it now.

Closing Advice: Be a Decision System, and Embrace AI

Katie Green: I realize we're coming up on time, so I want to ask the question I always end every episode with. If somebody listening right now is overwhelmed, with a team that isn't quite understanding where they can create simplification, because I think that's what you both do best as leaders, take a very complex issue and make it as simple as possible to create more impact, whether that simplicity is just for the receiver, whatever it is, what advice would you give to anybody who wants to create more simplicity in their program, whether it's tech stack, trust, metrics, whatever it is? What would you recommend they do tomorrow?

Ilya Izrailevsky: I can start. What I've learned throughout my career is that earlier on, I focused on scalability. I kept asking, how can we run more and more experiments, quantity felt like the goal, because the more experiments you run, the better things will probably be. But what I learned is that it's not really about quantity, it's about quality.

You can be running a lot of different experiments, but if nobody is really looking at the results, and it's not changing your initial intention of what you're going to do, you can just look at the results and ship your feature anyway, even if it's great on your guardrail metrics. So instead of pursuing some magic number, 10,000 experiments, 20,000, 100,000, focus more on decision quality. How many of the original calls that you were going to make did you actually make? Look at experimentation as a decision engine. That's why my group at DoorDash actually calls itself Decision Systems. We're not just in the business of running experiments or enabling feature flags, we're in the business of driving decision-making. So full focus on decision-making, not on pure reporting of experiments for reporting's sake.

Makram Mansour: I love it, I love it. You talk a very similar language, the more I listen to you, Ilya, the more I realize how much we have in common. These vanity metrics that we typically chase, I've been in so many of those meetings, “okay, what are we trying to achieve here, what are we doing here?” I love it.

From my side: embrace AI. We live in a different world now. If you think about the Venn diagram where a PM only does PM work and a designer only does design work, that's coming together. At ID.me especially right now, engineers are writing product PRDs, they're not waiting for a PM, if that's something they're capable of, let them do it. We're democratizing, everybody can write content, everybody can now launch an experiment, with the right checks and balances. Engineers aren't the only ones who can check in code anymore, because of pull requests and code review, and now everybody's using AI tools and can check in code, with the right rules and an agentic architect and agentic PM working alongside them, streamlining all of that. Those old habits are the difficult thing to let go of, but if you can get rid of them and embrace the new world, that's my advice.

Wrap-Up: Decision Systems, Humans at the Helm

Katie Green: Wonderful advice from both of you, be a decision system, and embrace AI. I think you can use AI to streamline your decision systems. Also, for anybody watching on YouTube, you're getting a really good view of my cat, who just muted me. Yeah, my cat is truly laying directly on my keyboard, it's fine.

I'll recap, which is just great feedback for everybody. I think there's a lot to learn from both of you. I wish we could have covered every topic we wanted to in thirty minutes, but it's just impossible, there's so much you both could say. I'd encourage anybody following along to find you on LinkedIn, and if you have a question, ask them. Really good leaders to learn from, leveraging AI and being a decision system, really important pieces of the puzzle. I'd add, be cautious of how you're leveraging AI to make decisions. AI is an amplifier, not a strategy, and you are the strategy that employs AI. So bringing those two together, I'll bridge it there.

That's a perfect way to pick up tomorrow and start. Thank you both for your time today, I really appreciate you, and I hope if anybody's watching on YouTube, they're getting a good view of my cat. In the meantime, we'll see you next time, thanks for tuning in to Unite Voices.

Makram Mansour: Thank you for having us.

Ilya Izrailevsky: Thank you so much. And as they say, human is always at the helm, right? AI is the system, but the person, the human, should be making the final call.

Katie Green: Oh, well, there's the title of the episode: Decision Systems, Humans at the Helm. I love it, so great. Thank you both.

Makram Mansour: Thank you.

Ilya Izrailevsky: Thank you.