Building fast beats building perfect: How AI accelerates experimentation velocity

Transcript

Welcome & Introductions

Katie Green: All right, Akash, welcome to Unite Voices. We're so happy to have you. For everybody who doesn't know me, I'm Katie Green, Principal Advocate at Kameleoon. I've been in experimentation for about 10 years, and my favorite one-liner is that I sell experimentation. We're here to talk about the things that make experimenters excited, and one of those things is your experience. So please introduce yourself to the group. I know you were formerly at Delta — let us know what you're up to and why everybody should listen to you.

Akash Doshi: Yeah, appreciate the intro. I've spent the past 10 years working across commerce, retail media, and product management. As Katie mentioned, I spent the past three and a half years leading part of the digital strategy for Delta Vacations, the leisure subsidiary of Delta Airlines, working across AI adoption, product management, and building out the consumer experience.

Katie and I met back in February at a Kameleoon conference with GAIN, and we had a great conversation — so we thought, why not share some of those ideas publicly?

Katie Green: We're a couple of nerds getting our conversation recorded for the internet to have forever, which is classic. It was so great to meet you in Vancouver. I think you've done a lot of these — you're an expert at talking in front of a camera and a mic, and I watched your episode with Quantum Metric. You had some really good points, and I was happy to get the chance to dig into what we talked about as it relates to experimentation and data. Something you talk a lot about is data fatigue and the battle experimenters face because people are drowning in numbers — but data is quite literally our whole life as experimenters. Can you tell us more about that argument? How do we balance too much data versus the right data?

The Battle Against Data Fatigue

Akash Doshi: Exactly. I think that's the great connection with the growing space of AI and experimentation — democratizing technical fluency so normal folks can run A/B tests without needing an entire technical stack or team leading that work. How that relates to data is really twofold.

Ultimately, folks sitting in the business — whether in product, marketing, or customer experience — are just trying to get better answers to questions. We want the right information to understand our next steps: what's our vision, our game plan, how do we solve a critical customer problem or build the next enhancement?

The role data is playing right now is that we spend so much time building infrastructure — think about the applications you host, whether for a B2B or B2C customer experience. We build out these really robust data layers with all these metrics coming in, and in my personal experience, that's led to a lot of data fatigue. We have so many metrics that we often don't know which ones are the right ones to actually get better answers and make better decisions.

When you use that framework, it helps you get rid of the noise and focus on the signal — and that all comes down to finding the right proxies. A lot of times, people are reversing the order of operations. They're saying, "Okay, we have all this data, let's try to make sense of it." I think the better way to go about it is starting with the key customer problem you're trying to solve or the enhancement you want to build, then asking what the right indicators are to measure whether that's a good idea. Then you go back to the data with a much sharper focus on what you're actually looking for. That's the role data fatigue has played, and how we can drill down our focus on what matters most.

Asking the Right Question

Katie Green: I want to zoom in on one piece you just said — asking the right question — because that's what experimenters are so focused on with our hypotheses. Our hypotheses and questions are what inspire really impactful, scalable programs. Tell us more about asking the right question, and why that's so important when it comes to experimentation and data in general.

Akash Doshi: I think that's a great question because it gets into how A/B testing solves some of the issues large teams — or even smaller teams — are facing right now. Let's build a contrived example: think about funnel conversion. An e-commerce site might have an intended flow they want customers to go through. A data and analytics team looks at that step-by-step conversion, notices a drop-off on a particular page or action, and zooms in to understand what's happening at that step.

That could be a variety of things. If it's the payments page, maybe an API is failing, or maybe customers are having trouble adding something to cart. Maybe the experience is confusing, or the UI doesn't make it clear how to proceed. It could also be a behavioral problem — the customer just isn't ready to convert, maybe they're checking prices. There are all these possibilities.

What we talked about in that Quantum Metric conversation was that when we have voice of the customer, we're able to really zoom in on the actual issue — rather than doing a lot of guesswork and extrapolating, we get a very firm indicator of what's going wrong. Then the opportunity for AI and experimentation is to use that as a launchpad: we know the core friction, we've identified the root cause rather than just targeting a symptom. Now, how do we A/B test a solution to confirm we're on the right path? So the first half is voice of the customer identifying the correct problem, and the second half is A/B testing to make sure we have the right solution in place.

Katie Green: Because of how much data expertise you bring to the table on experimentation, I think a lot of us come from different angles — that's something unique to this field. I come from the marketing and growth strategy side, agency-side for most of my career. Others come from dev, data, or design. There are so many different levels to it. But something we struggle with in the experimentation world — and I know you've seen this too — is democratization.

When you have so many different levels of experience with technology and data, it's really hard to democratize a program and bring people in so they can meaningfully make an impact without hitting roadblocks. I know you have a strong take on democratizing technology, not just data, and I want that to be a big focus of this episode, because that's what a lot of people come to Kameleoon for. That's our differentiator — with PBX, you don't have to be a developer to build a test, and if you are a developer, PBX helps reduce the time you spend on a given test build. Tell us how that relates to data, and what democratizing technology really means to you.

Democratizing Technology & the Vibe Coding Debate

Akash Doshi: Love that, and you're totally right. This dovetails into the broader trend we've seen over the past two years with agentic and generative tools — vibe coding. I'll preface this by saying I don't love the term, but there's data showing that with the rise of vibe coding, a normal person without a technical background — not a developer, not a software engineer — can now build and push applications to production.

A lot of that data points toward "AI slop" — the idea that we're producing far more technology, but not necessarily good technology. My view is a little contrarian: it's not that people sitting in the business can now build a full application end-to-end and push it to production. It's that they can own a greater share of that workflow. Developers can retain focus on what matters to them, and we can be better stewards of our technical teams by building a mockup in hours instead of days, or using a generative tool to build out infrastructure — or even just a PRD that gives developers a granular focus on exactly what they're building.

In my experience, one of the biggest pain points for engineers — not just with product teams, but with the business generally — is that there isn't enough definition for them to go build what the business wants. Part of AI and experimentation is that a tool like PBX doesn't just help you launch an A/B test yourself within minutes — for a more complicated, technical test, we can at least push out a V1. We can push out the source code and the infrastructure, then go to developers and say, "Here's our idea, here's what we've built so far — can we see it through to production, can we build V2?" That's become really powerful. If anything comes out of that, it's not just that you build end-to-end — it's that you own more technical fluency, then pass it off to developers and make their jobs a lot easier. I think that's generally a good thing.

Katie Green: There's so much I want to dig into there. Real quick, my dog is desperate to say hi — he wants to know why you don't like the term vibe coding. That's what Roscoe and I both want to know.

Akash Doshi: I think it's probably a function of connotation. Vibe coding has developed a bit of a stigma — maybe "stigma" is too strong a word, but it's taken on this life of its own, like, "I'm just going to go into a tool like Claude Code, Lovable, or Base44 and type in what I want to build." I think that's maybe the wrong way to look at it. It's more that people who've never written a line of code in their life are now being exposed to a functionality that's net new to them — it's making them more technical in the process. I think that's generally a good thing. I just think vibe coding has garnered a bit of a bad reputation, for better or worse.

Katie Green: I feel like that happens when you have a single term for a whole range of things. I do want to underline something else you said, because there are really two pieces to it. First, when people have more access to the workflow, they're able to present a much more well-thought-out, lower-barrier opportunity for engineering when it comes to testing. I love that point because it directly calls back to an episode I did with Marcella from Fossil — were you two able to meet at Unite Summit?

Akash Doshi: I don't believe we did.

Katie Green: We'll make it happen next time, whether it's Unite Paris or Unite Vancouver. In her episode, Marcella described it as a tool — her exact phrase was that it's harder to stop a moving train once you have that momentum. You say, "Here's a first version of this." But what I want to build on that is it also frees up time to focus on what matters. When you're not worried about coding, you're not clogging up the sprints with a button-color test — you can focus on things like feature flags that are really impactful to the bottom line.

Velocity and Better Questions: Two Sides of the Same Coin

Katie Green: So I think what this all comes down to is that AI is changing the workflow and how we work together. I'm curious — do you feel that AI and experimentation is allowing people to ask better questions, or are they just moving faster?

Akash Doshi: I think both things can be true at once. It is allowing people to ask better questions — you have a guided thesis on how your business or product should move forward, and AI experimentation is a great way of gut-checking that a lot quicker. Instead of taking two to three sprints to build out a test and a hypothesis, launch it, and then realize you targeted the wrong part of the funnel or built the wrong feature, AI experimentation de-risks you a bit. You're able to build a test plan, get it into production, and if it wasn't right, you've now narrowed down what your North Star should be.

The other side of that same coin is velocity — you get to market quicker. I think in 98% of scenarios, building fast is better than building perfect, because when you get to market quicker, feedback loops activate. You hear from the customer, which validates your guesswork instead of you just saying, "We think this is the right path forward." Both things exist in parallel: teams build faster, and they ask better questions as a result of building faster, because they know what their customers are telling them.

Katie Green: I'm saying this for the record for ourselves — please cut that section for social media, because it's almost like you read my mind on what I wanted you to say. Velocity is certainly one side of the coin, but it's also allowing us to get these indicators a lot faster. That's excellent — could not have said it better myself, which is why I asked you.

The Administrative Tax of Testing

Katie Green: Something else we've talked about is the tax, the administrative overhead that comes with testing and how much time it takes. You touched on this when talking about de-risking — it's an external de-risk where AI and experimentation are drastically reducing the cost of building and launching a test. Can you tell us more about the administrative overhead and how AI is playing into the overall experimentation workflow?

Akash Doshi: The best way to answer this is by example. Take a sample digital team today — the process might go something like this. You're in a monthly or quarterly business review and you see a metric or KPI that looks a little off. The team digs into it, starts to triage: what's the root cause, what's driving this, how can we fix it? The experimentation team comes in and says, "We think this is the problem — let's build out an A/B test or an A/B/C test to see how the audience interacts with two different experiences and find the winning one."

All of those steps take a significant amount of time. For a normal team balancing multiple workstreams, it could be two to three months between identifying the pain point and actually getting something into production. AI and experimentation helps speed up that pipeline considerably — a lot of that administrative overhead around identifying the problem, figuring out the solution, and developing and launching it now gets condensed into a one-to-two-week period, if your team has the autonomy to act once data and analytics have identified a key customer pain point.

Before you even go to product or engineering teams, you can launch something that helps you validate much quicker, which speeds up your time to market. That saves people a whole lot of time. Anyone who leads experimentation or A/B testing within a digital org today knows how long that backlog is — and when you compound that five or six times, you end up with a giant backlog of great hypotheses that never make it to market. They just sit in purgatory. AI's ability to insert itself into experimentation lets you clear that backlog and validate a gut feeling about how you should build a product or how your customers behave with it — reducing a lot of that administrative overhead, that admin tax.

Katie Green: That's what I tell a lot of people — it's a different way of putting it, but I'm going to start peppering in some of your talking points: that overhead is what slows you down, and then you're missing out on really critical impact.

Monday Morning Advice: Operationalizing AI

Katie Green: I ask the same question of everybody, based on what we've talked about during the episode — I call it the Monday morning advice. It might not be Monday morning when people are listening to this, but it is for us right now as we record. What I mean is: what's the tangible takeaway? You do such a good job of operationalizing your AI usage, and I think a lot of teams are struggling with, "My boss told me to use AI — okay, how?" or "I'm vibe coding these things, but they're not actually that powerful — what do I do here?" I know you're taking a bet on yourself and starting a startup, so tell us: how do we meaningfully implement AI into our workflows without it becoming another complicator in a chaos factory?

Akash Doshi: You're probably the fourth or fifth person in the last couple of months who's echoed some version of that sentiment — that enterprise is telling teams to adopt AI in their workflows, but there isn't a meaningful blueprint for how to do it.

The quick plug for Sequence is that it's AI middleware built for the consumer, helping you create repeatable workflows that let you use AI effectively. Getting to the root of the problem, I think we need a guided point of view on where using AI meaningfully helps the business. A few things come to mind: the content supply chain, for one — smaller marketing teams have always struggled to produce creative at scale, and AI has an obvious opportunity there to create versioning and new assets, delivering at a quicker clip than what exists traditionally.

For product managers and engineers, AI can now review your source code. If you notice a defect, logging it no longer has to be "hey, this button's breaking, go fix it" — you can dig into it yourself, have AI use computer use to go through the dev tools and logs, figure out the issue, and write out the full defect for you. Engineers are actually surprised when you hand them a defect write-up with that much context and granularity — they can action and solve it so much more quickly. There are opportunities like this that help narrow your focus, so when digital teams are told to go use AI, they can understand there's a guided point of view for a specific set of problems they encounter day to day — rather than just spinning their wheels.

Katie Green: I realize that wasn't the original question I planned on asking. Obviously, you and I talk before we record about what we're going to cover, but in that last answer, we ended up talking so much about AI, and I know you've done an excellent job of leveraging it. So many people lately come to me and say, "My boss told me to use AI, and I'm using ChatGPT, Gemini, Claude — I'm trying to build variations, but it's not meaningful. What do I do?" And I'm not the expert in this, but I try to elevate the people who are — so thank you for sharing that perspective.

Akash Doshi: You and I could probably have a whole separate episode on that exact pain point — how we take the workstreams people actually engage in day to day and insert AI in a way that's meaningful, rather than creating more work instead of less. But that's a topic for another time.

Katie Green: A hundred percent — you'll have to tune into part two, which we'll record in another three months. Thank you so much for being on Unite Voices. I appreciate you, and we'll talk soon.

Akash Doshi: Thanks for having me, Katie.

‍