Why past behavior is your best A/B test signal

Transcript

Introductions

Katie Green: Thank you so much for being on Unite Voices. We're stoked to have you. I'm obviously personally a fan of your work. You and I have talked about that previously. But before we dig into the nitty gritty details, I want to give you a chance to introduce yourself to the masses listening.

Kristen Berman: Great. Well, thanks for having me. Excited to be here and chat. So name's Kristen — behavioral scientist, CEO of Irrational Labs. And basically, we think about decision making. We study the psychology of decision making, and we apply that to all sorts of things in the product and tech world and sometimes outside of it. Started Google's behavioral economics group, helped run that for three years.

And then now we work with all these different types of companies from finance, health, and other companies with behavioral economics type problems. And we help change behavior for good.

What Makes a Product Actually Change Behavior?

Katie Green: The ‘change behavior for good’ is what I want to attach to really quick. I watch all your stuff on LinkedIn and you do really good content there — plug for your LinkedIn for anybody who’s not following. One of the things that I want to ask you about is obviously psychology and product — huge relationship there. What are some of the biggest differences in your view between a good product and one that actually changes behavior, for good? Can you tell us a little bit more about the differences between what is technically good and what is actually moving the needle?

Kristen Berman: Yep. So as a behavioral scientist, I become more and more pessimistic about human behavior change. It is really, really hard to get someone to do something different. The most likely thing that I will do is the same thing I did yesterday. So the biggest predictor of if I will exercise today is if I exercised yesterday — I did not exercise yesterday, I will likely not exercise today.

‍

And so sadly, the status quo kind of runs our life. For a product to be really good or great, it has to acknowledge that it’s hard to change behavior. And the way to do that is not to go light.

‍

It’s not about telling me how cool your product is. It’s not about assuming that I will discover the new feature that you just launched. It’s much more assuming that the thing lowest on the user’s priority list is discovering your feature. And so you have to build the feature so that it really intersects with my life.

‍

Many folks use Granola. I think the big unlock in this was that it has an alert when my Zoom meeting pops up — for me to start a Granola session, I don’t have to go to Granola and say, “I want to start that meeting.”

‍

So when you start thinking like your job is to help the user be successful and really intervene within their life — versus them coming to you, discovering the thing they want to do, finding the feature, knowing how to use it, and being successful with it — that’s kind of where products have good intentions but fail. They assume the user will just do the thing that you want them to do. Great products go to the user and make it easy.

Why “Ease” Beats Technical Quality Every Time

Katie Green: Love that. I use Granola. Plug for Granola, everybody — I think it’s like the best thing since sliced bread.

Kristen Berman: By the way, Granola as a transcript app is pretty bad in one specific way — it doesn’t actually tell me who’s talking. So if you go to most other transcript apps, it will tell me differently; it says “them,” which is crazy, right? I actually want to know who is saying the thing.

‍

So despite it being kind of lower quality in that weird way — it’s great quality on summary and transcripts, but lower quality on the speaker identification — we love it because it’s so easy. And so Granola can have a technically worse product, but a way better product overall because they fit into my life. They made it easy for me to get the transcript, get the summary, and share it with my team in a way that is more automated than other apps.

Katie Green: I think that’s such a good point. It’s the ease. And so in experimentation, right — that’s what we do. We often assume that our users are rational thinkers. But with experimentation, we learn some of the wildest stuff. You don’t really care about the technical elements because you are getting the ease.

Is there anything in your work that really shifts how you approach what might be a hypothesis? Because that’s what we deal in — we’re taking data, creating hypotheses, and trying to create data-backed assumptions. Is there any shift with how experimentation comes into user behavior?

Theory Before Testing: The Behavioral Science Approach to Hypotheses

Kristen Berman: Yeah. So when we think about experimentation, we’re thinking about what is the theory that we’re testing. We try to think about the model of human behavior and say, what would motivate somebody to do anything?

But the ideas — especially as a behavioral scientist — we lock those ideas in a theory about what would change. So you can say people aren’t doing this because the value isn’t high enough.

And so your theory there would say we need to either make the value higher or make the benefit more immediate. Those are our two theories. Either the value is not high enough, or it’s just not close enough to the action we’re asking the user to do.

It’s what we’d call present bias. So we think about what are the psychologies motivating people, and then we build our experiments around them. This is different because if you go to a user and ask them why they aren’t doing something, they will respond in a very rational way. They may say it’s too expensive — which, by the way, it could be.

That may not be the core reason why they’re not taking action right now. They could say things like, the feature doesn’t work for me and my team. That may be true, but it also could be true that the value isn’t there fast enough, good enough for them to try it. So they’re not going to give you all the information needed for them to get the benefit your team can provide.

So you as an experimental person would ask: what is the theory we’re working with? Somebody giving me qualitative answers is one part of the data collection to inform that theory. As behavioral scientists, we think psychology’s understanding of those core underlying motivators is the other part that informs the theory that drives the experiments.

The Invisible Forces That Drive User Decisions

Katie Green: Well, that’s the piece that I think experimenters struggle with the most — the invisible forces. The elements that we can’t measure. I think as an experimenter myself, I love black and white data. I’m like, but the numbers say this is better.

I’m curious if there are any invisible forces that you gravitate towards when you’re looking at the behavior of users. And AI will never have EQ — these are the elements that are hard for experimenters dealing with black and white data, and they’re also hard for AI. So I want to make sure we’re covering both of those elements.

Kristen Berman: Yeah. So when we say “invisible forces,” sometimes I want to clarify for folks — it’s separate from the things you can observe. The things I can observe are how many clicks it took me, or if I’ve answered a question a certain way that tells you I’m an enterprise buyer versus not, or if I’m a repeat user. So these are the observables, and you can make some guesses based on that.

When we think about the non-observables, we can think about confidence. Am I confident using this product? If I’m not confident, social norms will help me make a decision. If I am confident — like I’ve already bought this thing multiple times — telling me other people buy it is just not going to work.

It’s really hard to understand if someone’s confident or not confident in the decision-making process just with observables. Uncertainty is another thing. If you are uncertain, you delay a decision.

So if you have uncertainty in the purchase process or the usage process, people just may not take action. How do you measure uncertainty? It’s very hard to do. You have to make some level of understanding: do people have uncertainty? How much? And then, as a product team, how can I eliminate or mitigate this type of uncertainty?

Another unobservable is perceived friction. I understand that if I add a click or a button step, fewer people will continue. That’s not always true — sometimes friction helps increase conversion — but many times more steps decreases conversion. Now, what if somebody thinks there’s going to be more steps?

So that’s really hard to understand. If I think this is a hard sign-up flow process, I’m just going to give up. Now, it may not be, but if I think that, it decreases my likelihood to continue. Hard to measure. So when you’re a behavioral scientist, you understand that all of these forces are acting on somebody as they come across your product — on the top of the funnel, mid, or at the end — that could be driving them to behave in a certain way. And your job is not just to understand the observables, but also the unobservables.

AI and Behavioral Science: Back to Basics

Katie Green: With AI coming into the scene full force — and it feels like every day there’s a new AI product — ours at Kameleoon is a prompt-based experimentation tool. We call it PBX. It’s the full lifecycle of testing from ideation, build, analyze, to ship, all done through AI agents.

With behavioral science, how does the intersection of the digital environment with behavioral science work with emerging AI? Is there anything you’ve seen that’s successful, a big failure? How are they working in the industry today?

Kristen Berman: Yeah, so I’ll answer maybe a slightly different question, which is basically two things. One is when we’re thinking about AI in general, it actually presents the same type of problem that most product people have always had — which is you have to get somebody to do something. So your chatbot, your magic feature needs to drive a user to do something.

I think we can kind of forget that in the AI world because you don’t control as much. And so if you don’t actually control the full user journey, you may forget the core principles. In our world, it’s like: what is the behavior you want someone to do?

At the end, the user does a thing — like submit or accept something or ask a question. In an eval world, you say that is kind of the success metric. That is the thing. So when you’re designing AI products, you still need to have the thing you want someone to do. And then that is measured. And you say: can my AI product get them to do that?

Which is kind of where the theory comes in. So designing AI products is not that complicated — you just go back to basics. I want to get a user to do a thing. How do I get a user to do a thing? If you’re testing different prompts, system prompts to do it, great. If you’re testing different ways to design the feature page, great.

That kind of comes back to: what is good? At Irrational Labs, we think a lot about aligning teams and asking: what is the thing that we think is good? So if the chatbot’s responding in some way that doesn’t increase trust — that’s not good. But we, as a team, have to align that we want trust as a variable that we would measure. There’s a level of agreeing on what is good that teams haven’t had to do before because the output is so variable.

Normally, the output is finite and you can QA it — is this good or not? And it’s very clear to folks. I think the messy middle with designing AI products is just not clear many times what you’re going for. And you have to actually predefine these categories, which is kind of what evals help folks do.

Is AI Actually Doing Behavioral Science? Kristen’s Honest Take

Katie Green: And this is more for my personal curiosity. Are there AI tools that are focused on behavior and psychology right now? Are you using any? Do you think there is a space for AI in your field, specifically talking about psychology and user behavior?

Kristen Berman: So a lot of our work is really inserting psychology — the idea of behavior change — into different products and services, versus habit change, which is potentially more like personal behavior change. Let me separate the two.

I think in our world, behavior changes everywhere. Like if I want to just get you to do anything, the theory is I have to change your environment of decision-making. Instead of pounding over the head and sending five emails, if you’re a B2B app and you want the admin to add some setting that unlocks usage for everyone — I need to change the interface of the app. I need to get you to do something different.

And so I think fundamentally, AI has changed how people work. The way I get work done has changed. My whole workflows are changing. So at some level, that is the intervention — with AI, every single person is going to a different app than they did two years ago, five years ago to get work done.

Our team is figuring out how to all work in GitHub successfully. That is very difficult. It’s changing the function, the environment of our work, which means our output will be different because the functions are different. I don’t think there’s a particular app that does behavior change well now that AI is here. It’s basically AI changes the structure of our work fully — and how we get stuff done is changing, which means our behavior will change.

Katie Green: Okay — I’m very much enjoying that answer. I think that the environment is something experimenters kind of get lost in, where we’re really focused on the minute changes. Okay, what are we doing to change the environment, to influence the overall environment? Behavior is real. That’s just a really cool snippet, and I’m going to carry that with me for a long time.

The Behavioral Science Teardowns: How Kristen Evaluates Apps

Katie Green: I personally love your videos on LinkedIn. Can you tell us a little bit more about the teardowns that you do and what you look for when you’re generating these insights you’re sharing with people at large? I want to say finding your victims, your targets — but I say with the utmost respect.

Kristen Berman: So these videos are basically teardowns of apps or products that people normally use. You can imagine — if you’re using Granola, if you’re using Spotify — I go in and look with a behavioral science lens and ask: how would a behavioral scientist evaluate this app and product? The way I choose them is just the ones that I use.

It’s way easier to have an opinion on an application if you’ve used it before. In fact, anytime I’m doing a call with somebody new, if I meet them and they say they work on an app, I will literally download it quickly off Zoom so I understand what we’re talking about. A lot of times when we’re talking about an app or an experience, it’s in your head. And as behavioral scientists, we are ruthlessly tactical. We just want to see the thing.

And so the teardowns are really a fine tooth comb — going through every single step that it takes to get you to the thing you want to do in the app.

When I looked at Descript — a lovely video editing and podcasting tool — I’ve done a couple of teardowns with them. One was just using their AI tools. If I want to be a successful video editor, I have to figure out how to use their AI tools. Well, how would I do that?

When you’re going at things with a behavioral science lens, you’re basically looking for the small details of life. You’re looking for the very small things that change how I would behave if the button was different or the call to action on the top of the page was directing me in some different way.

As an example, Google launched their AI pricing very early and I did a fun teardown of that. For a Googler who’s used a lot of AI tools, this may have been very intuitive. But for a normal person, the way they framed it had cognitive overload — they actually had the most expensive product in the middle. I’ve never seen that. And then you just didn’t understand the value prop. How would anyone go through this flow and be successful?

Using these core psychologies can get you to unlocks of: how would anyone be successful here? Or — what are they doing that is so successful that we should copy it and pattern match it? I did the Whisperflow onboarding, and I don’t think I’ve ever done a teardown with no critical feedback. It was all positive.

So you don’t just have to learn from things that are bad. You learn from things that are good — what are people doing that drives you to actually complete the flow? Whisperflow is a really hard thing to do. You have to download the app, set up this voice, change your settings. Somehow they get you to be successful doing this. Let’s study how they do that.

How Kameleoon’s PBX Can Power Better Teardowns

Katie Green: I love your teardowns. It brings me to a point that I think would be really cool if you ever wanted to try it — which is Kameleoon’s PBX. You can use it on any website.

If you want to play with PBX — see what it would look like as a response to your teardown — you can go onto any website and it just does it on your machine. Obviously it doesn’t launch the test. But it’ll show on the website.

I actually did a funny one for Alaska Airlines — I just fly Alaska a lot, I’m based in Portland, Oregon. And for International Women’s Day, I was like, we’re still making 15% less than men. I was booking a flight somewhere and thought: wouldn’t it be great if we got 15% off for the whole month?

So I just went on PBX and said, make this 15% off and make the branding look like International Women’s Day. And it turned it into this purple and blue, 15% off for women travelers website. It was silly, but it was just trying to prove the point that you can do anything in PBX to show your point.

You can go in and say, “Descript — or Google — why is this in the middle? Move it to the right.” And PBX will move it in less than two minutes. It’s really fun for making your point land.

Building a Fail-Forward Culture: Starting with Low-Risk Wins

Katie Green: I want to go back to one thing you said — that you learn from what is good as much as what is bad. I see often in your communications the fail-forward mindset. You’re coaching really high-stakes people, and a fail-forward can be really scary. Do you have any wisdom you can share on how to create a culture for failing forward and prioritizing that kind of learning?

Kristen Berman: Yeah. I think a lot of experimentation culture comes from having early wins — because you understand the upside of what you’re doing.

So when we’re working with teams, we’re not going to test the most risky thing in the beginning. We want to prove the idea that some experimentation can drive an outcome. The first hypothesis is: take a place that doesn’t have as much downside risk. A lot of surfaces within products have downside risk — where if you mess this up, you will put revenue at risk in a way that is fundamental to the business.

We think people should start with more of the upside. If you change a button structure within your signup flow, you try four things — one of those things would likely drive higher conversion. Does that put a lot of revenue at risk? No. Is it fundamental to the business? Less so. If you change one thing in the onboarding, you’re not going to make or break the company. But people can build a high-confidence self-efficacy that they can do these small changes and get bigger wins.

So one principle is: don’t put a lot of the business at risk for small experiments. The second is that we do a fair amount of pre-testing out of product.

So if there is a big swing — where we want to fundamentally change our pricing, switch how our feature set works, redesign the overall value proposition — these are big things. Doing A/B testing on this as the first thing you do is risky. So instead of just doing qualitative work, which is what most teams do, we do what we call out-of-product testing — we have four versions of something and recruit people from an online recruitment platform, let’s say 1,000 people.

250 get one version, 250 get another, et cetera. Then we ask them questions about their likelihood to do the thing we’re trying to get them to do. Did they understand it? Would you give us your email address? Which is what we call a dependent variable — it drives our prediction of which version would work the most. So we’re basically doing out-of-product A/B testing.

And we want to simulate the environment as much as we can, which is actually so much easier now that we have AI — you can just mock up websites and people can actually play with the website. It feels real. And so for folks who are nervous about A/B testing, I would think about taking it out of product to get the confidence to launch something in product where you still would do some type of experiment.

The Biggest Red Flag in Experimentation Programs

Katie Green: Yeah, I often have this conversation. As a practitioner, I’ve been doing experimentation for many years, and I often say the biggest red flag is when the risk to experimentation is just not experimenting at all.

And I want to take this chunk and just plaster it on a billboard and say: you can start with what’s good. You can start with where you’re going to have less risk. It’s a really calculated, strategic way of approaching growth that is naturally de-risked. But people hear the word experimentation and they go, “Oh no — but what if it loses?”

When you’re coaching teams, is there a certain trend, characteristic, or trait that you can identify with the teams that do have some of the best growth practices and cultures of experimentation? What are some traits you find that maybe we aren’t thinking of beyond healthy data rigor?

Kristen Berman: Yeah. The fundamental thing is: do they have the logging data to be able to tell if something works? And can everyone in the company pull that?

We work with some companies that can’t, and some that can. The creativity of folks who can — the marketing manager can go pull the stats on how things are working — their creativity is just much higher because they’re able to think with data about what is actually going on.

So base level: do more logging from the start of the company. That’s hard for startups to prioritize, but it’s required.

And then the second trait is that these teams understand that they might not be right. There is upside to things. They say: it would be really surprising if the first thing you did is the optimal of all the things you could ever do. The first time your designer designed the homepage, the onboarding, the dashboard — it would just be really shocking if that was optimal.

So they have more humbleness — it is likely that one or two other things could be better. Now, sometimes there’s not a lot of upside there. If you’ve seen Google optimizing their homepage, there’s just not a lot of upside left. But most of the time, especially if you’re a new team with a new product, there is upside to be had by changing a few things.

So the teams that get this are just a little bit more humble about what they know about the world and their consumer. And so they’re interested to try things.

Katie Green: Okay, humility is a big one. Got it. No, that’s great.

“Best Practices” — Friend or Foe?

Katie Green: The words “best practices” — how do you feel about those words? Because as experimenters, we’re like, boo, best practices. They’re just somebody else’s guesses. And even if it’s correct in one environment, who’s to say it’s going to work for another? What about personalization? So when you’re thinking about consumer psychology, how does the term “best practices” feel to you?

Kristen Berman: Yeah. So in some ways, I like the idea that we don’t want people to start from scratch. I like to critique IDEO a lot for this — it’s a blank canvas, brainstorm as if no one’s ever thought about this problem before. And as behavioral scientists, we always start with literature. Somebody’s thought about this problem before, and there’s likely a meta-analysis on the question. Let’s go find that so that we’re starting from a point of insight.

Let’s create our hypothesis from the point of insight. We now have an insight-informed hypothesis — we’re not saying it’s right, but we have something grounded. We didn’t have to stand at a whiteboard together and brainstorm that. We like that humbleness — this idea that we’re not the first people to think of the problem.

And many times what that looks like is: look, if we’re trying to do an e-commerce nav bar, we should probably look at people that we know do experiments — Amazon, Wayfair, Booking.com. We don’t want to take a travel example and apply it directly to e-commerce, but these folks have experimented with nav. Where do they land?

And so generally, if you know that a company is experimenting and they’re in your domain, it is likely you can get some hypothesis from looking at what they’ve done. We would not take a random startup and get ideas from them because they’ve not done experiments — they don’t have enough sample size. Larger companies do.

Google’s homepage has just been experimented on extensively. If we were to start a search company, we would not start from scratch. So there are times where you should leverage best practices — and then times where you basically say, look, context does matter. My user is different. The context is different. I need some clear hypothesis.

The other thing is: if you just reinvent the wheel for every single thing, you just won’t have enough time to focus on the big things that do move the numbers for you. We are not opposed to saying, look, there are 10 things you could focus on — let’s pick eight where we start from a baseline we believe works, and the two that are going to drive your business, we are going to think very deeply about.

Will AI Flatten Product Design? The Vegas Signs Argument

Katie Green: It’s refreshing to talk to somebody who’s not just reflexively against best practices. I have one more piece I want to bring up. In the world of AI — AI is designing, AI is coding, AI is doing everything. Are we reinforcing the best psychological and behavioral requirements of users?

What I’m asking is: if AI is designing every website, is every website going to look the same and function the same regardless of industry? Does AI absorbing best practices and what it knows of behavioral psychology just reinforce the same things — and then we don’t get creative or have unique use cases for specific needs? I’m curious your take. There’s this AI whitewashing of the same function, the same product. Is everything going to be the same if everything is built with AI?

Kristen Berman: It’s a good question. And I think as more people become builders and more folks make things, they’re likely to take the default recommendation from AI and not necessarily change it. Now, I think in the world we’re in right now, the default recommendation from AI is still worse than a designer’s.

It will change — it could get better later. I don’t think we should hold on to that as a permanent statement. So you can imagine a world where the default recommendation is actually better than a designer’s, but it’s very bland and vanilla.

I tend not to believe in that world, because if you have more builders making more things, the world is going to be incredibly noisy — and people are going to need to stand out. With a noisy world, people do a lot of things to stand out. I like to compare it to Vegas — Vegas has no zoning regulations on signs. And so signs are crazy in Vegas because you just want to be bigger and better than the person next to you.

Most cities have regulated signs, so you can’t do that. But if you don’t regulate signs, every city would do that. So the idea is that human creativity could actually get higher because we need to be the best flyer on the flyer pole for people to read us.

Katie Green: I absolutely love that take. And to bring it full circle — that’s where experimentation comes in, right? To be the biggest and the best, test it before you launch it.

Monday Morning Advice: Start Noticing

Katie Green: The last question I always ask on all of these episodes — what is the Monday morning advice? For somebody who wants to integrate more behavioral science into their roadmaps, their product builds, whatever it is — what is your recommendation for how they would do that starting tomorrow?

Kristen Berman: Hire us. Just kidding. That’s a joke.

Katie Green: I’ve set you up for that one perfectly.

Kristen: Yeah. No — we think most people already have some insight. Everyone’s a little behavioral scientist. We’re all going about the world, kind of observing things.

And I think that’s the double-click you would do — you’d just notice more things. You’re in your environment, noticing what is hard, what is easy. You’re noticing where social norms come in. You’re noticing how the coffee shop asks you the question: do you want a small or a large? As soon as you start noticing things, you notice that the environment of decision-making is changing your decisions.

A lot of times we think that information changes our behavior — if I just knew more about how good a thing was. Different details. Financial literacy is a nice example. If you want to change financial outcomes in the US, you might think you could change financial literacy — get more people to know about FICO and compound interest.

Turns out when you teach people these things, it doesn’t actually change their end behavior. It doesn’t actually make them create a savings account or pay off their debt. The things that do are the small details of how a bank designs their credit card payment — the minimum first or last, is it defaulted or not? And then you can really change behavior.

As soon as you start noticing those small details of design that could change or influence your behavior, you’ll bring that back into your work life. We’ve seen this consistently — we have a bootcamp that trains folks in behavioral science, and we really see them changing the lens on how they look at their external world and bringing that back into work.

Katie Green: So you’re saying it’s okay if I put “little behavioral scientist” in my LinkedIn profile now.

Kristen Berman: Yes, yes, yes, yes — you got that.

Katie Green: That’s what I’m hearing. I think that’s an excellent place to end. Thank you so much for being on the show. I can’t wait to hear what people are learning from you from this and from following your LinkedIn, because it’s the place to be. Thank you so much for being on Unite Voices.

‍