Stop optimizing to optimize: How UX drives experimentation velocity

Transcript

Meet Craig Kistler: 14 years leading UX and experimentation at Signet Jewelers

Katie Green: Hi, everyone. Welcome to another episode of Unite Voices, hosted by me, Katie Green. I'm the principal advocate at Kameleoon, and I'm so excited to be joined by one of our Experimentation Thought Leadership Awards nominees for the UXpert category, Craig.

Craig, why don't you introduce yourself?

Craig Kistler: Thanks. My name is Craig Kistler. I am vice president of user experience, personalization, and online experimentation at Signet Jewelers. Signet is the largest jewelry retailer in the United States, the parent company of Kay, Jared, and Zales, so those commercials you see during the holidays, you've probably seen a lot of them.

I've been there almost 14 years now, which also makes me old, so there's that. When I was hired, I was the first and only UX designer they had. I started with a small UX team, and as the business grew, my role shifted. I started looking into A/B testing and experimentation, and I thought it was cool, but I wasn't sure what to do with it at first.

The team using it at the time wasn't getting much out of it. It felt like we were just doing stuff to do stuff. That's when it clicked for me: I could take user research and usability testing and run A/B tests against those insights to create a better experience. From there, it was off to the races.

Katie Green: I love that, and I think a lot of people can relate. I remember my first test. I was at an agency, and for the first time we thought, why haven't we actually tested whether this works, instead of just operating on best practices?

The people listening to this show are largely experimentation practitioners. They're interested in what you do, and you have a ton of UX experience, which is why you were nominated for an ETLA. That intersection between UX and experimentation is really critical, and you talk about it a lot online, which is partly why you were nominated. So, shout-out to anyone who wants to follow you on LinkedIn.

Breaking out of the testing plateau: why a UX lens unlocks momentum

Katie Green: I think that's a good place to start. Testing programs can plateau, and teams get stuck testing tiny little features. What does that look like from a UX perspective, and how do you recommend people get unblocked? A lot of people listening are probably thinking, I need more UX integrated into my roadmap, or I'm feeling stuck. How do you get over that plateau?

Craig Kistler: That's a great question, because it almost feels like there are two schools of thought. There's the original CRO mindset: test this thing to improve whatever KPI you're chasing. The infamous button color tests, headline tests, things that may or may not move the needle but generate a lot of hype. Everybody starts there, and that's fine. It's low-hanging fruit, so why not try it?

You'll get a win or two, but if that's all you're thinking about, you'll get frustrated, because it feels like it isn't doing anything. Your team doesn't care, your execs don't care, and you're not getting momentum.

For me, looking at it from a UX angle changed that. The job of any UX person is to ask how you make an experience better, more relevant, and more clear. When you start running experiments against those questions, it becomes more of a flywheel. You can start moving something relatively small, maybe not button color, but button placement, visibility, or clutter.

You start seeing patterns, finding usability problems, and testing those with hundreds of thousands or millions of visitors, instead of what you'd get in a usability lab. The lab is still valid, but things that test well or poorly in a lab can produce different results online, because you're in the real world with millions of eyeballs on it. That becomes a good bed of ideas: you identify problems, you hypothesize around them, and that becomes your testing roadmap.

Where AI actually helps in UX and experimentation, and where it falls short

Katie Green: Let's talk about roadmaps for a second, since I don't have a UX background myself. I come from the growth marketing side of things. Can you tell me how AI has influenced your industry as it relates to experimentation? I know AI is affecting design in a lot of ways, and UX research is so important on its own. As people talk more about synthetic user testing, I'm curious how AI has really infiltrated UX and the intersection of UX and experimentation.

Craig Kistler: I think it's really new, so it's going to keep changing. For my team, we're definitely using AI, but maybe not the way some companies would want us to use it. You hear a lot about automating tests, finding problems automatically, pushing a button and letting it run. Maybe.

But then the challenge is that you're just optimizing to optimize. You could be doing something for the experience, but do you truly understand the problem you're trying to solve? I don't want to pooh-pooh it. It's not where I need it to be right now.

Where I'd rather use AI is identifying patterns and problems that might not stand out to a typical group of people looking at data: analyzing different user flows, crunching a lot of numbers at the same time, and finding patterns you may not have seen before.

Katie Green: I was just taking notes on that, because Shirley, who edits this show, is going to love that clip. Is are you optimizing just to optimize, or are you being meaningful and understanding your users' behavior problems? I say behavior problems because it's really about the infrastructure we're giving people.

Craig Kistler: Right, and every situation has a different end goal. You mentioned best practices earlier, and I have to admit I cringe a little at that phrase. It's a broad brushstroke. If you're just starting out, fine, but you could be optimizing for different things depending on what that person is trying to do.

A product detail page is typically an entry point, so it has to function like a homepage: focus on and tell the story of why someone should care about this product and this brand. But for other visitors, it's not an entry point. They've been on your site for 20 minutes and narrowed it down to two or three items, so now the page has to work as a comparison tool. That one page has to serve very different visitor intentions.

If you go with a single best practice or one metric, it becomes an average of averages. It doesn't really mean anything.

Beyond best practices: designing for real user intent

Katie Green: And it doesn't differentiate you in the market either. I feel like we're getting to the point you make so well that I'm not even going to try to make it myself. You're a powerhouse in UX because of your interest in and passion for data. We've talked about how we have more data than ever to create personalized experiences from these entry points. Can you share more from your background about what it looks like to build better experiences with more of that data before you go into an actual test? What does that look like for shaping hypotheses and building roadmaps that are actually impactful?

Craig Kistler: For us, it becomes less one-size-fits-all and tips into experimentation as part of a broader personalization roadmap. It's not just A versus B. It's about understanding the groups of users coming to a site or page and what their needs are in that moment.

Instead of making a change for everybody, which, to be clear, we've done, we ask whether something could work better for a specific group of people. Take social proof as an example. The typical pitch is that social proof increases conversion, that everybody needs to see it. It does move some people forward, but not everyone. So you look at your data to understand who responds best to urgency, like a message that says an item is about to sell out, versus someone who just landed on your site and doesn't even know if they want what you're selling. Maybe they're choosing between jewelry, a nice dinner out, or a vacation as a gift. Urgency doesn't help with that decision.

If you can identify the intent of your shoppers, you can craft an experience for that intent that's more usable, clearer, and helps them move forward faster. On the flip side, you also have to design for intent shifting, so the experience for someone just starting out doesn't jar against the experience for someone ready to purchase, since people can move from one intent to another within a page or two. It gives you more power, but also more responsibility not to break things as someone moves through their experience.

Katie Green: Intent is so important and often overlooked. The more mature programs know it's important to understand user needs, but how often are teams actually checking that intent and being honest about what they're moving toward? I'm taking notes, this is such a good clip.

Now that we've covered the theoretical side, personalization, data, idea generation, can we talk about how you do this day to day? I have to bring up the buzzword: vibe coding. How are you using that internally? Are there systems that help you do this on a regular basis? I imagine people listening are thinking, I really do want to focus on intent, but I'm dealing with so many ideas flooding in, and balancing my team's ideas with what users need turns into politics and diplomacy. What internal tools are you creating to help you tangibly do the things we're talking about today?

Prioritizing ideas: how Craig's team scores and ships experiments

Craig Kistler: There's a lot to unpack there, because we go in a hundred different directions. I'll start by saying I'm very lucky. I have an amazing team: front-end developers, analysts, product people, and experienced designers. I'm fortunate that when I have a crazy idea or hypothesis, I can hand it off to them and have an experiment running within days to weeks. I know not everybody has that kind of team.

Then it comes down to prioritization. We have a whole bunch of ideas coming in from our own team, other teams, the brand teams, and leadership. Everybody and their brother wants to run an experiment, and it can turn into a game of politics. So we try to be incredibly fair. Whether it's the CEO or an intern with an idea, we look at it and score it based on the level of impact and the level of insight we'll get. Revenue naturally rises to the top from an impact standpoint, but we also weigh whether something will give us additional insight for future tests. That won't trump revenue, but it comes close, because learning something about a segment or a page that feeds into other work becomes valuable on its own. We score the ideas, and the highest-scoring ones come to the top.

Vibe coding as a prototyping tool for complex test ideas

Craig Kistler: From a vibe coding standpoint, we use it to flesh out an idea. We'll sketch up a design, maybe vibe code it, and see if it makes sense. I was playing around with an idea around natural diamonds versus lab-grown diamonds. They're chemically identical. One just takes thousands of years to form and the other doesn't. But a lot of people think of lab-grown as fake, like cubic zirconia, so I wanted to figure out how to explain that in a way that makes sense without overwhelming someone who's shopping for an engagement ring, necklace, or earrings. I vibe coded that to see if it would work, and it did, so it moved into our backlog and started going through the development cycle.

Katie Green: That's exactly what I was looking for. So many people are using vibe coding in all kinds of ways. At Kameleoon, we're creating internal tools to make our own day-to-day jobs easier. I know a lot of people use it for prototyping, and that's essentially what Kameleoon's PBX does: a way to build out variations, though PBX 2.0 now does a lot more than that. I love that you're prototyping complex test ideas that balance a lot of different customer needs, so you can show a developer something instead of writing another requirements document. I could live my whole life without ever writing another requirements sheet for a developer.

Craig Kistler: So much of the experience comes down to the interactions or transitions from point A to point B, and a wireframe alone doesn't explain that well to a developer. Using vibe coding to prototype it means you can show exactly what happens when someone clicks on something, what it transitions to, and how you want to walk them through that path. It's so much easier because you can see it visually instead of trying to describe it and having someone look at you like you have three heads.

Katie Green: And you're just thinking, I wish I knew exactly what I wanted.

Craig Kistler: Right, it's more like, I want something like this mashed up with something like that, and magic will happen. I know it when I see it.

Katie Green: I love that. You're very good at communicating exactly what you mean. Thank you so much for being on the show. We have time for one more question, the one I like to ask everybody at the end. I call it the Monday morning advice.

Turning big ideas into personalized experiences: starting with macro segments

Katie Green: What is your recommendation for the team lead who's trying to get out of the one-size-fits-all A/B testing rut? What's the one thing they should do tomorrow? Start prototyping in Claude? Take time to figure out user intent using data? What's your Monday morning advice for this episode?

Craig Kistler: I truly believe that creating personalized experiences will produce more revenue and have a greater impact than any single A/B test or series of A/B tests. The question becomes, how do I figure out what personalization actually is? I don't consider product recommendations personalization. That's just table stakes. You should have some algorithm that shows people similar products. If you layer personalization on top of that, it gets more interesting, but personalization to me is something bigger than "Hi, Katie" or product recommendations. It's really about how you change the experience in a meaningful way for one group of people versus another.

So how do you find those big buckets? For us, it started off very simply. We have people who come to our site to buy engagement rings, which is a very specific task, and very different from buying any other type of jewelry. So my segments were "soon to be engaged" and "everyone else," because everyone else wasn't shopping the same way as someone buying an engagement ring, which is a months-long process with a lot of emotion tied to it.

Imagine someone shopping for an engagement ring landing on a homepage littered with messages about a sale on gold necklaces or watches. Now flip it: someone who's been married forever lands on a page with a huge engagement ring sale. Neither is helpful. Once we looked at those two segments, we saw that a more relevant message to the right group of people led to higher conversion. It's not rocket science, it comes back to user experience: it's a better experience if someone shopping for engagement rings can easily find engagement rings.

From there, we looked for other big buckets of opportunity. When it came to intent, we started elementary: how many pages was a person viewing? We set thresholds for low, medium, and high intent. Looking back, it feels like common sense, but at the time it just meant asking how we should treat someone with low intent and little momentum differently from someone who's looked at 15 pages and is ready to go.

We called those our macro segments, and we talked about them in meetings for probably 18 months to two years. That was our talking point: engaged or everyone else, and what their intent level was. Behind the scenes, we were rolling out other smaller things, but that was the starting point. Now you take that same idea and put it on steroids to figure out how intent factors into building a better experience overall.

Katie Green: I love that. So many people listening, myself included, feel overwhelmed when they start with personalization, wondering where to even begin. It's really validating to hear someone with your expertise say it's not rocket science, start with your macro segments. Craig, I think that's the episode. Thank you so much for taking the time. Everyone should follow Craig on LinkedIn. We'll have that linked in the episode notes. Thanks so much for joining us.

Craig Kistler: Thanks for having me.