AI does the boring stuff: Why creativity still wins in CRO

Transcript

Welcome to Unite Voices

Katie Green: Katie. Katie and Katie. Welcome to Unite Voices.

Katie: Thank you for having me.

Katie Green: It's so nice to have a friendly face here on the pod. Thank you for lending your voice and your story. I know some people are going to learn a lot from listening to you on our show.

Katie: Hopefully. So introduce yourself. You've had a ton of responsibilities with Kameleoon, and outside of that, you've been in CRO a long time. You manage community. Give us the Katie Kelly spiel.

From a UX Agency to CRO: Katie Kelly's Origin Story

Katie: Yeah, okay, so it starts — this is going to date me — but I started in CRO in 2008. So going back a while. And when I first started, it was still relatively new in the general market. Like, obviously some of the major players had started doing CRO, but really it was an unknown thing.

And since then, I have, for most of my career, been in CRO. I've done the odd stint in more traditional digital agencies, but yeah, overall it's been CRO agencies — doing the marketing for them, doing the operations, helping them grow. So when it comes to agencies and conversion rate optimization and marketing, I'm your girl. So yeah, it's been a while.

Katie Green: I come from the agency world myself. I did a long stint at an agency doing CRO. It's where I really got into the nitty gritty of experimentation and where I got most of my experience. So I hear you on the agency side. Do you think there's anything about the agency world in particular that has shaped how you think about CRO? I'm curious for people who may not know what it's like to work at a CRO agency — is there anything in particular that you take to your day-to-day, or that shapes how you think about things, because you've been in such high-stakes growth environments like a CRO agency?

Katie: Good question. I think agencies are always trying to think one step ahead. I've spent a lot of time selling conversion rate optimization and speaking to clients, and it's really interesting to see that there are a couple of buckets when it comes to clients in terms of what they need and what their problems are. Those haven't really massively changed over the years — the same problems still come up, the same needs still come up. I think the agency world has changed a lot, obviously, to reflect what's happening in the market.

One of the things I found really interesting from working in the agency side is how they sell it, and the market's kind of gone in a similar way. So we started off — when I started, I worked at a UX agency, and we realized we were selling these UX recommendations and nobody did anything with them. So what we did was we started to think, okay, well maybe if we test them and we can prove to the client that this is actually going to change their business, it's going to be valuable for them.

That's how I think a lot of agencies started to get into CRO — that natural progression from maybe UX into testing. And then as things went on, all the agencies started to develop models and focus on prioritization, making the testing we were doing for clients better. Now we're kind of at the stage where we're talking about systems and processes and managing that. I don't think CRO ever really became massively mainstream, not the same way SEO did.

But it's been an interesting transition. From the outside, not working in-house with teams, we're always trying to think about how we can fit into the organization, and one of the biggest changes in the industry is that A/B testing is now sitting a lot of the time in product teams — that's still a relatively new thing for digital products. When I started, it was marketing — marketing was our main client, doing website testing. Now it's about testing systems and pricing and things like that. So it's been a really interesting transition, but I think fundamentally still the same beast.

Katie Green: That's super interesting. I started in — I remember the first test I ever ran was 2016, so a little bit after you. Really impressed with the 2008 drop, that's very impressive. You just have such incredible experience. I remember my first test, and it really was the marketing side of things, and I feel like that is still my persona — pretty squarely on the marketing side, because it's more about unlocking barriers to testing. That has been what has shaped my career most: trying to release the barriers to testing. But I'm curious with AI, right? That's probably going to be a huge part of our conversation today — AI is changing everything. To even fathom moving at the speed of AI is hard to wrap your head around. How does AI shift the role of the CRO agency?

How AI Is Reshaping the CRO Agency Model

Katie: I think that's a really good question. I think already we're seeing AI take out a lot of — in every industry — the junior-level roles. So before I talk about how it's going to impact the services agencies offer, what I think is really concerning is that entry-level jobs, where people cut their teeth and get to know the industry, are disappearing. Who's hiring these people now? How are people getting into this industry? Because we still need fresh people to replenish turnover as people move out of the industry, and we still need talent.

So I think agencies are all generally moving more towards strategic services rather than the doing of the testing — a lot of that now is being brought in-house even before AI. But then a lot of the project management is also being taken over by AI. So the position for the agencies is now often more high-end: how can we add the most value from a strategic point of view, more consultancy, that type of thing.

But that then brings the question: how are we going to get new people into the industry? Because juniors aren't coming in able to offer strategic advice and consultancy. That's one thing I'm really passionate about — especially making sure we have diversity. I think sometimes we forget that this is also, we focus a lot on the data science side of things when it comes to testing, but I think the other side is the creativity side. That's something I don't think can be automatically replaced by AI — it can be a tool to help, but at the end of the day, we're the people who are going to come up with a really creative and different approach to solving problems. We need diversity in the industry in order to really nail that. If we just end up with no fresh people coming in, or a lack of diversity, it creates problems — we're not as good at our jobs, I think.

Katie Green: I love that. I think it's so important. Drake Somm and I did a recorded interview webinar a few weeks back about PBX Ideate, and I brought up the question: what does this mean for human creativity? In 2018, 2019, 2020 — those were my golden years of testing. I don't know if they were the golden years for the world, but it was my golden age of testing personally. Other than getting the results back, which is always fun to know what's working and what isn't, the other thing that was most exciting to people was ideation. I brought it up to Drake — what does this mean for the people who love this part, this creative part?

I also brought up the point where here I am, it's 2023, I've been working on the same website for three years, and I'm like, I can't look at this homepage hero one more time. If you think about all the little things I've changed, it's a completely different experience from day one to now. But I was trying to balance the fact that ideation is fundamentally fun — pun intended — what is AI's role in taking that out? But I'm also getting this page blindness from looking at it too long. And his quote was so perfect: AI is supposed to do the boring stuff.

Something you'd have a unique perspective on is the client's perception of an agency in the age of AI handling the boring stuff. Because it's kind of like, is your agency handling the AI boring stuff for you, or are you handling it? What does governance for AI look like between third-party vendors? I'm curious if that's something you think about, or challenges you hear in your community — how are we managing the AI? Because we're all being told to use it, or being taught how to use it, depending on what your mandate is. I find it fascinating, because in my own experience, I mean, we had an AI tool, but it was ChatGPT 1.0 — just an LLM. But now with agents, everything is different. So how does that work with a client's perception of AI and agency relationships?

AI Governance, Compliance, and the Client Relationship

Katie: I think one of the things I worry about is agencies rushing in to use AI without the governance or the full understanding of what the tools do. I think any agency worth its weight will pause before it rushes in. I know there's this desperate need to be at the forefront and to offer clients the newest, latest shiny thing, but real research and understanding into what you are using is really needed — because the markets that many of your listeners work in, if you're in Europe, if you're in certain parts of America, there are particularly stringent rules around data and how you use it.

Using tools without fully understanding their capabilities is opening you and your clients up to trouble, and clients within certain industries are going to be more skeptical and more concerned about signing up with agencies that are using things without fully knowing what they are. Obviously it's exciting, and many in-house businesses are using a ton of AI tools, but the governance bit is probably still lacking from experience — not pointing any fingers, but I've seen a lot of people using stuff they don't know about, just thinking, oh, I found this new thing and it does this for me. So I think it's super important that there's more governance and more thought into it, because everything's changing so quickly. I'm writing about AI and conversion rate optimization right now, and every single day things are changing — it's hard to keep up with where things are going.

I think looking at the basics, the simple stuff of automation rather than necessarily AI — how can we automate processes, how can we simplify them first — is one big thing. Often I look at processes within agencies and they're already crazy, and then it's like, let's automate this crazy process. That's not the question. Let's start by simplifying the process, and then automating it. Taking those small things and looking at safer options like automation first, and then starting to think about how you can use more of this AI technology in a safe environment that's not going to jeopardize anything you're doing, or any client data, or anything sensitive.

Katie Green: Yeah, the compliance-sensitive industries — I'm constantly seeing webinars on AI in healthcare, AI in banking, because for e-com it's kind of like, well, at least in the US, we have pretty lax consumer protection laws compared to EMEA. But the AI of it all is so interesting, and that's a huge part of what Kameleoon does — we have PBX, which is prompt-based experimentation. I always forget I need to say that, because PBX is just so natural to me — PBX is in my DNA, Collin will love that one. With AI, it is a huge part of what Kameleoon does — PBX is absolutely massive, it is the product. When I say I work at Kameleoon, I work at an AI company that does CRO. It's not a CRO company that does AI.

Kameleoon's PBX: Simplify First, Then Automate

Katie Green: With AI experimentation, I love your point of simplify first, then automate, because I think that's what PBX does well, personally. I came to Kameleoon in the first place because they gave me a free PBX account — they were like, play with it. And I saw it and was like, oh my God, this would have changed my life ten years ago. This would have changed everything about how I would have done testing, and the fact that now it's in a single place — that's where I think some of the issue comes in, when you're using Claude for ideation, you're using Gemini to analyze data, you're using Granola to take notes that you're putting into an ideation formula. You see what I'm saying? But PBX is a single place that the AI lives, and the data lives in this one place. So with Kameleoon, at least, I am very hopeful for the future of it, because it does simplify the process. Do you think AI is simplifying the process, or making it more complex?

Katie: I think the goal of AI is to simplify the process. I'm not sure right now that it is, because everyone's still kind of grappling with what exactly it can do and how it's doing it. I think for some things, the basic, easy tasks, it's simplifying things. But one of the things I was going to mention, especially for agencies and tooling — for clients, it's another thing to get through procurement. If you're now saying you're doing AI, there's a whole bunch more questions and concerns to get that signed off.

With Kameleoon, you've met different criteria to have that signed off. You can show clients, particularly in those high-compliance industries, that you meet these criteria, so it makes it a lot easier. But for agencies who don't have that, it can potentially be a stumbling block in terms of having clients easily work with you. That's something agencies need to consider going forward. It's all fair and well saying, oh, we have this tool, we're using AI, we're going to make your life easy — but if I were the client, or if I were doing the marketing for that agency, I'd be asking: how are we going to make sure the client feels safe, that they know they're going to meet their compliance criteria, and that what we're using is okay for them to use? It's a knock-on effect — it might seem simple just to start using a tool to handle data or review stuff, but what knock-on impact does that have? If it's going to start impacting sales, then it's not so simple. So that's another thing I'd be thinking about as a marketer of agencies — I don't mean to be the party pooper, but we should probably think about this.

Katie Green: Yeah, I could live the rest of my life in happiness and bliss if I never had to fill out another security questionnaire. I think we've covered a lot about agencies and AI, and it's really important that we talk about this — you have such an incredible brain for agency relationships. But there's the other part of your agency experience I want to make sure we cover today, which is community building.

Beyond the Work: Community Building and the Experimentation Thought Leadership Awards

Katie Green: You are excellent at scaling for people, excellent at community building. I want to do a little plug for the Experimentation Thought Leadership Awards — you knew that was coming. You help us significantly with the ETLAs, and you help us a lot with our Who's Who project. I'd love if you gave the people a taste of Who's Who, and I can give the spiel on ETLA quickly.

So the Experimentation Thought Leadership Awards recognize some of the industry's — I want to call them influencers, they're CRO influencers — and it's not specific to CRO, but it encompasses everything that relates to CRO. We have UX experts listed in there, very rigorous data and analytics folks, and obviously the big prize is Experimentation Thought Leader of the Year. You work with us to create that nominee list, getting people into these categories who are talking about things. There's a new category this year, AI Innovation — evangelism of AI — and honestly every nominee could probably be in that category, but we narrowed it down to AI-focused folks. A lot of those people are also on our Who's Who list. So why don't you give us a taste of what that is, because you've done so much work over the years on Kameleoon's Who's Who?

Katie: Yes, I have. So the Who's Who started because we wanted to recognize people doing great stuff in the industry, but also start to represent voices that weren't necessarily being seen. For me personally, and I think this is shared by Kameleoon, I really genuinely care about getting more diverse voices into the industry, spotlighting them and giving them a platform. It helps people's careers — being active on social media has helped me personally find work, get jobs, get better jobs. So I think in order for there to be more diversity in the industry, we need to help platform those people and show them off. That's really important for the industry as a whole.

So the Who's Who aims to do that, in a fair and transparent way. There's a very set criteria that was developed to help us score, and I basically comb the internet — I've reviewed thousands and thousands of people online who work in the industry against this criteria, trying to find people who aren't necessarily at the top of other people's lists, but who are doing great stuff and deserve to be celebrated.

Inside Kameleoon's Who's Who: Lessons from Screening Thousands

Katie Green: And what have you learned over the probably thousands of people you've screened? Is there anything in particular that's stood out to you — could be a specific story, or just a general trend. What do you take away from those folks?

Katie: I think the general trend is we still need more diversity, we still need more fresh blood coming in. There's a lot of people who have been in the industry a long time who are still doing amazing stuff, and they are most definitely thought leaders — they're doing new things, pushing the boundaries, doing a lot of research and original learning and sharing that with the industry, which is amazing.

On the other end of the scale, the younger blood coming in — I would love to see more of that. I think people at the fresher end of the industry don't feel as confident sharing, but there's still a ton of stuff they can share that's really valuable. Having the perspective of someone with fresh eyes coming into the industry, who's not been tainted and worn down over the years, who doesn't have all that history — they haven't been through the argument of "CRO is dead, let's not call it conversion rate, only going to call it optimization" — this whole thing that's gone on for years and years. They don't have any of that baggage, and I think it's lovely — I want to see more of their thoughts and what they observe, because their experience is going to be a lot different to people who've been in the industry for twenty-odd years. I think that's valuable.

Katie Green: Are you saying I have baggage?

Katie: I'm saying we all have a little baggage.

Katie Green: Yes, we do, we all do — my therapist would agree, for what it's worth. Shout out, Tina.

Katie: One thing I think isn't discussed much is there's a lot of new talent emerging in South America.

Untapped Markets: CRO Beyond the Usual Suspects

Katie: I'm seeing a lot of new blood coming from regions that haven't traditionally been highlighted. CRO has been in focus quite a lot in Europe, the US, somewhat Australia — predominantly English-speaking markets, also Europe. But really the rest of the world is just this grey area where a lot of stuff happens, but no one really sees it. I think that's starting to change. I moved to Asia about seven years ago, and it's such an untapped market in terms of optimization — it's still really behind the times in terms of even just getting the basic UX stuff right. I think there's a massive opportunity here, because these markets are huge, they're massively expanding, there's so much potential and a real demand for it. As we struggle in some of the more mature CRO markets, where people are arguing over what the role of the agency is, or what we do in-house versus what's going to be fully AI, there's this whole opportunity just sitting there waiting for somebody to tackle and to help. That's one thing I'd love to see in the industry.

Culture, UX, and the Real Cost of Friction

Katie Green: I'm ready to be sent to Japan for a work trip to hang out with you. I noticed when I was in Japan last year — reserving tickets, I was like, oh my God, this is so difficult. Do you think there's a cultural difference in UX, where the expectations are different? I saw on somebody's LinkedIn — their exact quote was, if people want to do something bad enough, they're going to do it regardless of friction. And I was like, that's interesting.

Katie: True.

Katie Green: I would say culturally that's very different. In the US, where there's so much competition — you know what, I'll reference something that just happened to me. I bought a candle warmer. I promise it's a thing — it just heats your candle so it's flameless, instead of lighting it. And it was a lot of money, I'm not going to lie, we're talking over a hundred dollars. But the point still stands: there's so much competition in the US. Why am I going to buy your candle warmer versus a different candle warmer? Maybe I buy a lamp instead, you know what I mean? So it feels like there's so much competition, but do you feel that's different in Asia? You've been there seven years — I didn't realize it had been that long, that's incredible. Do you think that changes the UX expectations?

Katie: I think the markets are completely different, and it depends where in Asia. I've lived in Singapore and I've lived in Japan, and both are very different markets. The language being different in Japan also changes UX, but culturally, vastly different — you'd need someone on the ground to understand that. One thing: the amount of text and information consumers expect in Japan is incredible. You go onto websites here and the whole page is just copy. In the UK, or other Western countries, we'd be like, what are you doing, cut the copy down, cut it again, get to the point — and that's just completely not what consumers here want.

The market itself also changes people's perceptions. There are so many more independent businesses here when it comes to something like the restaurant industry, but fewer online digital versions, so your choice is a lot more limited. I was trying to pay for something online the other day and I couldn't do it — my alternative was to physically go to a location, have it stamped, and then post it. So there's a real demand and need for the digital experience to be improved still, I think. There's a massive opportunity here, but you need to really understand the culture to make that work — you can't just copy and paste from other countries.

Katie Green: And I wonder how that will affect the Who's Who down the line, right — when we're thinking about cultural voices in CRO, they could be counterpoints to what we might see from our networks already.

AI, Creativity, and the Risk of Sameness

Katie Green: Back to AI — this is not the first time I've talked about this today — but when AI is doing all of our building, all of our designing, all of our ideating, does it all end up being the same website? Are we going to end up with the same products and the same websites across the entire world? I think that's where the human in the loop and the EQ is so important, because it does have a different cultural context. Do you see that as a threat to CRO at all, or just a tool for accelerating? For me personally, it accelerates the flow so much that you can get more of those creative ideas in. But I'm curious — I don't know the right term for it, but the whitewashing of AI in terms of design.

Katie: I think we were doing a little bit of that before AI anyway — I remember the "blocky walky" phase, where all the websites — I can't remember who wrote the article, but everything's "blocky walky" was the headline. We were doing that before. I think it's always going to be a threat if everyone just blindly follows it. It needs to be seen as a tool — there has to be a human in the loop. Two of the main areas that individuals, agencies, and people working in the industry really need to double down on: the creativity side of things — doing things differently, not just doing what everyone else is doing, not doing what AI is suggesting, using that as a starting point and then thinking further.

That's one. The other area is user research. I know there's synthetic audiences and all that, but for me, there's nothing better than actually listening to another human explain their problems, what they're feeling. I was just doing some user research last week for an app in Japan, hearing people talk about the subject, what problems they're having in their life, what they're doing tomorrow — and it all feeds into what you can create, completely different ideas that never would have come about if you were just using something synthetic, or if you'd just skip it altogether. It's so vital that you do the research. I see a lot of agencies going down the very hard data-analytics side of things, and that's great, but there has to be the qualitative part, the creativity part too — those are the two areas humans are amazing at. I think those are the areas we need to double down on.

Why Synthetic User Research Isn't a Shortcut

Katie Green: I went to an AI conference — actually it was called an unconference, very cool, in Portland, where it's literally just roundtables with topics we determined that morning. In one of my sessions, we talked about synthetic user research and how dangerous it can be, but also how helpful it can be as a starting point — balancing "I have to start from somewhere" versus "I have a system in place." The best quote that came out of that session was: AI is like my teenager — it's 100% confident and 70% right. And I was like, yeah, and that other 30%, you have to fill the gap — you, as the human, have to fill the gap. I thought you'd like that quote.

Katie: I do like that quote. I'd also argue it's so easy to do user research — a lot of places overcomplicate it, a lot of agencies oversell it. But at the end of the day, pick up the phone, speak to some of your customers, go buy some people coffee, sit with them for ten minutes, get your mobile phone, record it, and ask some questions. It's really not rocket science, but the payoff of doing that is so massive. If you're an experienced user researcher, you know what you're doing going into those sessions, so I think the barrier to entry can be a lot lower than many people think. And the payoff, as I say, is huge — I'd be a strong advocate for still just going out there and speaking to people. But if you're not experienced and you want to start somewhere, having that back and forth, understanding what questions you should be asking and the types of responses you might get, then yes, synthetic user research can be helpful.

Katie Green: It's really the "I have no clue where to start" use case — what should I ask these people. Going back to Drake's point: let AI do the boring stuff. It's funny, because it's actually so exciting what it's able to do. But of course, it will never replace human creativity.

Final Word: How to Get on the Who's Who List

Katie Green: I realize we're coming up on time, but I always like to end these interviews with one final question. Let's say somebody is really passionate about — I'll stick with the Who's Who — they want to be on our list. What should they do tomorrow, coming out of this podcast? Like, they listen to this and say, oh my gosh, the agency relationship is totally changing, I'm in-house and I want to be part of this Who's Who to accelerate my career. How would I get on this list?

Katie: If you want to be on the Who's Who list, start sharing online what you're doing — without breaking any confidentiality rules you might have for your business, but you can talk generally, share your own opinion. I think, often, especially people from more diverse backgrounds think what they have to say isn't important, or question their level of expertise and whether they should be sharing. Both of those are not good reasons not to share online. Everyone, at every stage of their career, has something valuable to share. Even if you're new to the industry, sharing your perspective on what you're seeing, what you're experiencing, what your take is on how things are changing, is really, really valuable — even more so for people who've been in the industry a long time, because we can get so stuck in our bubble.

A lot of in-house teams and clients don't think this way. They're not in this tiny little bubble we're all in — they have lives, they do things outside, and we can sometimes forget that. I think it's really valuable to have that insight. So if you want to be on the list: start sharing, share regularly, share honestly. Don't try to dress it up, don't use AI — just write from the heart, and try to think about things that might be interesting for other people to read. Simple as that.

Katie Green: I love that. I'd also say: share your experience. That's something that just performs best on my personal LinkedIn — when I share something I'm like, this is really cool, and you change the way you frame it, where you say, I was once in your shoes and this is really cool, it totally changes how it's perceived.

I appreciate you taking the time to be on the show. Obviously, you and I are buddies, so we get to talk all the time, but it's a unique opportunity for people to peek into our conversations. Thank you for being on Unite Voices, Katie.

Katie: I really appreciate it. Thank you very much.

‍

‍