Why synthetic users can't replace real usability testing

Transcript

Welcome, and celebrating the UXpert award

Katie Green: Thank you so much for being on Unite Voices and making time, pretty last minute. So shout out to you for being able to do that. We appreciate you so much. I’m Katie Green. I’m serving as Principal Advocate at Kameleoon. I’m responsible for managing the lovely community of experimenters that join episodes like this. The other thing I got to do was host our experimentation thought leadership awards, which Abi, you won the UXpert.

Abi Hough: Yay! It was brilliant.

Katie Green: I love a good pun. I’m so happy you won the UXpert category. You share really helpful stuff online. So tell us a little bit about yourself.

From microbiology to computer science: Abi’s path into UX

Abi Hough: Oh, right. Okay, I’ll take you way back in time then. I went to university, as you do, and I started off studying microbiology and zoology. That was my thing. That’s what I wanted to do, and I thought it was massively enjoyable. Then my course got completely canned. They just cancelled it for some reason, and I was left with a dilemma as to what I should swap to.

It came down to two things. The first was geography, because I really like geography and I found it easy to do. The other choice was computer science, and there are reasons why that was on the shortlist which I’m not going to go into. Totally selfish reasons why I decided on computer science.

Katie Green: I noticed you started your career in web development.

Abi Hough: Yes. The computer science course ran, this is late 1999, I think it was. At that time, usability wasn’t a thing. Nobody knew what usability was. My course was nearly all programming, databases, hardware, all of that traditional computer stuff. There was one course on usability at the time, and it was very niche. The lecturer thought he was amazing because he’d come up with this new thing, and for the three or four years I was there, it was the only module I actually enjoyed for the entire course.

Learning UX the hard way at LoveFilm

Anyway, I left uni, went to do the London thing like everybody does, got a job, and I was a front-end developer. I was working on a website called LoveFilm at the time. It was a bit like a modern-day version of Blockbuster. You went onto a website, chose your films, got posted a DVD in the post, watched your film, and sent it back.

At that point, we actually did some user testing on the website, because I was busy building it and designing it, thinking I was amazing. But we did some user testing, and I watched the videos of these people using what I created, and I was horrified, to be honest with you, because nobody knew what they were doing.

Katie Green: That was going to be my question. In that era, was it around 2004 or 2005 that you were there?

Abi Hough: Yeah, that sort of time.

Katie Green: So in that era, how were you making decisions on design and functionality? Was it just kind of like, oh, this looks cool, we’re going to build it?

Abi Hough: Absolutely that. It was whoever had the biggest opinion in the room. Nobody else thought about users, or whether people could actually do this. It was whatever got floated, people thought was a good idea, we built it, released it, end of story, job done.

Katie Green: I don’t think you’d be surprised to hear that’s still happening today.

Abi Hough: Yeah, I’d like to say things have moved on, but apparently not. It’s just got fancier names, and people are poking around the edges still. They’re not really understanding what their users need when they’re on a website, and a lot of stuff still gets pushed through which is of poor quality and doesn’t address the needs that really do need addressing. This is why I’m still in the industry and still banging on about the same thing 20 years later. I’m not getting very far, apparently.

Why UX and CRO still get deprioritized today

Katie Green: You’re getting far. I’ve been an experimenter for about 10 years. My favorite line is that I ran my first test in 2016, I remember it well. I think the baseline expectation is higher now. You are expected to come to the table with data. But something I’ve noticed throughout my career is that UX and conversion rate optimization, when people are cutting corners on budget, those are some of the first things to go.

I know, God, a budget cut? Never heard of her. It’s very puzzling to me, because it is what separates a fine program from an excellent program. It’s a mature digital experience. So have you found throughout your career, and I know you were Director of Optimization at one point, that usability and UX are becoming more central to the business objectives of the companies you’ve worked for, or is it plateauing?

Abi Hough: No, from my view, it’s still plateauing. Good user research and good insights are still being done, but only by a minority of companies, and rarely to the level I would expect. People are still nowhere near that. And we have even more of an issue now, and I’m going to say it if anybody’s got a bingo card, because of AI.

AI is fanning the fire of misunderstanding

AI is simply fanning the fire towards even less understanding. In my view, yes, it can assimilate huge amounts of data. It does a far better job at that than we do. But in terms of everything else, it’s still being used as a shortcut. People will feed it all into AI and go, oh look, there’s our top 10 themes, let’s run an A/B test to address that. But they don’t understand the underlying reason why that came out as a theme, and they don’t bother to investigate it, so they end up wasting time on tests that will never do anything, because they’re not actually addressing a problem or a need from a user’s point of view.

In terms of business goals, I wish more businesses would recognize the importance of aligning their goals with their users, rather than the usual thing we see, which is just profit, profit, profit, more users, more users, how many more can we convert. Oftentimes businesses don’t even consider things like retention, which is another one of my top soapbox moments. Businesses don’t understand that if they can retain customers, it is cheaper than trying to get more people in through the door who, if you’re lucky, will convert once and then they’re gone.

Companies would rather invest millions of pounds getting people in the door rather than taking care of the people who’ve already bought into whatever it is they’re selling, whether that’s a service, a product, or a brand story. There’s a general dislike of trying to understand our fellow humans, because, if I’m being honest, we’re all massively complicated. It’s a complicated job. We can’t be put into numbers and dashboards and black and white. There’s so much greyness and ambiguity and context that needs to be taken into consideration, and I think in today’s fast-paced world, that’s too much of a mountain for people to try to overcome, even though they should. So it’s a hard sell, I think.

Katie Green: I kind of want to take the first part of your answer and post it directly on LinkedIn, because one of my favorite questions to ask (and Johnny Longden actually posted me asking him this on his LinkedIn) is, everyone’s using AI. It’s kind of a requirement of the job these days, AI literacy. The thing I asked Johnny is, are you using AI to just move faster, or to ask better, deeper questions?

Synthetic audiences are, I think, the most surprising thing about this job I’m in now, because I’m an experimenter by practice. I’ve run experiments for some of the biggest e-commerce brands in the world, and I’m very fortunate for that experience. But it never even crossed my mind to use AI for synthetic users. I’m realizing it’s such a trend in experimentation, but I’m wondering, is that just helping you fail faster? Are you failing more? And even by failing, you usually learn something from it, but I don’t know if synthetic users are providing a valuable fail.

So I wanted to ask you: is there a way, in this world as we see it today, to use AI in a way that asks better questions and creates better understanding of users? Where do you see its role in usability and optimization?

Should we trust synthetic users?

Abi Hough: That’s a big question. You said these were going to be easy.

Katie Green: I lied to you to get you on the podcast.

Abi Hough: I can only give you AI’s role in terms of my own context, how I use it, as a starting point. Would I use synthetic users? No, I wouldn’t, not at this point, because no matter how much you dress it up, whatever fancy terminology you use, or how many people you try to map it against, it’s never going to be that one person sat at a website trying to figure out what to do.

Katie Green: That’s the general consensus I’m hearing. You’re not alone.

Abi Hough: It hasn’t got the real-world stressors. You could prompt it and say, imagine you’re making dinner and your toddler’s hanging off your ankle, but it still doesn’t experience adrenaline going up, or feeling like you want to punch somebody. It hasn’t got that real biological response to human stressors that people experience when they’re trying to buy something on a website, or whatever it might be.

So I’m not a fan of the synthetic user at this point. I’d rather speak to humans, and I’m talking face-to-face communication here, because there’s a lot you can pick up in body language and the way people talk, just human stuff, that actually feeds into an analysis if you’re doing those proper one-to-ones.

Where AI actually helps: organizing research

For me, AI’s place at the moment is trying to organize all of the research that I’ve got. Most research programs, or maybe I’m just a bad example, end up with everything, and then you try to organize it. That’s just the way I work. So AI, for me, is a mechanism to be able to take that mess and turn it into some kind of chain of thought, to try to understand it.

But even that has its downsides. I remember in the early days, I fed a load of research into AI and said, pull me some quotes out from these transcripts. Bearing in mind, I had done the interviews in the morning and was asking this in the afternoon, and I discovered it was just making random things up. It was making quotes up out of the transcripts. So I said, I don’t remember somebody saying that, and you get the typical response: you’re absolutely right, yes, I completely made that up. I said, well, please don’t ever do that again. So that’s where the benefit is for AI, for me.

The bigger question: how is everyone else using AI?

It’s interesting, I think the more interesting question is how normal folk are using AI, because on LinkedIn, in this industry, we’re all in our own bubble, thinking about how we’re going to use AI to make our jobs more efficient. The more interesting point to me is how Joe Bloggs Public is using AI. That’s a far bigger, and far more impactful, question. As optimizers, we can do whatever we like: optimize process, make it faster, assimilate 5,000 transcripts or 3,000 surveys, whatever it might be. But how are our consumers, the people we’re trying to influence or target or optimize for, using AI, and how does that affect the work we’re doing?

I think that’s a much more interesting question, and it’s what I’m trying to answer at the minute. I still don’t think I know. But what I do know from the real world is that people in our industry who assume normal people aren’t using AI are misguided and short-sighted. I only have to go to the supermarket, or look at people waiting at the bus stop, or speak to my own dad, who’s 80, to see they’re all using AI in various different ways. Primarily, it’s for research. They’re fed up of going to Google to search for something, that’s too much effort, so they’ll just chat into this thing instead.

And it tells them all the answers, and nine times out of 10, they don’t even question it. They just take it verbatim. This is the downside. So the interesting question for businesses and optimizers is, how do we influence what the AI is citing, to give people those responses? I think in three, four, five years’ time, the UI as we know it might not even exist in its current format, and we’ll all be doing things for no apparent reason. So that upstream element is what I’d be focusing on.

I’m lucky, I’m planning on retiring in five years, so it won’t be my problem. But if I were younger in my career, that’s the kind of thing I’d be looking at, because that’s where the money will be, in my view. I might be completely wrong, but it’s more interesting right now to look at that than to look at another website and say, your form is terrible, you should improve that. There’s only so many times you can do that before you’re like, I’ve been in this industry for too long. So for me personally, I’m using the AI shift as a pivot point, still related to users, massively related to that, but it’s taken things away from the actual website UI, further upstream, to figure out what’s going on up there. It’s an interesting place to be.

Katie Green: I think people are. The thing I talk to people about all day is literally my job at Kameleoon, which is incredible for someone like me. Also, my dad is 80, and I’ve never thought about asking him how he uses AI. Shout out to Johnny. I’m probably going to call him today and ask, dad, how are you using AI? I fear the answer, but honestly, it’s probably because he takes care of about a thousand outside cats. I know he’s such a cat person.

Abi Hough: That’s different to my dad. My dad is an outdoor garden rail enthusiast, so he uses it to help plan his track layouts, get planting suggestions, and find weird and wonderful stock to put on the track.

Katie Green: It’s like the smart ones are optimizing for how to be part of those answers.

Abi’s Monday morning advice

Katie Green: I realize we’re coming up on time, so there’s one question I like to ask all of my guests at the very end, which I call the Monday morning advice. It’s assuming people are listening to this on Friday, and after the weekend, on Monday, they’re going back to work thinking, okay, how am I using AI? How am I taking stock of this? Is it answering real questions about my user problems? Is it helping me understand people better? Where do they start, in your opinion?

Abi Hough: Step away from the computer. Seriously, step away from the computer, and go do some old-school analysis to see if the two tally. Do your own analysis manually. It doesn’t matter how long it takes. Just do a benchmark between what AI is telling you and what you, as a human, consider to be true, and see where you’re at between the two. You’ve probably guessed I’m a big advocate of actually speaking to people, and not getting so technologically isolated that you lose that connection with the work you’re doing, and why you’re doing it.

So that would be my Monday advice: go away and speak to somebody, do some proper user sessions. It’s a morning and afternoon’s worth of work, and it doesn’t have to be fancy. Go do some guerrilla testing, get a coffee, and ask somebody’s opinion on what you’re building while you’re having a latte and a slab of cake. Don’t ever lose sight of what the work is that you’re doing, and don’t assume AI has all of the answers, because it doesn’t, nine times out of 10, or it certainly can’t necessarily interpret them correctly.

So that would be my one piece of advice: if you’re in user research in any way whatsoever, don’t lose the reason why you started doing this work in the first place. Always keep it there to your side, like a little devil and angel on your shoulder. You’ve got the angel going, go speak to the humans, and you’ve got the devil going, I’ll just do it all via AI because I want to know, whatever. Don’t do that to yourself. Keep true to it and keep going. Maybe adjust where the ship is going, so maybe not so much on the website, maybe a bit further upstream elsewhere. That would be my one piece of advice to people, I think.

Balancing qualitative and quantitative research

Katie Green: I think that’s great advice. I feel like the existence in between the devil and the angel on your shoulders is balance. If I were to answer that question for somebody who doesn’t have a user testing or user experience background at all, I’ve done user testing in the sense that I’ve asked users to test things, but I’ve never held my own blind study about something. I feel like this underlines the importance of qualitative and quantitative research.

Abi Hough: Oh, absolutely. You can’t rely on either in isolation. It has to be a mix of both. Our industry is very guilty of relying on numbers over everything else, and it’s been the case ever since I’ve been working in this industry. My view has always been, you can’t weigh them differently. They have to have an equal amount of weight on both sides, which is why I get very concerned about automated A/B testing and things just going out in the wild and being regulated based on numbers alone. That concerns me, but that’s an entirely different podcast episode.

Katie Green: That’s episode two. I was thinking it would be fun to have you and a data scientist, because I just interviewed a data scientist I really love chatting to, and I know you would too. She’s so smart, has an advanced degree in statistics. You two are the devil and the angel on my shoulders, depending on the day.

Where to find Abi

Katie Green: I want to encourage anybody listening who’s made it this far to follow you on LinkedIn, because you post a lot of really helpful stuff that has opened my eyes to parts of the business. I’m an experimentation generalist, I’ve worked more on the marketing side of things, not nearly as technical as some of my colleagues, but you’ve really opened my eyes to parts of the experimentation process that are actually really key. I previously worked at a product that’s an analytics tool, so I got really into the numbers for a second, but you’re totally right, it’s about finding that balance. So everybody listening should totally follow Abi on LinkedIn. Thank you again so much for making the time, especially so last minute, to be on Unite Voices.

Abi Hough: You’re welcome, more than a pleasure. And yeah, read the LinkedIn stuff. I don’t know why I post on there sometimes, it’s a wasteland. But if you’re doing it for anybody, keep doing it for me. I like posting there because usually I’m saying something different from everybody else. I’m just causing trouble.

Katie Green: Cause trouble. All right, thank you again, Abi.

Abi Hough: Perfect, thank you.