
Why Your Brain Is Your Worst Enemy When A/B Testing
- Finding relations between unrelated events
- Can’t handle sample size
- Looking for and Interpreting information to confirm your own beliefs
- Seeing patterns when there are none and thinking past events influence future probabilities
- Thinking what’s in front of him is everything he needs to draw conclusions
- Basing all subsequent thinking on the first piece of information received
- Throwing reason out the window when ego and emotion are involved
- Preventing you from thinking like your customers
- Taking things for granted because you’ve always done them this way
- Overestimating the degree at which people agree with you
Did you know that we, humans, SUCK at statistical reasoning? We’re also irrational, flawed, and subjective. Why? Because we’re influenced by a list of cognitive biases.
You can perfectly live (but biased) without knowing about them, but if you’re here, it means you’re A/B Testing or contemplating to start so. A/B Testing is a science experiment which must by definition be objective to provide actionable data. Cognitive biases are then a real threat. To get that out of the way, cognitive biases are personal opinions, beliefs, preferences that influence your ability to reason, remember, evaluating information. Let’s go down the rabbit-brain (sorry, had to do it) and make sure we’re not subjectively influencing our tests too much by being our flawed (but lovable) selves.
This article is the last in our series on A/B Testing Mistakes. You can get the whole shebang as an epic Ebook HERE by the way.
Here’s what your brain does and what we’ll cover:
- Will find a relationship between unrelated events
- Can’t handle sample size
- Will subconsciously look for and interpret information to confirm its own beliefs
- Will see patterns when there are none
- Thinks that past events influence future probabilities
- Thinks what’s in front of him is everything he needs to draw conclusions
- Will base all subsequent thinking on the first piece of information received
- Will throw reason out the window as soon as ego/emotion is involved
- Prevents you from thinking like your customers
- Takes things for granted because you’ve always done them this way
- Overestimates the degree at which people agree with you
1 Finding relations between unrelated events
Remember how we talked about external validity threats?
Well, if you didn’t know about them, you could assume that the lift you see was indeed caused by the put a pink CTA in your variation. Not because there is a storm coming that scared people into buying your product for example. You’d have been victim of the illusory correlation bias. You perceived a relationship between 2 unrelated events.
2 Can’t handle sample size
When we talked about fixing a sample size before testing, we actually were also partially preventing another bias called insensitivity to sample size. Our brain struggles to apprehend correctly sample size and underestimates variations in small samples. Example from D. Kahneman’s book: A certain town is served by two hospitals. In the larger hospital about 45 babies are born each day, and in the smaller hospital about 15 babies are born each day. As you know, about 50% of all babies are boys. However, the exact percentage varies from day to day. Sometimes it may be higher than 50%, sometimes lower. For a period of 1 year, each hospital recorded the days on which more than 60% of the babies born were boys. Which hospital do you think recorded more such days?- The larger hospital
- The smaller hospital
- About the same (that is, within 5% of each other)
3 Looking for and Interpreting information to confirm your own beliefs
Confirmation bias is also not to be ignored. It’s the fact that you will seek, interpret or focus on information (subconsciously or not) that confirm your beliefs.
- Does it really prove your hypothesis in an objective way?
- Did you push this idea with your ego on the line to begin with?
- Aren’t there other factors that could have produced (or at least considerably helped) this lift?
4 Seeing patterns when there are none and thinking past events influence future probabilities
Let’s tackle two biases at once. The clustering illusion: the intuition that random events which occur in clusters are not really random events. A fun story illustrating the clustering illusion is the one about the Texas Sharpshooter. It’s the story of a Texan who shoots on the blank wall of his barn then draws a target centered where his shots are most clustered. And then he proceeds to brag about his shooting skills.
5 Thinking what’s in front of him is everything he needs to draw conclusions
This is what D. Kahneman called “what you see is all there is” in his book. It’s the notion that we draw conclusions based on information available to us, i.e. in front of our eyes. Doesn’t sound too bad, uh? Let’s try with this: a bat and a ball together cost $1.10. The bat cost $1.00 more than the ball. How much does the ball cost?
6 Basing all subsequent thinking on the first piece of information received
Called the Anchoring bias, it’s the fact that we allocate more importance to the first piece of information we’re given. Here is an example from a study by Fritz Strack and Thomas Mussweiler: 2 groups of people were asked about Gandhi’s age when he died. The first group was asked if it was before 9 years old or after. The second if it was before 140 or after. Both answers are pretty obvious. But what was very interesting, were the answers from both groups when they asked to guess Gandhi’s actual age when he died. Answers from the first group had an average of 50 vs 67 for the second. Why such a difference? Because they were subconsciously influenced by their respective first questions. Here’s a picture illustrating a similar study:
7 Throwing reason out the window when ego and emotion are involved
This one can be painful. You put your heart and soul in a redesign, you spend hours on it and you’re super-proud of what you made. Then you test it. And it flops. Badly. Ouch … What do you do? “Screw these people my design is perfect, they don’t know what they’re talking about!” Or when you bring the news to your boss he says: “No way, this design is clearly better, go with this one.” No! Be strong, I know your pain. It’s hard but that’s one of the reasons you’re A/B Testing, to not lose money on redesigns or decisions based on guts and personal opinions. But rather do things people actually want. Swallow your frustration, go back to the hypothesis that led to this redesign, aaand to the drawing board again. Being able to throw out hours —days even, of work through the window if your data say so, is a sign you’re truly becoming data-driven. It’s freakishly hard though.8 Preventing you from thinking like your customers
Called the curse of knowledge, it’s when you’ve been so absorbed by a subject that you’re having a hard time thinking about problems like someone who has little to no knowledge about it.
9 Taking things for granted because you’ve always done them this way
Functional fixedness is when you’re stuck in linear thinking. You see an iron, you only think about its obvious use—for clothes (linear thinking), you don’t think to use it as a toaster (lateral thinking) if you don’t have an oven (or a real toaster). This is called lateral thinking, or “thinking out of the box”.
- Turn your problem on its head, try to solve the opposite
- Think about the most obvious, stupid solution
- Fragment your problem in series of small, very precise problems
- Don’t be satisfied with finding one solution
- Flip your perspective, how would you think about this problem if you were an engineer, a scientist, a complete beginner?
10 Overestimating the degree at which people agree with you
“Everyone hates popups.” Well, YOU hate them. They usually annoy people a bit but they actually convert quite nicely when used the right way (i.e they don’t pop in your face the second you arrive on a website). This type of bias (aka the false consensus effect) can be tricky when gathering feedback. We can sometimes believe strongly in something, and think we’re on the side of the majority when we’re really not. We tend to assume people have the same opinions we do. It’s better to get feedback from individuals rather than from a group. Group feedback will be plagued by biases. But when asking individuals, don’t get caught up in personal preferences either. Always take everything with a grain of salt. Be careful not to do it yourself too! When you think that something is doing as good as it could be, stop in your tracks and reconsider. Test what would have the most impact, but also test what is doing fine. You can always do better. This concludes our article on cognitive biases and our series on A/B Testing Mistakes. It was quite a ride! Riddled with self-doubts, highs, lows … as it was the first time I ever wrote this type of content (length and topic). See you next time :-)- If this article was helpful in any way, please let met know. Either leave a comment, or hit me up on Twitter @kameleoonrocks.
- If you’d like me to cover a topic in particular or have a question, same thing: reach out.