If you run A/B Tests (or just got into it) you might have come across what is called an A/A test and wondered what the heck it is and when you should (if at all) run it. Well, I got you covered. Here is a short and sweet answer :)
What’s an A/A testAn A/A test is an experience where the variation is an exact copy of the original page.
Why do an A/A testThere is actually not much use to an A/A test except making sure your A/B testing solution is set up properly and the results you get are indeed accurate. If everything is right, you should get virtually the same conversion rate at the end (since both variations are identical). Should it not be the same, you got a serious problem on your hands.
How come your results can be inaccurateUsually, results are very close. But it can very well happen to get a winner in your A/A test, even though you KNOW it’s not possible. What could be causing this mishap?
- Your A/B Testing solution has a problem (it could be badly set up, not recording data properly, …).
- You have a false positive (details on this in our article on interpreting A/B tests results), which with a statistical significance of 95% has 5% chance of happening.
- You ended your test too early and got a fake result (if you need help on when to stop your test, we touched on this here).
You may also enjoy these articles