For A/B Testing, there two main statistical methods:
The frequentist approach allows a simple read on the results reliability thanks to a confidence level: with a level of 95% or more, you have a 95% chance of obtaining the same result should you reproduce the experiment in the same conditions. But this method has a downside: it has a « fixed horizon », meaning the confidence level has no value up until the end of the test.
The bayesian approach provides a result probability as soon as the test starts. No need to wait until the end of the test to spot a trend and interpret the data. But this method also has prerequisites: you need to know how to read the confidence interval given to the estimations during the test. With every additional conversion, the trust in the probability of a reliable winning variant improves.
To reliably answer your optimization hypothesis, and allow you to make the right decisions, our statistical engine aligns with your needs.
On high-traffic tests—which duration would then be shorter, the frequentist approach is preferable since it has the benefit of being simple and reliable.
On tests with a lower traffic—which duration will then be longer, the bayesian approach will allow you to make decisions quickly based on a clear trend.
To develop our new statistical engine, our data scientists relied on 2 years of statistical research and our experience acquired with hundreds of clients and billions of tested visitors.
Make the right decisions
by relying on the statistical significance of your tests results.
Shorten your decision cycle
by acting if a strong trend appears before the end of your tests.
Focus on what matters
by letting our technology provide reliable results to your questions.
Our data scientists are among the best statistical experts. They count more than 5 years in statistical research and have published several papers on these methods. Don't hesitate to contact us if you want more information.