What is Hypothesis testing?

A/B testing is a process used to determine if there is enough statistical evidence behind your hunch to conclude that it is likely correct.

The hypothesis testing procedure consists of taking some data and ruling out certain potential causes.

Hypothesis testing steps

1. Start with the page as it is now, or use a variation from an earlier test.
2. Consider establishing an alternative plan to the proposed design. This may be referred to as a variation or treatment B and should have its own set of predefined criteria for determining success.
3. A calculator can help determine the sample size required to detect a difference in performance. The formula is based on baseline conversion rate, desired statistical power (i.e. how accurate you want the results), and minimum difference in performance that must be detectable.
4. Launch the test and let it run until a predefined sample size has been reached. People often can’t restrain themselves from testing their results prematurely, which greatly compromises data reliability.

This gives the designer a chance to see which variations perform better in a statistically significant manner.