What is P2BB?

The term Probability-to-Be-Best indicates how likely is a certain variation to outperform the variations used in a test. The tool uses data from every variation in the test.

An A/B test engine evaluates each variation for an online experiment, calculating the probability of that variation to be best or P2BB. This quality lets the tester discern success out of this stage and move confidently on to collecting conclusive results.

For a standard A/B test, this threshold is used to declare the winning variations and the losers. If the Probability to be best of one variation goes over 94 or 95%, then it can be concluded as the final winner.

The input is dependent on the experiment’s clearly set objective and adjusted KPIs. For a binary objective, the input includes both denominator and numerator values–for example, for clicks-to-impressions (CTR) experiments that involve users clicking on an element to reveal a message below it, this number would be “1 / Number of impressions”.

For revenue-based experiments, the input data is total sales and conversion statistics.

These experiments require experimenters to adhere to strict traffic allocation rules.

Any changes to the test or revisions made during testing require that the new variant be treated as a completely separate study.

When settings are changed for any test variation in the middle of traffic, a brand new version to be tested is created to reset the traffic distribution and corresponding metrics. The ‘Probability to Be Best’ calculation then begins anew.