A/B test sample size calculator

How is the sample size for an A/B test calculated?

For a conversion-rate test you compare two proportions: the control rate p₁ and the rate p₂ you hope the variant reaches. The visitors needed in each group are n = (z₁₋α/₂ × √(2 × p̄ × (1 − p̄)) + z₁₋β × √(p₁(1 − p₁) + p₂(1 − p₂)))² ÷ (p₁ − p₂)², where p̄ = (p₁ + p₂) ÷ 2, z₁₋α/₂ is the normal quantile for your significance level (1.960 for α = 0.05, two-sided) and z₁₋β is the quantile for your power (0.842 for 80%). This is the formula behind R’s power.prop.test, and it does not use a continuity correction. The three levers are the baseline rate, the size of the difference you want to be able to detect and your tolerance for error: halving the detectable difference roughly quadruples the sample.

Worked example: 5% baseline, detect a lift to 6%

Suppose 5% of visitors buy and you want an 80% chance of detecting a lift to 6% (a 20% relative improvement) at α = 0.05, two-sided. Then p̄ = 0.055, and the formula gives 8157.7, which rounds up to 8,158 visitors per variant, 16,316 in all. Asking for 90% power raises it to 10,921 per variant. Trying to detect only 5% → 5.5% (a 10% relative lift) takes 31,234 per variant. At 2,000 eligible visitors a day split evenly, the first test needs 9 days. As a check on the formula itself, the case in R’s own documentation, 50% against 75% at 90% power, comes out as 76.7 per group, the 76.7 that R prints.

What is the minimum detectable effect, and how do I choose it?

The minimum detectable effect (MDE) is the smallest true difference you would still care about, not your prediction of what the variant will do. Choose it from the business value: if a 2% relative lift would not change a decision, there is no point paying for the traffic to detect it. You can enter it as a relative change (20% of a 5% baseline is 1 percentage point) or as absolute percentage points; the page shows both so you can check you meant the right one. A test powered for a large MDE will usually miss smaller real effects, and a non-significant result then means “no large effect”, not “no effect”.

Why not just stop the test when it becomes significant?

Because repeated looks inflate the false positive rate. The sample size above is computed for a single analysis at the planned end of the test. If you check the p-value every day and stop at the first p < 0.05, you give random fluctuations many chances to cross the line, and the real false positive rate ends up well above 5%. Fix the sample size and the stopping rule in advance and analyze once. If you must monitor early, use a method designed for sequential testing, which adjusts the thresholds; this calculator does not.

What changes when I test more than two variants?

Every variant compared against the control is another chance at a false positive. A simple, conservative fix is the Bonferroni correction: divide α by the number of comparisons. With three variants (two comparisons) α becomes 0.025 for each, and the 5% → 6% example grows from 8,158 to 10,882 per variant. Choose “Bonferroni” in the calculator to apply it. Bonferroni is deliberately cautious; other procedures give more power, but they are beyond this tool. Testing many metrics or many audience segments at once has the same problem: decide the single primary metric beforehand.

One-sided or two-sided test?

A two-sided test detects a difference in either direction, which is the default for A/B tests because a variant can also make things worse. A one-sided test puts all of α on one direction, so it needs fewer visitors (6,426 instead of 8,158 per variant in the 5% → 6% example) but cannot declare the variant harmful. Choose one-sided only if you decided before the test that a result in the other direction would lead to exactly the same action as no difference.

Limits of this method

The formula assumes each visitor converts or not independently, with a fixed true rate in each group, and a large enough sample for the normal approximation. It is less reliable for very low rates, since the sample must hold enough conversions: check that n × p is well above ten in both groups. It does not account for novelty effects, seasonal swings, or users who see both variants on different devices, any of which can break independence. Use visitors who could actually reach the tested page as the traffic number, not all site visitors.

For general information only. Sources: R documentation, stats::power.prop.test (CRAN reference manual; the reference values 76.7, 73.37 and 10451937 reproduce exactly); standard two-proportion sample-size formula as in J. L. Fleiss, B. Levin and M. C. Paik, Statistical Methods for Rates and Proportions (the book was not opened; the formula is checked against R). The Bonferroni correction and the cost of peeking are standard results of multiple-testing and sequential-analysis theory. Related: two-proportion z-test calculator, statistical power calculator and sample size calculator. How we check our calculators.