A/B Test Sample Size Calculator
Running an A/B test without calculating the required sample size first is one of the most common mistakes in experimentation. Too few visitors and your test results will be unreliable. This calculator uses the standard two-proportion z-test formula to compute the minimum sample size needed per variation, given your baseline conversion rate, the minimum detectable effect (the smallest lift you care about), your significance level (typically 5%), and your desired statistical power (typically 80%).
A/B test sample size formula
n = (z(alpha/2) + z(beta))^2 x (p1 x (1 - p1) + p2 x (1 - p2)) / (p1 - p2)^2
Where: p1 = baseline conversion rate, p2 = expected treatment rate (p1 x (1 + MDE/100)), z(alpha/2) = z-score for the significance level (1.96 for alpha = 0.05), z(beta) = z-score for the power level (0.842 for power = 0.80). This is the standard Pearson two-proportion z-test formula, widely used in statistical practice and cited in statistics textbooks and NIST statistical reference materials.
A/B testing best practices
- Always calculate sample size before starting: do not stop a test early just because results look promising. Early stopping inflates false positive rates.
- Test one change at a time in a standard A/B test. Testing multiple changes simultaneously requires a multivariate test with a larger sample.
- Run tests for at least one full business cycle (typically one to two weeks) to account for day-of-week effects even if you reach sample size faster.
- Pre-register your hypothesis and success metric before launching to avoid HARKing (Hypothesizing After Results are Known).
- Use a lower MDE (detect smaller changes) only when the cost of running the test is low relative to the value of the decision. For high-traffic pages, a 5-10% MDE is reasonable; for low-traffic pages, focus effort on larger changes.
A/B test sample size: frequently asked questions
What is an A/B test?
An A/B test (also called a split test or randomized controlled experiment) is a method of comparing two versions of something (a webpage, email, ad, or product feature) to determine which performs better. Version A is the control (current version) and Version B is the treatment (proposed change). Users are randomly assigned to each version and outcomes are measured.
Why does sample size matter in A/B testing?
Sample size determines whether your test has enough statistical power to detect a real difference if one exists. Too small a sample and you risk a false negative (missing a real effect). Too large and you waste time and traffic. The required sample size depends on your baseline conversion rate, the size of the difference you want to detect, your significance level (alpha), and your desired power (1-beta).
What is statistical significance in an A/B test?
Statistical significance (controlled by the significance level, alpha, typically 5% or 0.05) is the probability of declaring a winner when there is actually no difference. A p-value below your alpha threshold means the result is statistically significant. A 95% confidence level (alpha = 0.05) means there is at most a 5% chance of a false positive.
What is statistical power?
Statistical power (1 minus beta) is the probability that your test correctly detects a real difference when one exists. A power of 80% means there is an 80% chance of detecting the minimum detectable effect if it truly exists, and a 20% chance of a false negative. Higher power requires larger sample sizes.
What is a minimum detectable effect (MDE)?
The minimum detectable effect (MDE) is the smallest change in conversion rate that you want to be able to detect. A smaller MDE requires a larger sample size. Set your MDE based on the smallest change that would be practically meaningful for your business, not the smallest change you hope to see.
Official sources
- NIST/SEMATECH: NIST Engineering Statistics Handbook - Sample Size for Two-Proportion Test.
- National Institutes of Health: A-B Testing and Statistical Power (PubMed Central).
Reviewed by the CalculatorHub team, edited by James Graham, 14 June 2026. See our methodology.