Plan Statistically Valid Experiments

Free A/B Test Sample Size Calculator

Calculate the number of visitors you need per variation to run a statistically significant A/B test. Set your baseline conversion rate, minimum detectable effect, and confidence level to get an accurate sample size estimate.

Significance Level

Statistical Power

Traffic Split

Variations

Including control

Test Type

Detects both improvements and regressions

MDE Type

Percentage change relative to the baseline

Baseline Rate

Daily Visitors

Sample Size Calculations

Relative MDE	Absolute Effect	Visitors / Variation	Conversions / Variation	Days	Weeks
5%	0.25%	122,123	6,106	25	4
10%	0.5%	31,232	1,561	7	1
%	0.1%	752,708	37,635	151	22

What is A/B Test Sample Size?

Sample size is the number of visitors each variation in your A/B test needs before you can draw a reliable conclusion. Run the test with too few visitors and you risk a false positive — concluding that a change helped when it didn't — or a false negative — missing a real improvement.

A proper sample size calculation balances four factors: your current conversion rate, the smallest improvement worth detecting, the confidence level you require, and the statistical power of the test. Getting these right before you launch prevents wasted traffic and misleading results.

How Does the Sample Size Formula Work?

The calculator uses the standard two-proportion z-test formula for comparing conversion rates between a control and a variant:

n = (Z_α + Z_β)² × [p₁(1−p₁) + p₂(1−p₂)] / (p₁ − p₂)²

p₁ — your baseline conversion rate

p₂ — the expected conversion rate after improvement (p₁ × (1 + MDE))

Z_α — z-score for your chosen significance level (e.g., 1.96 for 95%)

Z_β — z-score for your chosen power level (e.g., 0.84 for 80%)

Understanding the Key Parameters

Baseline Conversion Rate

Your current conversion rate before making changes. A lower baseline requires more visitors to detect the same relative change.

e.g., If 5 out of 100 visitors convert, your baseline is 5%

Minimum Detectable Effect (MDE)

The smallest relative improvement you want your test to detect. A smaller MDE requires a much larger sample size.

e.g., 20% MDE on a 5% baseline means detecting a lift to 6%

Statistical Significance

The confidence level — the probability that a detected difference is real and not due to random chance (1 minus the false positive rate).

e.g., 95% significance means a 5% chance of a false positive

Statistical Power

The probability that the test will detect a real effect when one exists. Higher power reduces the chance of missing a genuine improvement.

e.g., 80% power means a 20% chance of missing a real effect

Test Type

Two-tailed tests detect both improvements and regressions. One-tailed tests only look for changes in one direction, requiring fewer visitors.

e.g., Use two-tailed unless you only care about improvement, never regression

Common Mistakes in A/B Test Sample Size Calculation

Stopping the test early when results look promising — this inflates your false positive rate and leads to unreliable conclusions
Setting the MDE too small — a 1% relative lift on a 2% conversion rate requires millions of visitors; be realistic about what matters
Ignoring multiple variations — each additional variation increases the total sample needed; a test with 4 variations needs roughly 4× the visitors of an A/B test
Not accounting for low-traffic pages — if your page gets 500 visitors per day, a test requiring 50,000 per variation will take over 3 months
Forgetting weekday/weekend effects — traffic patterns vary throughout the week, so always run tests for full weeks (7, 14, 21 days)
Reusing traffic for sequential tests without adjusting — running back-to-back tests on the same audience increases the overall false positive rate
Using one-tailed tests to reduce sample size — one-tailed tests miss regressions, which can silently hurt your conversion rate

Sample Size Quick Reference

Approximate visitors per variation at 95% significance and 80% power (two-tailed):

Baseline Rate	5% MDE	10% MDE	20% MDE
2%	~306,000	~77,000	~19,500
5%	~118,000	~30,000	~7,700
10%	~56,000	~14,500	~3,800
20%	~25,000	~6,500	~1,700

A/B Testing Best Practices

Always calculate your required sample size before launching a test — never start without knowing how long the test should run
Run tests for full-week increments to account for day-of-week traffic patterns
Set a realistic MDE — focus on changes that would meaningfully impact your business, typically 10-20% relative lift
Use 95% significance and 80% power as your defaults — they provide a good balance between sensitivity and practicality
Avoid peeking at results before the test reaches its calculated sample size
Document every test with a hypothesis, sample size calculation, duration, and outcome for your team
Prefer two-tailed tests in most cases to catch both improvements and regressions

Frequently Asked Questions

It depends on your baseline conversion rate and the minimum effect you want to detect. For a typical e-commerce site with a 3% conversion rate looking to detect a 15% relative improvement, you would need approximately 23,000 visitors per variation at 95% confidence and 80% power.

A good MDE is the smallest change that would be worth implementing. For most e-commerce sites, a 10-20% relative MDE is practical. Setting the MDE below 5% usually requires impractically large sample sizes unless you have very high traffic.

A two-tailed test checks whether the variant is either better or worse than the control. A one-tailed test only checks for improvement in one direction. Two-tailed tests are recommended in most cases because they also catch regressions — you want to know if a change hurt performance.

No. Stopping a test early because results look promising is one of the most common A/B testing mistakes. Statistical significance fluctuates over time, and early results often reverse. Always run the test until it reaches the pre-calculated sample size.

Statistical significance (confidence level) controls the false positive rate — the probability of declaring a winner when there is no real difference. Statistical power controls the false negative rate — the probability of detecting a real difference when one exists. Both matter: significance prevents false wins, power prevents missed opportunities.

At minimum, until you reach the required sample size. Always run for at least one full week (ideally two) to account for day-of-week effects. Enter your daily visitor count in the calculator above to estimate the test duration.

Yes, but with diminishing returns. A larger sample size reduces the minimum detectable effect and increases the precision of your results. However, the calculated sample size already accounts for your desired confidence and power — going beyond it provides marginal additional benefit.

Low-traffic sites can still run valid A/B tests, but need to accept tradeoffs: increase the MDE (only detect larger effects), reduce the number of variations, or run the test longer. Focus on high-impact changes where the expected lift is large enough to detect with your available traffic.

Free A/B Test Sample Size Calculator

What is A/B Test Sample Size?

How Does the Sample Size Formula Work?

Understanding the Key Parameters

Common Mistakes in A/B Test Sample Size Calculation

Sample Size Quick Reference

A/B Testing Best Practices

Frequently Asked Questions

How many visitors do I need for an A/B test?

What is a good minimum detectable effect (MDE)?

What is the difference between one-tailed and two-tailed tests?

Can I stop my test early if results look significant?

What is the difference between statistical significance and statistical power?

How long should I run my A/B test?

Does a larger sample size make results more reliable?

What if I have very low traffic?

Related Reading