How to work out an AB test sample size
A plain guide for PMs running their first A/B test: the three inputs you need, the steps to calculate sample size, and options when traffic is low.

If you are running your first experiment, knowing the ab test sample size you need is the single best way to avoid wasted work. You only need three inputs: baseline conversion, the minimum detectable effect (MDE) you care about, and the significance and power you will test at. Do this before you launch, not after.
In brief
Step 1 — baseline conversion
Step 2 — minimum detectable effect (MDE)
Step 3 — significance and power
Step 4 — calculate and interpret
Why stopping when it looks significant is not a test
Peeking at results and stopping as soon as you see a p<0.05 inflates the false positive rate. The statistics behind significance assume a fixed sample size decided in advance. If you stop early you break that assumption and the reported p-value is no longer valid.
If you treat tests like a live scoreboard, you will get misleading winners. Decide sample size and stopping rules up front.
What to do when you do not have enough traffic
You have four practical options when required sample size exceeds what your site can deliver in a reasonable time:
- Increase the MDE you’ll detect. A larger MDE requires fewer visitors, but it only finds big changes.
- Reduce the number of variants. Fewer variants means more traffic per variant.
- Extend the test duration. If the metric is stable and seasonal effects are accounted for, waiting is valid but slower.
- Use alternative methods: run a longer pilot, use cohorts, or consider Bayesian approaches that accept different trade‑offs. Pick one method and pre‑register it in your experimentation plan.
Checklist before you ship the test
- Baseline conversion measured on a stable window
- MDE decided and documented
- Significance and power set, and sample size computed
- Stopping rules and analysis plan written down
- Traffic and timing checked against required sample
Worked example
You have a baseline conversion of 4.0% and you care about a 10% relative uplift — that is, 4.4% (from 4.0% to 4.4%). You choose 95% significance and 80% power. Plugging those inputs into a standard sample‑size calculation gives about 39,500 visitors per variant.
Interpretation: you need roughly 39,500 people to see the control and 39,500 people to see the variant to have a good chance of detecting that 0.4 percentage‑point lift at the chosen confidence levels. If your site gets 2,000 eligible visitors per week, that would take about 20 weeks per variant — so either increase the MDE you care about, reduce variants, or accept a long test.


