How to write an A/B test plan before you run the test
Eight steps to a one-page A/B test plan, written before the test starts, so the result cannot be argued into what someone hoped for.

You are about to run an A/B test. Before it starts, write one page that says what you expect, what you will measure, how long you will run, and what you will do with each outcome. This guide shows you how.
Product managers who are about to run an A/B test.
A one-page test plan, written before the test starts, so the result cannot be argued into whatever someone hoped for.
Why write the plan first
Once results are in, almost any number can be read two ways. A small lift becomes a good start. A flat result becomes proof that the audience was wrong. People are not lying when they do this. They are reading a result they already care about.
A plan written in advance takes that away. You fix the rules while nobody knows the outcome. Then the result is checked against the rules, not the other way round.
Decide all of this before the test starts: the hypothesis, the primary metric, guard metrics, the minimum detectable effect, the sample size, the duration, and the decision rule. The eight steps below cover each one, in order.
Write the hypothesis
Use this format and fill every gap:
The evidence is what you have actually seen. The change is the one thing you will alter. The audience is who will see it. The metric is the number you expect to move, and the amount is how far. If you cannot fill a gap, you are not ready to test yet. PikoPM's Hypothesis card is one place to write it down.
Choose one primary metric
One number decides the test. Name it now. If you watch five metrics and call the test a win when any one of them moves, you will usually find a win. The primary metric should be the same metric named in your hypothesis.
Choose one or two guard metrics
Guard metrics are numbers that must not get worse. A change can lift your primary metric by hurting something else. Pick one or two that would catch that, and write them down next to the primary metric. Keep the list short. A long list of guards is another way to leave room for argument.
Set the minimum detectable effect
The minimum detectable effect is the smallest change worth shipping. Ask yourself: below what size would I not bother to ship this? That size is your answer. Set it before you touch a calculator, because the sample size depends on it. A smaller effect needs more visitors.
Work out the sample size
Use a sample-size calculator. It asks for four inputs:
- the baseline rate of your primary metric
- the effect you want to detect, from step 4
- 95% significance
- 80% power
It returns the number of visitors you need in each variant. Write that number in the plan.
Run whole weeks, at least one
Divide the sample size by the daily visitors each variant will get. That gives you days. Then round up to whole weeks, and never run less than one. People behave differently on weekdays and at weekends. A test that covers only part of a week can see only part of that pattern.
Do not stop early because the numbers look good
Early numbers swing. A lead on day three is not a result. Write the end date and the sample size into the plan, and wait for both. If you would stop the moment the chart looks good, the test is no longer measuring anything.
Write the decision rule
Before you see any results, write what you will do in each case: ship, do not ship, or iterate. Be exact. Say what the primary metric has to do, and how far a guard metric may fall before the answer changes. A good rule can be applied by someone who has never heard your opinion of the change.
A worked example
The numbers below are illustrative. They are not from a real test.
Say you want to test a change to checkout. Checkout conversion is 4.0%. The smallest lift worth shipping is 10% relative, which takes conversion to 4.4%.
At 95% significance and 80% power, a sample-size calculator says you need about 39,500 visitors per variant.
You get 10,000 checkout visitors a day, split evenly between two variants. That is 5,000 per variant per day, so the sample takes 8 days. Eight days is not a whole number of weeks, so round up to two full weeks.
Your hypothesis, in the format from step 1:
The finished plan fits in one table:
| Part of the plan | Decision |
|---|---|
| Primary metric | Checkout conversion, baseline 4.0% |
| Guard metrics | Average order value and refund rate |
| Minimum detectable effect | 10% relative, 4.0% to 4.4% |
| Sample size | About 39,500 visitors per variant, at 95% significance and 80% power |
| Duration | Two full weeks, rounded up from 8 days at 10,000 visitors a day |
| Decision rule | Ship if conversion rises significantly and neither guard metric gets worse by more than 2% |
Now imagine conversion rises, but refund rate has got worse by more than the 2% limit. Nobody has to debate it. The rule was written first, and it says do not ship as it stands.
The plan above does not say what to do if conversion rises and a guard metric breaks the limit, beyond not shipping. If you want an iterate branch, write what it means before you start. For example: if conversion rises but a guard metric gets worse by more than 2%, keep the change off, find what made the guard metric worse, and run a new test with a new plan.
Check the page before you start
- The hypothesis has evidence, change, audience, metric and amount.
- There is exactly one primary metric.
- One or two guard metrics are named.
- The minimum detectable effect is written down.
- Sample size comes from a calculator at 95% significance and 80% power.
- Duration is whole weeks, at least one.
- The end date is fixed, and nobody stops early.
- The decision rule covers ship, do not ship and iterate.
Then share it
Send the page to the people who will care about the result, and do it before launch. If someone disagrees with the metric or the rule, you want to hear it now, while it costs nothing. After the test, they have already agreed to the terms.


