Experiment readout calculator
A two-proportion z-test tells you whether the difference between two conversion rates is bigger than the noise you would expect from sampling alone. The confidence interval on the difference is the more useful output, because it shows the range of effects your data is consistent with.
Experiment readout, step by step
The p-value answers one narrow question: if there were truly no difference, how often would you see a gap this large by chance? Below 0.05 is the usual bar. It does not tell you how big the effect is, how likely your hypothesis is to be true, or whether shipping is a good idea.
The confidence interval does better work. A result of "+2.1%, 95% CI −0.4% to +4.6%" is honest: the test leans positive but is consistent with a small loss. A result of "+2.1%, CI +1.8% to +2.4%" is a decision. Reporting the interval rather than a verdict is the single biggest improvement most teams can make to how they read experiments.
Where this number misleads you
Significance is not importance. With enough traffic, a 0.1% lift is statistically significant and commercially irrelevant; with too little, a real 20% lift fails to reach significance and gets abandoned. And a test stopped the moment it crossed 95% is not a 95% result — peeking inflates the false-positive rate to roughly a third.
Questions people ask
What does statistical significance actually mean in an A/B test?
It means that if there were genuinely no difference between the variants, a gap at least as large as the one you observed would appear by chance less than 5% of the time. It says nothing about the size of the effect, or about the probability that your variant is better.
Should I report a p-value or a confidence interval?
A confidence interval, with the p-value beside it. The interval shows the range of true effects your data is consistent with, which is what a shipping decision needs. A p-value compresses that into a pass or fail and routinely leads teams to ship effects that might be zero.
My test is not significant. Can I run it longer?
Only if you decided the duration in advance and have not reached it. Extending a test because the result is not yet significant, and stopping when it becomes so, is peeking under another name, and it produces false positives at several times the stated rate. Plan the sample size first.