Free A/B Test Significance Calculator

Enter the visitors and conversions for your control and variant above. The calculator runs a two-proportion z-test and returns the confidence level and p-value. A result is usually considered significant at 95% confidence (p-value below 0.05), meaning there is less than a 5% chance the difference is down to random noise.

Use the Sample size tab. Enter your baseline conversion rate, the smallest uplift you want to detect (minimum detectable effect), your confidence level and power. The calculator returns the required visitors per variant and, from your weekly traffic, how long the test needs to run.

Run it until each variant reaches the required sample size, and ideally for at least one to two full business cycles (typically two weeks) to even out day-of-week effects. Stopping early, as soon as a result looks significant, is the most common way to get a false positive.

95% is the standard for ecommerce A/B testing - it balances catching real wins against false positives. Use 99% when a change is risky or expensive to ship, and 90% only for low-stakes, fast iterations where some extra false positives are acceptable.

A two-tailed test checks whether the variant is different from control in either direction (better or worse) and is the safer default. A one-tailed test only checks whether the variant is better, so it reaches significance faster but cannot tell you if the variant is actually hurting performance.

Power is the chance your test detects a real effect when one exists (80% is the usual target). The Simulation tab runs thousands of Monte Carlo trials of your planned test so you can see how often it would actually reach significance - a quick way to spot an underpowered test before you waste traffic on it.