A/B Test Significance Calculator
Get statistical significance, confidence, and the sample size you still need. Two-sided z-test for proportions.
Variant A (control)
Variant B (test)
Enter your test data
Drop in visitors and conversions for both variants. We'll compute confidence, p-value, and sample size needed.
Frequently asked questions
The industry default is p < 0.05 (95% confidence) for a two-sided test. For high-stakes decisions like checkout flow or pricing changes, tighten to p < 0.01. For low-stakes creative tests, some teams accept p < 0.10 — but expect more false positives. This calculator runs a two-sided z-test for proportions.
It depends on your baseline conversion rate and the lift you want to detect. To reliably detect a 10% relative lift on a 5% baseline at 95% confidence and 80% power, you typically need 30,000+ visitors per arm. The 'sample needed' output here uses your actual observed rates so you know exactly how much more traffic to drive before calling the test.
Generally, don't. Peeking and stopping inflates false-positive risk. Pre-commit to a sample size, run until you hit it, then evaluate. The exception is a sequential or Bayesian test design — but those have their own setup. If your test is at 99%+ confidence with a sample well above the calculated minimum, calling it early is defensible.
Confidence is just (1 − p-value), expressed as a percent. A p-value of 0.04 means 96% confidence. p-value answers 'how likely is this lift if the variants were actually identical?' Lower p means less likely, which means the observed lift is more credible.
Three common reasons: (1) variants are too similar — small lifts need huge samples; (2) the change isn't where the friction actually is; (3) traffic is too low to detect anything in a reasonable timeframe. Test bigger swings — different headlines, different hooks, different visuals. ppl.studio generates radically different AI UGC variants in seconds so you can test bold ideas without a creator team.