Q: How big a sample do I need?

It depends on your baseline conversion rate and the lift you want to detect. To reliably detect a 10% relative lift on a 5% baseline at 95% confidence and 80% power, you typically need 30,000+ visitors per arm. The 'sample needed' output here uses your actual observed rates so you know exactly how much more traffic to drive before calling the test.

Q: When can I stop a test early?

Generally, don't. Peeking and stopping inflates false-positive risk. Pre-commit to a sample size, run until you hit it, then evaluate. The exception is a sequential or Bayesian test design — but those have their own setup. If your test is at 99%+ confidence with a sample well above the calculated minimum, calling it early is defensible.

Q: What's the difference between p-value and confidence?

Confidence is just (1 − p-value), expressed as a percent. A p-value of 0.04 means 96% confidence. p-value answers 'how likely is this lift if the variants were actually identical?' Lower p means less likely, which means the observed lift is more credible.

Q: Why do my A/B tests keep being inconclusive?

Three common reasons: (1) variants are too similar — small lifts need huge samples; (2) the change isn't where the friction actually is; (3) traffic is too low to detect anything in a reasonable timeframe. Test bigger swings — different headlines, different hooks, different visuals. ppl.studio generates radically different AI UGC variants in seconds so you can test bold ideas without a creator team.

Question 1

What significance level should I use?

Accepted Answer

The industry default is p < 0.05 (95% confidence) for a two-sided test. For high-stakes decisions like checkout flow or pricing changes, tighten to p < 0.01. For low-stakes creative tests, some teams accept p < 0.10 — but expect more false positives. This calculator runs a two-sided z-test for proportions.

Question 2

How big a sample do I need?

Accepted Answer

It depends on your baseline conversion rate and the lift you want to detect. To reliably detect a 10% relative lift on a 5% baseline at 95% confidence and 80% power, you typically need 30,000+ visitors per arm. The 'sample needed' output here uses your actual observed rates so you know exactly how much more traffic to drive before calling the test.

Question 3

When can I stop a test early?

Accepted Answer

Generally, don't. Peeking and stopping inflates false-positive risk. Pre-commit to a sample size, run until you hit it, then evaluate. The exception is a sequential or Bayesian test design — but those have their own setup. If your test is at 99%+ confidence with a sample well above the calculated minimum, calling it early is defensible.

Question 4

What's the difference between p-value and confidence?

Accepted Answer

Confidence is just (1 − p-value), expressed as a percent. A p-value of 0.04 means 96% confidence. p-value answers 'how likely is this lift if the variants were actually identical?' Lower p means less likely, which means the observed lift is more credible.

Question 5

Why do my A/B tests keep being inconclusive?

Accepted Answer

Three common reasons: (1) variants are too similar — small lifts need huge samples; (2) the change isn't where the friction actually is; (3) traffic is too low to detect anything in a reasonable timeframe. Test bigger swings — different headlines, different hooks, different visuals. ppl.studio generates radically different AI UGC variants in seconds so you can test bold ideas without a creator team.

A/B Test Significance Calculator

Variant A (control)

Variant B (test)

Enter your test data

Frequently asked questions