What is statistical significance in A/B testing?

Statistical significance tells you how confident you can be that your test results aren't due to random chance. A 95% significance level means there's only a 5% chance the difference you're seeing is random noise. But here's what most people miss: 95% confidence doesn't mean 95% chance your variant is better — it means if there was NO real difference, you'd see results this extreme only 5% of the time.

Why do most A/B tests give wrong conclusions?

Three main reasons: 1) Peeking — checking results daily and stopping when you see a 'winner' inflates false positives from 5% to 30%+, 2) Underpowered tests — not enough sample size means you miss real effects or see fake ones, 3) Multiple comparisons — testing many variants without correction means some will 'win' by chance. Our calculator warns you about all three.

How many visitors do I need for an A/B test?

It depends on your baseline conversion rate and minimum detectable effect (MDE). For example: 3% baseline CVR, 10% relative lift (to 3.3%), 80% power, 95% significance = ~25,000 visitors per variant. Want to detect a 5% lift? You need ~100,000 per variant. Our calculator shows exactly how long your test needs to run based on your traffic.

What's the difference between confidence and power?

Confidence (typically 95%) is about avoiding false positives — calling a winner when there isn't one. Power (typically 80%) is about avoiding false negatives — missing a real winner. Low power means you might run a test, see 'no significant difference,' and abandon a variant that was actually 15% better. Both matter, but most tools only show confidence.

Should I use Bayesian or Frequentist A/B testing?

Frequentist is stricter and better for high-stakes decisions where false positives are costly. Bayesian is more intuitive (gives you 'probability variant B is better') and allows continuous monitoring. For most marketing tests, either works if used correctly. The bigger issue is usually sample size and peeking, not methodology.

Can I check my A/B test results daily?

With traditional frequentist testing, no — each peek inflates your false positive rate. If you peek daily on a 2-week test, your 'real' significance is closer to 70-80%, not 95%. Solutions: 1) Sequential testing methods designed for peeking, 2) Set a fixed end date and don't look until then, 3) Use Bayesian methods which handle continuous monitoring better.

My test shows 94% significance — should I ship it?

Probably not yet. 94% means 6% chance you're shipping a worse experience — that's 1 in 17. More importantly, ask: What's the cost of being wrong vs. the cost of waiting? If shipping a loser costs you $50K/year and waiting 3 more days gets you to 97% confidence, wait. Our calculator shows the expected value math to make this decision clear.

A/B test calculator

Plain English verdicts. Know the real risk of shipping your "winner."

Try example:

Test Name (optional)

Control (Original)

Visitors *

Conversions *

Variant B (Challenger)

Visitors *

Conversions *

Your data stays privateInstant results

5 seconds • 10 free/day • Sign up for unlimited

The "3 Day Test" Problem

Most teams run tests for 3 days, see a lift, and ship. But with small samples, your "winner" has a 30-50% chance of being worse than control.