A/B Test Significance Calculator
Enter the visitors and conversions for two versions of a page, email or ad and find out whether the difference is real: conversion rates, uplift, p-value, confidence interval and the chance that B beats A.
How to Use
- Enter the visitors (or sessions, emails sent, impressions) and conversions for A, the original, and for B, the change.
- Pick the confidence level you decided on before the test (95% is the usual choice) and whether you test both directions or only “B is better”.
- Read the verdict: the uplift, the p-value, whether it is significant, and the chance B beats A.
- Check the interval for the difference: if it includes 0, the test has not shown which version is better.
- If the traffic split warning appears, fix the test setup before trusting any result.
Worked Example
A new headline. A (old) gets 500 sign-ups from 10,000 visitors, 5.0%; B (new) gets 570 from 10,000, 5.7%. The difference is 0.7 percentage points, a 14% relative uplift. Under “no difference” both share the pooled rate 1,070 ÷ 20,000 = 5.35%, so the standard error of the difference is √(0.0535 × 0.9465 × (1/10,000 + 1/10,000)) = 0.003182. Then z = 0.007 ÷ 0.003182 = 2.1996 and the two-sided p-value is 2 × (1 − Φ(2.1996)) = 0.0278, below 0.05: significant at 95%. The 95% interval for the difference, from Newcombe’s method (it combines each rate’s Wilson interval, 4.59–5.45% for A and 5.26–6.17% for B), runs from +0.08 to +1.32 points: B is better, but the true lift may be small.
The common mistake: a big lift from a small test. 8 of 200 (4%) against 12 of 200 (6%) looks like a 50% improvement. But the pooled rate is 5%, the standard error √(0.05 × 0.95 × 2/200) = 0.0218, z = 0.02 ÷ 0.0218 = 0.918 and p = 0.359. Four extra conversions are well within chance: the test says nothing yet, and the interval for the difference runs from about −2.5 to +6.6 points.
Show Work
Formulas
Why A/B Tests Go Wrong
The 5% significance level comes from Ronald Fisher, who in Statistical Methods for Research Workers (1925) called a result that would turn up by chance 1 time in 20 “convenient” to treat as significant. Comparing two proportions this way is classic statistics; what changed with the web is scale and speed. A site can split thousands of visitors a day between two versions and see results within hours, which makes it tempting to look too often.
That is the most common mistake. Each look is another chance for random noise to cross the line, so checking every day and stopping at the first p < 0.05 produces far more false winners than 5%; Armitage, McPherson and Rowe showed this for repeated significance tests on accumulating data in 1969. The fix is to decide the sample size in advance (the Sample Size Calculator does this), run full weeks, and read the result once. The second mistake is too little data: a small test can only detect a large difference, so a “not significant” result usually means “not enough visitors”, not “no effect”.
The third is a broken experiment. If a 50/50 test ends with noticeably more visitors in one arm than chance allows, something is wrong with how visitors are assigned or counted: redirects that lose users, bots, or a tracking tag that fires on one version only. Microsoft’s experimentation team calls this a sample ratio mismatch and treats it as a reason to throw the result away; this calculator checks for it automatically.
About This Calculator
This calculator runs the standard two-proportion z-test on the counts from an A/B test and adds what you need to act on it: each conversion rate with a Wilson interval, the interval for the difference, the relative uplift, the Bayesian chance that B is better, a sample-ratio check of the traffic split, and the number of visitors you would need to confirm a difference of the size you saw.
Every step of the working is shown, and all calculations run in your browser; no test data is sent anywhere. It suits marketers, product managers and developers checking landing pages, emails, ads and app features, and students learning hypothesis tests on proportions.
Related calculators: Sample Size Calculator, P-Value Calculator, and Confidence Interval Calculator.
Frequently Asked Questions
What does statistically significant mean in an A/B test?
That a difference as large as the one you measured would be unlikely if A and B really converted equally. At 95% confidence the p-value must be below 0.05: 500 of 10,000 against 570 of 10,000 gives p = 0.028, so the 0.7-point difference is significant. It does not mean the uplift is exactly 14%; the interval shows the plausible range.
How many visitors does an A/B test need?
It depends on the base rate and the smallest lift worth detecting, and it should be fixed before the test starts. Going from 5% to 5.5% with 80% power at 95% confidence needs about 31,000 visitors per variant; spotting 5% to 6% needs about 8,200. The Sample Size Calculator works this out.
Can I stop the test as soon as it becomes significant?
No. Checking again and again and stopping at the first p < 0.05 raises the real false-positive rate far above 5%, because random swings cross the line along the way. Decide the sample size first, run whole weeks so weekdays and weekends are both in, and read the result once at the end.
Should I use a one-sided or two-sided test?
Two-sided, unless you decided in advance that you only care whether B is better and will treat “B is worse” the same as “no difference”. A one-sided test halves the p-value for a lift in the expected direction, which is why switching to it after seeing the data is a mistake.
What is the “chance B beats A”?
A Bayesian answer to the question most people actually ask. Starting from no preference, it is the probability that B’s true conversion rate is higher than A’s, given the data. For 500/10,000 against 570/10,000 it is about 98.6%. It is not the same as 1 − p, but the two usually point the same way.
How do I use the A/B Test Significance Calculator?
Simply type your numbers and read the result, which refreshes the instant you change something. There is nothing to submit and nothing to wait for.
Is it free? Does it work without internet?
Yes to both. It is free with no sign-up, and once the page has loaded it keeps working even with no internet.
Where does my data go?
Nowhere — every calculation runs on your own device. Nothing you enter is uploaded, logged, or stored.
Common Use Cases
Landing page tests
A new headline gets 570 sign-ups from 10,000 visitors against 500 for the old one: +14% uplift, p = 0.028, significant at 95%.
Email subject lines
Two subject lines sent to 5,000 people each: 300 opens against 240 is p = 0.008, but if the test was the click rate, enter clicks, not opens.
Small tests
8 of 200 against 12 of 200 looks like a 50% lift, but p = 0.36: the test is far too small to tell.
Catching a broken test
A 50/50 test with 10,000 visitors in A and 9,500 in B fails the traffic split check (p = 0.0003): something is dropping B’s visitors.
Ads and app features
Click-through rates, trial starts or feature adoption: any yes/no outcome counted over a number of users works the same way.
Last updated: