Home Free tools A/B testing calculator

A/B testing calculator for statistical significance

Find out if your A/B test has a clear winner. Enter the visitors (or email recipients) and conversions for version A and version B, and a two-proportion z-test will show whether the difference is statistically significant.

Visitors

Conversions

Conversion rate

Visitors

Conversions

Conversion rate

3.00%

Visitors

Conversions

Conversion rate

3.80%

Hypothesis

One-sided

Two-sided

Confidence

90%

95%

99%

Calculate

Result

Significant result: B wins

Version B converted 26.7% better than A. You can be 95% confident this difference isn’t due to chance.

Relative uplift

+26.7%

P value

0.0273

95% confidence interval of the difference

+0.09 to +1.51 pp

Calculate

What is statistical significance in A/B testing?

Statistical significance in A/B testing tells you whether the difference between version A and version B is bigger than you’d expect from random chance alone. A significant result means the gap is unlikely to be a fluke, so you can roll out the winner with more confidence. The smaller the p-value, the stronger the evidence that one version performs better.

How to use the A/B testing calculator

1
Enter your results

Add the number of visitors (or email recipients) and conversions for each version. You’ll find these in your email or website analytics.
2
Choose your settings

Pick your hypothesis (one-sided or two-sided test) and a confidence level: 90%, 95%, or 99%. Most tests use a two-sided test at 95%.
3
Read the result

You’ll see if the difference is statistically significant, plus the relative uplift, p-value, and confidence interval. If not, keep the test running or try a bigger audience next time.

How to read your results

B wins

B beat A, and the p-value is below your threshold.Next step: roll out version B, then move on to your next test.

A wins

B did worse than A, and the p-value is below your threshold.Next step:keep version A and note what didn’t work, so you don’t test it again.

Not significant yet

There’s a difference, but the p-value is above your threshold and you haven’t reached your planned sample size.Next step: keep the test running until you do.

Example: calculating significance for an email subject line test

Say you test two subject lines and send each one to 5,000 subscribers. Since Apple Mail Privacy Protection inflates open rates, you measure clicks instead. Version A gets 150 clicks (a 3.0% click rate), and version B gets 190 (3.8%). That’s a 26.7% relative uplift with a p-value of 0.0273, which is significant at 95% confidence but not at 99%. You can send version B to the rest of your list with reasonable confidence.

Version	Recipients	Clicks	Click rate
A	5,000	150	3.0%
B	5,000	190	3.8%
Result	+26.7% uplift · p = 0.0273 · significant at 95%, not at 99%

With a smaller list (fewer than 5,000 subscribers per version), test bigger changes, like the offer, format, or send time, instead of small wording tweaks. Bigger changes are easier to detect with fewer subscribers per version.

How is A/B test significance calculated?

The calculator compares the two conversion rates with a two-proportion z-test.

Conversion rate: pA = conversionsA ÷ visitorsA, pB = conversionsB ÷ visitorsB

Pooled rate: p = (conversionsA + conversionsB) ÷ (visitorsA + visitorsB)

Test statistic: z = (pB − pA) ÷ √[ p(1 − p) × (1/nA + 1/nB) ]

The p-value is the chance of seeing a difference at least this large if both versions performed the same. If it’s below 0.05 at 95% confidence (or 0.10 at 90%, 0.01 at 99%), the result is statistically significant.

Run your next A/B test in Omnisend

Test subject lines, email content, and forms, then check which version won without doing the math yourself.

Start free

Resources

Keep learning

Email A/B testing guide

Blog

Email A/B testing guide

Email marketing benchmarks

Blog

Email marketing benchmarks

Email ROI calculator

Free Tools

Email ROI calculator

FAQ

How do you calculate statistical significance for an A/B test?

Divide conversions by visitors for each version, then run a two-proportion z-test on the two rates. The test returns a p-value, which is the chance of seeing a difference at least this large if both versions performed the same. If the p-value is below 0.05, the result is significant at 95% confidence.

What sample size do I need for an A/B test?

It depends on your baseline conversion rate and the smallest uplift you want to detect. At a 3% click rate, spotting a 20% relative uplift at 95% confidence and 80% power (an 80% chance of catching the uplift if it’s real) takes about 13,900 recipients per version. Smaller uplifts need bigger samples.

How long should I run an A/B test?

For website tests, run it until each version reaches the sample size you planned, and cover full weeks so weekday and weekend behavior are both included. For email tests, the sample is fixed once you send, so wait at least 24 hours before you read the results. That way, late opens and clicks are counted.

What confidence level should I use: 90%, 95%, or 99%?

Most tests use 95%. Go with 90% for low-risk tests where a faster answer matters more than certainty, and 99% for high-stakes changes like pricing or checkout. The higher the confidence level, the bigger the sample you need to reach significance.

What’s the difference between a one-sided and two-sided test?

A one-sided test only checks whether B is better than A. A two-sided test checks for a difference in either direction, so it also tells you when B performs worse. Two-sided is more conservative, while one-sided reaches significance with less data.

Why is my A/B test not statistically significant?

Usually, the sample is too small for the size of the difference, or the two versions really do perform about the same. If your list or traffic is small, test a bigger change, such as a different offer or email format, so the difference is easier to detect.

Can I stop my test as soon as it reaches significance?

No. Results swing a lot early in a test. Checking results again and again (known as peeking) and stopping at the first significant reading makes a false winner much more likely. Set your sample size before you start, and read the result once you reach it.

What should I A/B test in email marketing?

Start with the changes most likely to move results: subject lines, the offer or discount, the main call to action, send time, and email layout. Test one change at a time so you know what caused the difference, and check the sample size you need first.

Start free
today

Built on feedback from 150,000+ brands. Try Omnisend and see why they stay.