PollPe Logo

Free research tool

Significance Calculator

Check whether the gap between two groups is real or sampling noise. Compare two percentages (crosstabs, brand awareness by city, A/B conversion) or two averages (ratings, spend), and get the p-value, confidence interval and the sample you’d need.

Free, no sign-up. Last reviewed against source documents on .

Compare
Confidence level

Use for anything counted as yes/no: % who prefer a pack, top-2-box, conversion in an A/B test, % aware of a brand in two cities.

Group A

22%

Group B

18%

ResultSignificantDetails ↓

How big a gap is significant?

The smallest difference between two equal-size groups that is significant, for results near 50% (the hardest case).

Minimum significant difference by group size
People per group90% confidence95% confidence99% confidence
10011.6 pts13.9 pts18.2 pts
2008.2 pts9.8 pts12.9 pts
3006.7 pts8.0 pts10.5 pts
5005.2 pts6.2 pts8.1 pts
7504.2 pts5.1 pts6.7 pts
1,0003.7 pts4.4 pts5.8 pts
2,0002.6 pts3.1 pts4.1 pts
5,0001.6 pts2.0 pts2.6 pts

For results near 10% or 90% the required gap is smaller (about 60% of these values).

How the tests work

Two percentages:

z = (p₁ − p₂) ÷ √( p̄ (1 − p̄) (1/n₁ + 1/n₂) ), p̄ = (x₁ + x₂) ÷ (n₁ + n₂)

Two averages (Welch):

t = (x̄₁ − x̄₂) ÷ √( s₁²/n₁ + s₂²/n₂ )

Worked example: 22% of 1,000 people in Mumbai know a brand vs 18% of 1,000 in Delhi. Pooled p̄ = 0.20, z = 0.04 ÷ 0.0179 = 2.24, p = 0.025: significant at 95%, not at 99%.

Planning the study? The sample size calculator sizes each group. Comparing loyalty scores? The NPS calculator has its own test.

Questions, answered

What does statistically significant mean in a survey?

It means the difference between two groups is larger than you would expect from sampling noise alone. At 95% confidence, a significant result has less than a 5% chance of appearing if the two groups were truly the same (p < 0.05). It does not mean the difference is large or important.

Which test does this calculator use?

For percentages it uses a two-proportion z-test with a pooled standard error, and an unpooled interval for the gap. For averages it uses Welch’s t-test, which does not assume the two groups have equal variance, so it is safe for unequal group sizes.

How big a difference is significant with 500 people per group?

Around 6.2 pts at 95% confidence when results are near 50%. With 200 per group you need about 9.8 pts; with 1,000 per group about 4.4 pts.

Can I use this for an A/B test?

Yes. Enter visitors as the sample size and conversions as the count for each variant. Decide the sample size before the test and avoid stopping the moment it turns significant, since repeated peeking inflates false positives.

Can I compare a subgroup with the total?

Not directly with this test, because the subgroup is part of the total and the samples are not independent. Compare the subgroup with everyone else instead (for example, women vs men, not women vs all).

What is the p-value?

The probability of seeing a difference at least this large if there were truly no difference. A p-value of 0.03 is significant at 95% confidence (below 0.05) but not at 99% (above 0.01).

Found a real difference? Dig into why.

Run a follow-up survey in PollPe with open-ended questions and reach the segment that moved.

Statistical Significance Calculator for Surveys | PollPe