Numbers & Math

Statistical Tests Reference

Which hypothesis test to use — t-test, chi-square, ANOVA, non-parametric alternatives.

Picking a test

QuestionParametricNon-parametricNotes
Mean differs from known valueOne-sample t-testWilcoxon signed-rankSingle sample vs constant
Two means (independent samples)Two-sample t-testMann-Whitney UUnequal variance? Welch's t
Two means (paired samples)Paired t-testWilcoxon signed-rankBefore/after or matched
More than two meansOne-way ANOVAKruskal-WallisFollowed by post-hoc
Two categorical variables independent?Chi-squareFisher's exactUse Fisher for small counts
CorrelationPearson rSpearman ρSpearman for non-linear / rank
ProportionsZ-test / binomial—For large n use Z
Regression significancet-test on coefficient—Via OLS regression

Assumptions

t-testApproximately normal (robust at n > 30), roughly equal variances (Welch relaxes this)
ANOVANormal residuals, equal variances, independent observations
Chi-squareExpected counts ≥ 5 per cell (else Fisher's exact)
Non-parametricLess power; use when normality violated and n small

P-value interpretation

  • p-value is NOT the probability the null is true.
  • It is P(observed data | null hypothesis).
  • α = 0.05 is a convention — not magical. Effect size and confidence intervals often matter more.
  • Multiple testing: use Bonferroni or FDR correction when running many tests.
  • Pre-register hypotheses to avoid p-hacking.

Worked examples

Welch t-test. Group A: n = 10, mean 52, standard deviation 6. Group B: n = 10, mean 47, standard deviation 5. The standard error is √(6²/10 + 5²/10) = √(3.6 + 2.5) = √6.1 = 2.47, so t = (52 − 47) / 2.47 = 2.02. The Welch degrees of freedom come out near 17.4, and the two-sided p-value is about 0.058. A 5-point difference looks large but, with this spread and sample size, is just short of the conventional 0.05 line, which is a reminder to report the effect and confidence interval and not only the verdict.

Chi-square. A 2×2 table with counts 30, 20 / 20, 30 (n = 100) has expected counts of 25 in every cell, so χ² = 4 × (5² / 25) = 4.0 with 1 degree of freedom. The critical value at 0.05 is 3.84, so p is about 0.046 and the association is just significant.

Multiple tests. Run 20 independent tests at α = 0.05 on pure noise and the chance of at least one false positive is 1 − 0.9520 = 1 − 0.358 = 64 percent. A Bonferroni correction tests each at 0.05 / 20 = 0.0025 to keep the overall rate near 5 percent.

Common mistakes

  • Using an independent test on paired data. Before and after measurements on the same subjects need the paired test, which is usually far more sensitive.
  • Several t-tests instead of ANOVA. Comparing three groups pairwise multiplies the false positive rate; run ANOVA first, then post-hoc comparisons with correction.
  • Equating “not significant” with “no effect”. A small sample may simply lack power to detect a real difference.
  • Choosing the test after seeing the data. Decide the test and the one- or two-sided hypothesis before collecting data.
  • Forgetting independence. Repeated measurements from the same person or batch break the assumptions of almost every test above.
Was this article helpful?