Picking a test
| Question | Parametric | Non-parametric | Notes |
|---|---|---|---|
| Mean differs from known value | One-sample t-test | Wilcoxon signed-rank | Single sample vs constant |
| Two means (independent samples) | Two-sample t-test | Mann-Whitney U | Unequal variance? Welch's t |
| Two means (paired samples) | Paired t-test | Wilcoxon signed-rank | Before/after or matched |
| More than two means | One-way ANOVA | Kruskal-Wallis | Followed by post-hoc |
| Two categorical variables independent? | Chi-square | Fisher's exact | Use Fisher for small counts |
| Correlation | Pearson r | Spearman ρ | Spearman for non-linear / rank |
| Proportions | Z-test / binomial | — | For large n use Z |
| Regression significance | t-test on coefficient | — | Via OLS regression |
Assumptions
| t-test | Approximately normal (robust at n > 30), roughly equal variances (Welch relaxes this) |
|---|---|
| ANOVA | Normal residuals, equal variances, independent observations |
| Chi-square | Expected counts ≥ 5 per cell (else Fisher's exact) |
| Non-parametric | Less power; use when normality violated and n small |
P-value interpretation
- p-value is NOT the probability the null is true.
- It is P(observed data | null hypothesis).
- α = 0.05 is a convention — not magical. Effect size and confidence intervals often matter more.
- Multiple testing: use Bonferroni or FDR correction when running many tests.
- Pre-register hypotheses to avoid p-hacking.
Worked examples
Welch t-test. Group A: n = 10, mean 52, standard deviation 6. Group B: n = 10, mean 47, standard deviation 5. The standard error is √(6²/10 + 5²/10) = √(3.6 + 2.5) = √6.1 = 2.47, so t = (52 − 47) / 2.47 = 2.02. The Welch degrees of freedom come out near 17.4, and the two-sided p-value is about 0.058. A 5-point difference looks large but, with this spread and sample size, is just short of the conventional 0.05 line, which is a reminder to report the effect and confidence interval and not only the verdict.
Chi-square. A 2×2 table with counts 30, 20 / 20, 30 (n = 100) has expected counts of 25 in every cell, so χ² = 4 × (5² / 25) = 4.0 with 1 degree of freedom. The critical value at 0.05 is 3.84, so p is about 0.046 and the association is just significant.
Multiple tests. Run 20 independent tests at α = 0.05 on pure noise and the chance of at least one false positive is 1 − 0.9520 = 1 − 0.358 = 64 percent. A Bonferroni correction tests each at 0.05 / 20 = 0.0025 to keep the overall rate near 5 percent.
Common mistakes
- Using an independent test on paired data. Before and after measurements on the same subjects need the paired test, which is usually far more sensitive.
- Several t-tests instead of ANOVA. Comparing three groups pairwise multiplies the false positive rate; run ANOVA first, then post-hoc comparisons with correction.
- Equating “not significant” with “no effect”. A small sample may simply lack power to detect a real difference.
- Choosing the test after seeing the data. Decide the test and the one- or two-sided hypothesis before collecting data.
- Forgetting independence. Repeated measurements from the same person or batch break the assumptions of almost every test above.