Picking a test
| Question | Parametric | Non-parametric | Notes |
|---|---|---|---|
| Mean differs from known value | One-sample t-test | Wilcoxon signed-rank | Single sample vs constant |
| Two means (independent samples) | Two-sample t-test | Mann-Whitney U | Unequal variance? Welch's t |
| Two means (paired samples) | Paired t-test | Wilcoxon signed-rank | Before/after or matched |
| More than two means | One-way ANOVA | Kruskal-Wallis | Followed by post-hoc |
| Two categorical variables independent? | Chi-square | Fisher's exact | Use Fisher for small counts |
| Correlation | Pearson r | Spearman ρ | Spearman for non-linear / rank |
| Proportions | Z-test / binomial | — | For large n use Z |
| Regression significance | t-test on coefficient | — | Via OLS regression |
Assumptions
| t-test | Approximately normal (robust at n > 30), roughly equal variances (Welch relaxes this) |
|---|---|
| ANOVA | Normal residuals, equal variances, independent observations |
| Chi-square | Expected counts ≥ 5 per cell (else Fisher's exact) |
| Non-parametric | Less power; use when normality violated and n small |
P-value interpretation
- p-value is NOT the probability the null is true.
- It is P(observed data | null hypothesis).
- α = 0.05 is a convention — not magical. Effect size and confidence intervals often matter more.
- Multiple testing: use Bonferroni or FDR correction when running many tests.
- Pre-register hypotheses to avoid p-hacking.
Was this article helpful?