Statistical Tests Reference
Which hypothesis test to use — t-test, chi-square, ANOVA, non-parametric alternatives.
Reference
Picking a test
| Question | Parametric | Non-parametric | Notes |
|---|---|---|---|
| Mean differs from known value | One-sample t-test | Wilcoxon signed-rank | Single sample vs constant |
| Two means (independent samples) | Two-sample t-test | Mann-Whitney U | Unequal variance? Welch's t |
| Two means (paired samples) | Paired t-test | Wilcoxon signed-rank | Before/after or matched |
| More than two means | One-way ANOVA | Kruskal-Wallis | Followed by post-hoc |
| Two categorical variables independent? | Chi-square | Fisher's exact | Use Fisher for small counts |
| Correlation | Pearson r | Spearman ρ | Spearman for non-linear / rank |
| Proportions | Z-test / binomial | — | For large n use Z |
| Regression significance | t-test on coefficient | — | Via OLS regression |
Assumptions
- t-test
- Approximately normal (robust at n > 30), roughly equal variances (Welch relaxes this)
- ANOVA
- Normal residuals, equal variances, independent observations
- Chi-square
- Expected counts ≥ 5 per cell (else Fisher's exact)
- Non-parametric
- Less power; use when normality violated and n small
P-value interpretation
- p-value is NOT the probability the null is true.
- It is P(observed data | null hypothesis).
- α = 0.05 is a convention — not magical. Effect size and confidence intervals often matter more.
- Multiple testing: use Bonferroni or FDR correction when running many tests.
- Pre-register hypotheses to avoid p-hacking.
Last updated: