Benchmarks
| Benchmark | Coverage | Notes |
|---|---|---|
| MTEB | 56 tasks across 8 categories | De-facto text embedding leaderboard (HuggingFace) |
| BEIR | 18 retrieval tasks | Zero-shot retrieval focus |
| C-MTEB | Chinese counterpart to MTEB | |
| MIRACL | Multilingual IR (18 languages) | |
| BUCC | Bitext mining | Parallel sentence alignment |
| STS benchmarks | Semantic textual similarity | STS-B, SICK-R |
Retrieval metrics
| Metric | Meaning |
|---|---|
| Recall@k | Fraction of queries where correct doc is in top-k |
| Precision@k | Fraction of top-k that are relevant |
| MRR | Mean reciprocal rank — 1/rank of first correct |
| MAP | Mean average precision |
| NDCG@k | Normalized discounted cumulative gain — graded relevance |
| Hit@k | Binary: any relevant in top-k? |
Intrinsic vs extrinsic
- Intrinsic: STS tasks, analogy, word similarity — measure embedding geometry directly.
- Extrinsic: retrieval, classification, clustering — measure embedding usefulness for downstream tasks.
- Rule: always evaluate on tasks close to your actual use case, not just benchmark rank.
Practical tips
- Build a gold set from your domain — 100–500 query/document pairs.
- Use semi-hard negatives when fine-tuning — too-easy negatives don't teach much.
- Evaluate at the k your product shows: recall@10 vs recall@1 can pick different winners.
- Check latency and cost alongside quality — a 1% quality gain is rarely worth 10× cost.
Worked example: computing the metrics
Say you test five queries and the correct document appears at rank 1, 3, 2, beyond rank 10, and 1. Recall@1 is 2 / 5 = 0.40, because only two queries have the answer on top. Recall@3 is 4 / 5 = 0.80. MRR averages the reciprocal ranks: (1 + 1/3 + 1/2 + 0 + 1) / 5 = 2.833 / 5 = 0.567, counting the miss as zero. Precision@5 is different: if two of the top five results are relevant, it is 2 / 5 = 0.40.
Do not over-read small differences. With a gold set of 100 queries and a true recall near 0.80, the standard error is √(0.80 × 0.20 / 100) = 0.04, so two models 3 points apart may be statistically indistinguishable. Larger gold sets, or paired comparisons on the same queries, give firmer answers.
Common mistakes
- Mixing models. Queries and documents must be embedded by the same model (or a pair designed to work together), with the same normalization.
- Ignoring the similarity function. Cosine similarity and dot product agree only when vectors are normalized to unit length.
- Skipping chunking tests. Chunk size and overlap change retrieval quality as much as the model does; evaluate them together.
- Leaderboard transfer. A model that tops a public benchmark can still lose on your domain vocabulary, such as part numbers or legal terms.
- Evaluating without a baseline. Compare against plain keyword search (BM25); hybrid retrieval often beats either alone.