Web & Dev

AI Embedding Evaluation

How to measure embedding quality — benchmarks, retrieval metrics, and intrinsic evaluation.

Benchmarks

BenchmarkCoverageNotes
MTEB56 tasks across 8 categoriesDe-facto text embedding leaderboard (HuggingFace)
BEIR18 retrieval tasksZero-shot retrieval focus
C-MTEBChinese counterpart to MTEB
MIRACLMultilingual IR (18 languages)
BUCCBitext miningParallel sentence alignment
STS benchmarksSemantic textual similaritySTS-B, SICK-R

Retrieval metrics

MetricMeaning
Recall@kFraction of queries where correct doc is in top-k
Precision@kFraction of top-k that are relevant
MRRMean reciprocal rank — 1/rank of first correct
MAPMean average precision
NDCG@kNormalized discounted cumulative gain — graded relevance
Hit@kBinary: any relevant in top-k?

Intrinsic vs extrinsic

  • Intrinsic: STS tasks, analogy, word similarity — measure embedding geometry directly.
  • Extrinsic: retrieval, classification, clustering — measure embedding usefulness for downstream tasks.
  • Rule: always evaluate on tasks close to your actual use case, not just benchmark rank.

Practical tips

  • Build a gold set from your domain — 100–500 query/document pairs.
  • Use semi-hard negatives when fine-tuning — too-easy negatives don't teach much.
  • Evaluate at the k your product shows: recall@10 vs recall@1 can pick different winners.
  • Check latency and cost alongside quality — a 1% quality gain is rarely worth 10× cost.

Worked example: computing the metrics

Say you test five queries and the correct document appears at rank 1, 3, 2, beyond rank 10, and 1. Recall@1 is 2 / 5 = 0.40, because only two queries have the answer on top. Recall@3 is 4 / 5 = 0.80. MRR averages the reciprocal ranks: (1 + 1/3 + 1/2 + 0 + 1) / 5 = 2.833 / 5 = 0.567, counting the miss as zero. Precision@5 is different: if two of the top five results are relevant, it is 2 / 5 = 0.40.

Do not over-read small differences. With a gold set of 100 queries and a true recall near 0.80, the standard error is √(0.80 × 0.20 / 100) = 0.04, so two models 3 points apart may be statistically indistinguishable. Larger gold sets, or paired comparisons on the same queries, give firmer answers.

Common mistakes

  • Mixing models. Queries and documents must be embedded by the same model (or a pair designed to work together), with the same normalization.
  • Ignoring the similarity function. Cosine similarity and dot product agree only when vectors are normalized to unit length.
  • Skipping chunking tests. Chunk size and overlap change retrieval quality as much as the model does; evaluate them together.
  • Leaderboard transfer. A model that tops a public benchmark can still lose on your domain vocabulary, such as part numbers or legal terms.
  • Evaluating without a baseline. Compare against plain keyword search (BM25); hybrid retrieval often beats either alone.
Was this article helpful?