RAG: From First Principles · Appendix

Appendix B · Every Metric on One Sheet

Notation: for one question, R = set of relevant chunk ids (from the golden set's sources, mapped to chunk ids), L = [l₁, l₂, …] = retrieved chunk ids in rank order, L@k = first k of them, rel(i) = 1 if lᵢ ∈ R else 0. All…

For each metric: what it measures, the formula, how to compute it (function names refer to code/ch10/metrics.py), what "good" looks like on a corpus like the handbook, and what to change when it is low. Chapter 10 has the code and worked examples; this is the reference you keep open.

Notation: for one question, R = set of relevant chunk ids (from the golden set's sources, mapped to chunk ids), L = [l₁, l₂, …] = retrieved chunk ids in rank order, L@k = first k of them, rel(i) = 1 if lᵢ ∈ R else 0. All metrics are computed per question and averaged over the golden set; report the mean and, ideally, the per-tag breakdown (numeric / multi_hop / unanswerable …).

B.1 Retrieval metrics (no LLM needed: run these first, they are free)#

Reading them together

• Hit rate high, recall low → finding some evidence but not all; multi-source questions suffer. Multi-query / decomposition.
• Recall high, precision low → context is noisy; the generator pays in tokens and distraction. Rerank, lower k.
• Recall high, MRR low → right chunks, wrong order. Rerank.
• Everything low for numeric/code tags only → embeddings blur numbers; add BM25.
• Everything low for one document only → chunking or parsing problem in that document.

B.2 Context quality (LLM-judged, per retrieved chunk)#

B.3 Generation metrics (LLM-judged, on the answer)#

B.4 End-to-end and business metrics#

B.5 Operational metrics#

B.6 Evaluation-process metrics (is your judge trustworthy?)#

B.7 The formulas you should be able to write on a whiteboard#

precision@k = |retrieved@k ∩ relevant| / k
recall@k    = |retrieved@k ∩ relevant| / |relevant|
MRR         = mean over questions of 1 / rank(first relevant)      (0 if none)
AP          = (1/|relevant|) · Σ_{i : rel(i)=1} precision@i ;  MAP = mean(AP)
DCG@k       = Σ_{i=1..k} rel(i) / log2(i + 1)
nDCG@k      = DCG@k / IDCG@k
faithfulness = supported claims / total claims           (claims from the answer, judged vs context)
context recall = reference statements supported by context / total reference statements
RRF(d)      = Σ_lists 1 / (60 + rank_list(d))
BM25(q,d)   = Σ_t IDF(t) · tf·(k1+1) / (tf + k1·(1 − b + b·|d|/avgdl)),  k1≈1.5, b≈0.75
cosine(a,b) = a·b / (|a||b|)
kappa       = (p_o − p_e) / (1 − p_e)

B.8 A minimal eval report template#

Config: collection=handbook_v3  chunk=800/120  embed=text-embedding-3-small  k=5  rerank=none  model=gpt-5.4-mini
Golden: 47 questions (42 answerable, 5 unanswerable)   Judge: gpt-5.4-mini, rubric v2

Retrieval        hit@5 0.93   recall@5 0.86   precision@5 0.41   MRR 0.78   nDCG@5 0.81
Generation       faithfulness 0.94   relevance 0.88   correctness 0.83   keyword-hit 0.86
Refusal          precision 0.83   recall 1.00
Cost/latency     1,420 in / 96 out tokens   $0.0015/q   p50 1.9 s   p95 3.4 s
Worst tags       multi_hop: recall 0.62   numeric: correctness 0.75
Judge check      agreement 0.90 (n=40)   kappa 0.78

Write one of these per change. The diff between two reports is the result of your experiment; anything not in the report did not happen.

B.9 Before you trust any number above#

Every metric in this appendix is an estimate from a sample, computed against labels you invented, and half of them are produced by a stochastic judge. Four checks separate a result from a story: all are derived and measured in Chapter 21 and Chapter 26.

The practical rule: no single metric moving alone is a result. Trust changes where several related metrics move together, where the CI excludes zero, and where you can name the mechanism.

And the sentence to have ready in an interview: recall@k is stage one of four: retrieved, then survived context assembly, then used by the model, then correct. Quoting retrieval recall as an accuracy figure claims the other three stages are lossless.