A RAG Evaluation Framework: Golden Datasets, Retrieval Recall and Citation Accuracy

A RAG assistant that answers five rehearsed questions well in a demo tells you almost nothing about how it handles the thousand questions real users will ask, or how it behaves after the next reindex. Yet that demo is the entire quality gate on most projects. The result is a system that degrades silently — a prompt tweak helps one question and breaks three, and nobody notices until a user does.

Treat a RAG system like the production software it is. That means a repeatable test harness: a fixed set of questions with known-correct answers, metrics that score retrieval and generation separately, and an automated run on every change. The effort is modest and it converts 'we think it is better' into 'recall went from 0.72 to 0.85 and citation accuracy held'.

A golden dataset is a curated set of representative questions, each paired with the source passages that contain the answer and a reference answer. It should come from real user questions — pulled from logs, support queues, the actual SOPs people look up — not invented ones, because invented questions flatter the system by matching how the documents are written.

Size it for coverage, not volume: 50 to 150 well-chosen questions spanning the easy, the ambiguous, the multi-document and the deliberately out-of-scope will tell you more than a thousand near-duplicates. Include questions whose correct answer is 'that is not in the available documents', because a system that invents an answer rather than declining is the most dangerous failure mode and the one demos never test.

Retrieval is measured independently of the language model, because if the right passages never came back, no amount of prompt engineering produces a correct grounded answer. The core metrics are recall at K (of the known-relevant passages, how many appeared in the top K retrieved) and precision at K (of the top K retrieved, how many were actually relevant). Recall is usually the one that matters most for grounding — a missing passage is an unanswerable question.

Because you know the correct source passages for each golden question, these are computable automatically and cheaply on every change. They localise the problem precisely: if recall is low, the fix is chunking, hybrid search or filters; if recall is high but answers are still wrong, the problem has moved to generation. That separation is the whole point — it stops teams tuning the prompt to fix a retrieval fault.

Once the right passages are retrieved, the generation layer is judged on two things that matter more than fluency. Groundedness (or faithfulness) asks whether every claim in the answer is supported by the retrieved passages, or whether the model added something plausible but unsourced. Citation accuracy asks whether the citations the answer gives actually point to the passages that support each claim — a fluent answer with confident but wrong citations is a trust disaster, because users verify by the citation.

These can be scored with a stronger model acting as judge against the reference answer and sources, which scales but carries its own error and cost, or by human review on a sample, which is accurate but slower. A sensible programme uses automated groundedness and citation scoring on every change for trend, with periodic human review to keep the judge honest. Answer relevance and completeness round out the set, but groundedness and citation accuracy are the two you cannot ship without.

The framework earns its keep when it runs automatically. Every change that can affect output — a new embedding model, a reindex, a chunking change, a prompt edit, a model version bump — triggers the harness, and the metrics are compared against the previous baseline. A change that improves one metric while quietly regressing another is caught before it reaches users, which manual testing never reliably does.

Wire it into the deployment pipeline so a material regression blocks release, the same way a failing unit test does. Store the results over time so you can see drift — model providers update models underneath you, and an answer quality that silently falls after a provider-side change is invisible without a standing baseline to compare against.

The golden dataset needs an owner and a refresh cycle. Documents change, new question types emerge, and a golden set that is never updated slowly stops representing reality — passing it becomes meaningless. Budget for curation, and treat the golden set as a living asset, not a one-time deliverable.

LLM-as-judge scoring is convenient but imperfect: the judge model has biases and makes mistakes, so automated scores are a trend signal, not ground truth, and must be calibrated against human review. And strong retrieval metrics do not guarantee a good answer — the model can still misread correctly-retrieved passages — which is exactly why both layers are measured. Evaluation reduces risk; it does not eliminate the need for a human in the loop on high-stakes answers.

Start small and real. A golden set of fifty genuine questions is worth more than a perfect framework you never populate.

• Pull 50-150 real questions from logs, support queues and the documents people actually look up, including out-of-scope ones
• For each, record the correct source passage(s) and a reference answer
• Compute recall and precision at K for retrieval first — fix retrieval before touching the prompt
• Add automated groundedness and citation-accuracy scoring, calibrated against a human-reviewed sample
• Run the harness on every change and store results over time to catch drift, including provider-side model changes