29 June 2026 · Updated: 30 July 2026
Evaluating RAG — how do you know it answers accurately and from sources
A RAG system can be confident and wrong at the same time. To trust it, you have to measure it — not once, but on every change. RAG evaluation is the set of metrics that separate “sounds reasonable” from “is correct and comes from a source”: retrieval relevance, answer groundedness, citation correctness and refusal correctness. Without them, “it works on my three questions” is not proof, it’s a hunch.
RAG reduces hallucinations but does not switch them off. The question is not “does RAG hallucinate” but “how often, and do we catch it before the customer does”. The answer is evaluation.
Separate the two layers: retrieval and generation
RAG has two stages, and each fails differently. First the retriever fetches passages, then the model writes an answer. If the retriever doesn’t find the right passage, the best model won’t save you. If it does find it and the model makes something up anyway, the problem is in generation. Measure the two layers separately, or you won’t know what to fix.
What to measure — five metrics
- Retrieval relevance. Is the passage that contains the answer among the ones fetched? (recall@k, hit rate). Without this, the rest doesn’t matter.
- Groundedness. Does every sentence of the answer follow from the fetched passages, rather than from the model’s “memory”? This is a direct measure of hallucination.
- Citation correctness. Does the cited source actually contain what the model attributed to it? A citation that doesn’t support the sentence is worse than no citation — it builds false trust.
- Answer relevance. Does the answer address the question, or just sit next to the topic?
- Refusal correctness. Does the system say “I don’t know” when the knowledge base has no answer, instead of making one up? It’s the metric easiest to skip and most expensive to ignore.
How to measure — the method
- A golden set. 50–200 real questions with an expected answer and source. This is your regression test: you build it once and use it forever.
- Judge: human first, then model. Some metrics (groundedness, relevance) can be scored by a model (“LLM-as-judge”), but the judge is fallible too — calibrate it against a human-scored sample before you trust it.
- Regress on every change. Changing the model, the prompt, the chunking, or refreshing the knowledge base can fix one question and break ten. Without a golden set you won’t see it until the customer does.
- Monitor in production. Data and questions shift — what passed tests in March can drift by June. Keep measuring after launch: refusal rate, the distribution of retrieved sources, user reports.
Scorecard: from metric to action
A scorecard is useful only when every metric leads to a decision. The thresholds below are an example for a higher-stakes process, not a standard or industry benchmark. The process owner sets the real values before the test, based on the impact of an error, the cost of escalation, and whether an answer merely informs someone or affects a decision.
| Metric | What it measures | How to calculate it | Example threshold | Action if it fails |
|---|---|---|---|---|
| Retrieval hit rate@5 | Whether retrieval found the right source | answerable questions with a relevant passage in top 5 / all answerable questions | ≥95% | fix chunking, filters or the index; do not tune only the prompt |
| Groundedness | Whether claims follow from retrieved sources | supported claims / all evaluated claims | ≥98% | block release; tighten the instruction and refusal rule |
| Citation correctness | Whether the citation supports the claim | correct citations / all citations | ≥95% | do not present the answer as verified; fix citation mapping |
| Answer relevance | Whether the response resolves the question | relevant answers / all answerable questions | ≥90% | inspect context, answer format and source coverage |
| Refusal correctness | Whether the system refuses without evidence | correct refusals / questions with no answer in the base | ≥98% | narrow the scope or escalate to a human; never guess |
“Answerable questions” are questions for which the approved corpus contains an unambiguous basis. Questions with no basis must be counted separately; otherwise a system can improve its apparent result by answering everything, including questions it should refuse.
Worked example: golden set → result → go/no-go
Consider an illustrative, non-production test of an internal procedures knowledge base. The golden set contains 100 questions: 70 answerable from approved sources and 30 deliberately out of scope. Before the run, the team records the thresholds above and one extra rule: any critical error — a claim that contradicts a safety source — produces a no-go regardless of the average.
Retrieval found the correct passage for 66 of 70 questions (94.3%). Of those 66 answers, 64 were fully grounded (97.0%), 62 cited the right passage (93.9%), and 63 of 70 answered the question directly (90.0%). The system correctly refused 29 of 30 unsupported questions (96.7%). No critical error was found, but four of the five gates missed their agreed threshold.
Decision: no-go for unassisted use. That does not mean “discard the project”. It sets the repair order: retrieval and source coverage first, then citations and the refusal rule, followed by a rerun on the same versioned set. Until the gates pass, the system can operate only as a drafting aid with mandatory human review.
Turn the result into evidence, not a screenshot
The AI Assurance Evidence Pack should contain more than an average: the system and corpus version, golden-set definition, thresholds approved before testing, raw per-question results, critical-error list, go/no-go decision, decision owner and next-test date. That makes the result reproducible and comparable after a change to the model, prompt, chunking or documents.
For metric terminology and automation, compare the documentation for Ragas and RAG evaluation in Amazon Bedrock. Neither tool can set business risk or acceptance thresholds for the process owner.
Why this is ongoing work, not a project
The model updates, documents accumulate, questions evolve. An evaluation that doesn’t keep running ages along with them. That’s why for us RAG evals are part of ongoing care (the retainer), not a one-off sign-off — they are what decides whether quality holds over time. As we wrote about guardrails: without tests and evaluations, a safeguard is decoration. The same is true of RAG.
In short
Measure retrieval and generation separately. Five metrics: retrieval relevance, groundedness, citation correctness, answer relevance, refusal correctness. Build a golden set, calibrate the model-judge against human scores, regress on every change, and keep measuring in production. Then “it works” stops being a hunch and becomes a number.
What next
How we build RAG with sources is on the RAG / knowledge bases page. Evaluations and maintaining quality over time are part of ongoing care. If you already have a RAG system and aren’t sure you can trust it, start with an audit. Before building, also check data readiness for RAG and compare the evaluation loop with technical guardrails.