Short answer: a useful RAG evaluation checklist should test four layers separately: retrieval quality, grounding, citations, and final answer quality. If those layers are merged into one “the answer looked good” score, production failures become hard to diagnose.
Retrieval-augmented generation looks simple in demos: retrieve relevant context, pass it to a model, and ask for an answer. In production, the hard part is not only whether the answer is fluent. The hard part is whether the system found the right evidence, ignored the wrong evidence, grounded each important claim, cited sources accurately, and behaved safely when the available context was incomplete or hostile.
This checklist is written for product teams, AI engineers, and security reviewers who need a practical way to evaluate RAG systems before they become customer-facing workflows.
Why RAG evaluation needs its own checklist
Generic LLM evaluation is not enough for RAG. A chatbot can produce a helpful answer from its parametric memory, but a RAG system is usually expected to answer from a specific knowledge base, policy corpus, customer document set, or internal ticket archive. That changes the release criteria.
A RAG system can fail in at least four different ways:
- The retriever does not find the right documents.
- The retriever finds useful documents but also includes distracting or unsafe context.
- The model sees the right context but makes claims that are not supported by it.
- The answer cites sources, but the cited passages do not actually support the claims.
Tools and frameworks increasingly reflect this separation. Ragas documents metrics for dimensions such as faithfulness, answer relevance, context precision, and context recall. LangSmith describes evaluation around datasets, evaluators, experiments, and comparisons. LlamaIndex includes evaluation workflows for retrieval and response quality. OpenTelemetry’s GenAI semantic conventions also reinforce the need to capture model, token, request, response, and operation-level telemetry for production analysis.
1. Define the task before measuring it
Start by defining what the RAG system is allowed to do. A legal document assistant, customer support copilot, internal engineering search tool, and medical policy helper should not share the same evaluation rubric.
For each task type, write down:
- The target user and their expected intent.
- The allowed source corpus.
- The answer format expected by the user.
- Whether citations are mandatory.
- What the assistant must do when evidence is missing.
- What sources or actions are explicitly out of scope.
This matters because “I do not have enough evidence to answer” may be the correct result for a regulated support workflow, even when a fluent speculative answer would impress a demo audience.
2. Evaluate retrieval quality first
Retrieval should be scored before generation. If the right evidence never reaches the model, the generator cannot reliably produce a grounded answer.
At minimum, test:
- Context recall: did the retrieved set include the documents or chunks needed to answer the question?
- Context precision: how much of the retrieved context was actually useful?
- Ranking quality: were the most useful chunks near the top?
- Freshness: did retrieval prefer current policy or product documentation over stale copies?
- Permission boundaries: did the system retrieve only content the user is allowed to see?
Do not evaluate retrieval only on happy-path questions. Include vague questions, misspellings, synonyms, old product names, multilingual queries, short queries, and queries that mention concepts across multiple documents.
3. Evaluate grounding and faithfulness
Grounding asks a direct question: are the answer’s material claims supported by the retrieved evidence? This is different from asking whether the answer sounds reasonable.
A practical grounding review should check whether the answer:
- Uses the retrieved evidence rather than unsupported model memory.
- Avoids adding numbers, dates, prices, requirements, or policies that are not in the context.
- Distinguishes between confirmed facts and uncertain interpretations.
- Refuses or asks for clarification when the evidence is insufficient.
- Does not combine two unrelated passages into a false conclusion.
For high-risk workflows, manually review a sample of model outputs even if automated evaluators are used. Automated grading is useful for regression detection, but it should not be the only gate for customer-facing answers that affect money, safety, compliance, or reputation.
4. Evaluate citation quality
Citations are often treated as decoration. They should be treated as part of the answer contract.
For every cited source, verify:
- The cited source exists and is accessible to the user.
- The cited passage supports the exact claim next to it.
- The citation is not merely topically related.
- The answer does not cite one source while borrowing a claim from another.
- The system does not cite retrieved documents that were not used.
A common failure pattern is “citation laundering”: the answer makes an unsupported claim, then attaches a real source that looks authoritative but does not prove the claim. This is especially damaging in audit, legal, healthcare, financial, and enterprise procurement use cases.
5. Evaluate answer quality separately
Once retrieval and grounding are tested, evaluate the answer as a user experience. Good RAG output should be correct, but it also needs to be useful.
Score answer quality on:
- Directness: does it answer the user’s actual question?
- Completeness: does it include all required steps, exceptions, or constraints?
- Conciseness: does it avoid burying the answer in generic explanation?
- Actionability: does it tell the user what to do next?
- Format compliance: does it follow the required output structure?
Keep this score separate from grounding. A response can be perfectly grounded but poorly written. It can also be well written but unsupported. Those failures require different fixes.
6. Include negative and adversarial tests
RAG systems are exposed to untrusted text. Documents, webpages, tickets, emails, PDFs, and tool outputs can contain instructions that try to manipulate the assistant.
Your evaluation set should include:
- Documents containing prompt injection instructions.
- Conflicting sources where one document is newer or more authoritative.
- Questions where the correct answer is “not enough evidence.”
- Cross-tenant or cross-customer retrieval attempts.
- Queries that ask the assistant to ignore citations or reveal hidden prompts.
For a deeper security-specific test plan, see our prompt injection testing checklist and our guide to prompt injection in RAG and tool-using agents.
7. Build a golden dataset and regression suite
A RAG system should not be evaluated only once. Each change to embedding model, chunking, reranking, prompt template, source corpus, or model provider can change behavior.
Maintain a golden dataset with:
- Real user questions from logs, anonymized where necessary.
- Expected relevant documents or chunks.
- Expected answer requirements.
- Examples where the assistant must refuse or ask for more context.
- Known hard cases that previously failed.
Run this dataset before major releases and after any retrieval or prompt changes. Track pass rates over time rather than relying on isolated manual reviews.
8. Log the evidence trail
If production users report a bad answer, you need enough telemetry to reconstruct what happened. Store the query, retrieved chunk IDs, ranking scores, source versions, model name, prompt version, answer, citations, token counts, latency, and evaluator result where appropriate.
This evidence trail also helps reduce cost. If the system retrieves too many low-value chunks, uses an expensive model for simple questions, or adds unnecessary generation steps, evaluation logs will expose the bottleneck.
For a broader production evaluation framework, see our article on how to evaluate an AI agent across task success, safety, cost, and latency.
Minimum RAG evaluation checklist
- Define task scope, allowed sources, and refusal behavior.
- Measure retrieval recall, precision, ranking, freshness, and permissions.
- Check whether every material answer claim is grounded in retrieved evidence.
- Verify that citations support the exact claims they are attached to.
- Score answer quality separately from grounding.
- Include missing-evidence, conflicting-source, and prompt-injection cases.
- Maintain a golden dataset and run regression tests before release.
- Log enough trace data to debug failures after deployment.