Evaluate a RAG system at three levels: whether it retrieved the required evidence, whether its answer is supported by that evidence, and whether the final response correctly serves the user’s permitted task. Retrieval recall, faithfulness and answer relevance measure different properties. None alone proves that a chatbot is accurate or safe to deploy.

This guide provides a gold-test-case template, a runnable scorecard calculation and a release checklist. The demonstration uses explicitly supplied labels, not a live retriever or an automated judge. Its results illustrate how failures differ; they are not a production benchmark.

Start with the questions the application must answer

• Can the retriever find every passage required for a supported answer?
• Does the response preserve qualifications and avoid unsupported claims?
• Does it answer the actual question rather than merely repeat related content?
• Does it abstain or clarify when evidence is absent or ambiguous?
• Are source permissions and document versions respected?

RAG evaluation metrics: what each one tells you

Retrieval recall measures whether known required evidence is present. Retrieval precision measures how much of the retrieved set is relevant. Define the evidence unit, retrieval depth and duplicate treatment before comparing numbers. Source passages, chunks and answer claims are not interchangeable units.

Faithfulness or answer support asks whether generated claims follow from the supplied context. Response relevance asks whether the response addresses the question. Factual correctness additionally requires an appropriate reference or authoritative judgment. A response can faithfully repeat an outdated policy and still be wrong for today’s question.

Ragas provides documented implementations of faithfulness, context recall and response relevance. Read the selected implementation’s required inputs and calculation rather than assuming two tools with similarly named metrics produce equivalent scores.

A gold dataset template for enterprise questions

A useful test case includes the question, caller scope, corpus version, required source references, expected behavior and a judging rubric. Keep original source references stable even when chunking changes. Otherwise the evaluation may reward the splitter that happened to create the labeled chunk IDs.

Include straightforward queries, exact identifiers, paraphrases, multi-passage questions, stale-source cases and no-answer scenarios. Add denied-access cases separately so the system is not rewarded for finding restricted evidence. Use real permitted examples when available and label synthetic cases as synthetic.

Separate tuning from evaluation

Use a development set for changes and a held-out set for release assessment. Keep the expected answers out of the retrieval corpus unless they legitimately belong there. Review labels with the source owner, resolve disagreements and version any changed rubric. A test set contaminated by repeated prompt tuning can overstate generalization.

Runnable scorecard: locate the failure layer

The following Python demonstration calculates ID-based recall and precision plus a manually labeled claim-support fraction. It does not run Ragas, query a vector database or infer whether a claim is supported. The labels are declared inputs, and the code only performs the arithmetic.

import json

CASES = [
    {'case': 'complete retrieval, extra claim',
     'gold': {'leave-rule', 'approval-rule'},
     'retrieved': ['leave-rule', 'approval-rule', 'travel-rule'],
     'supported_claims': 2, 'total_claims': 3},
    {'case': 'missed evidence',
     'gold': {'error-procedure'}, 'retrieved': ['general-overview'],
     'supported_claims': 0, 'total_claims': 1},
    {'case': 'fully supported fixture',
     'gold': {'current-policy'}, 'retrieved': ['current-policy'],
     'supported_claims': 1, 'total_claims': 1},
]

def score(case):
    gold, retrieved = set(case['gold']), set(case['retrieved'])
    hits = len(gold & retrieved)
    total = case['total_claims']
    if not 0 <= case['supported_claims'] <= total:
        raise ValueError('Invalid claim labels')
    return {
        'case': case['case'],
        'id_recall': hits / len(gold) if gold else None,
        'id_precision': hits / len(retrieved) if retrieved else None,
        'support_fraction': case['supported_claims'] / total if total else None,
    }

if __name__ == '__main__':
    scores = [score(case) for case in CASES]
    assert scores[0]['id_recall'] == 1
    assert scores[1]['id_recall'] == 0
    assert scores[2]['support_fraction'] == 1
    print(json.dumps(scores, indent=2))

The local run produced these values from the supplied labels. The first fixture shows why complete retrieval does not guarantee a supported response: an extra generated claim lowers the support fraction. The second points to missing evidence. The third passes these narrow checks, but still needs relevance, correctness, freshness and access judgments before being treated as a successful application task.

The code returns an undefined value for a metric with no applicable denominator. In production, define how no-answer and empty-reference cases are handled instead of silently converting every undefined metric to a perfect score. Report exclusions and sample counts.

Automated judges need calibration and versioning

A model-based judge can help scale review, but it is another fallible component. Compare its decisions with expert-reviewed cases, especially negation, exceptions, numerical values and source conflicts. Inspect false passes as well as false failures.

Record the judge model, prompt, metric implementation and configuration. Keep the evaluated answer and retrieved content clearly separated from judge instructions. An adversarial passage should not be able to tell the evaluator to assign a perfect score. Recalibrate when the judge or rubric changes.

Use human review for ambiguous or consequential cases and for samples that expose disagreement. Do not describe an automated metric as ground truth merely because it returns a number between zero and one.

Diagnose before changing the prompt

Low recall

Inspect parsing, source availability, chunk boundaries, filters, query formulation and candidate depth. Confirm the expected passage exists and is permitted. Changing answer wording cannot recover a passage that never reached the context.

Good recall, weak support

Inspect whether qualifications were omitted, conflicting passages were mixed or the response introduced unsupported claims. Evaluate context assembly and answer validation, not only the underlying model.

Supported but irrelevant response

Check whether the retrieved material answers the actual question and whether the response addresses the requested scope. Quoting a correct paragraph from the wrong policy is not a useful answer.

High average, poor exception handling

Break results down by query type and risk. A large set of easy lookups can hide failures on exceptions, no-answer questions or restricted documents. Report those slices with their sample counts.

For configuration choices that influence retrieval, see hybrid, vector and keyword search and RAG chunking strategies.

Release and production monitoring checklist

1. Freeze the evaluated corpus, retrieval settings, prompts and relevant model versions.
2. Agree acceptance criteria with the application and source owners.
3. Run held-out retrieval, support, correctness and boundary tests.
4. Review failures by slice rather than relying only on one composite score.
5. Measure latency and accounted cost alongside quality.
6. Retain a rollback path and the evaluation evidence for the release.
7. Sample permitted production tasks under the approved data-handling policy.
8. Add reviewed incidents to regression tests without contaminating the held-out set.

There is no universal score threshold that proves every enterprise RAG application is ready. Define objectives from task requirements and failure impact. Keep changes attributable by testing one meaningful variation at a time.

Common questions about RAG evaluation

Evaluate the evidence behind the answer

Request an enterprise RAG architecture review with Prominent AI Services LLP. Bring representative questions, approved sources and examples of supported and unsupported answers.

Technical references reviewed 2 November 2026. The scorecard calculation was run locally on declared fixtures; no client performance results are claimed.