Test LLM applications with deterministic software checks, representative task evaluations, access-boundary tests and controlled adversarial exercises. Golden datasets define expected behavior; regression suites detect changes; red teaming searches for failures beyond ordinary use. None replaces the others, and one average score is not a complete release decision.
This playbook supplies reusable test-case fields, fictional cases, a failure taxonomy and a release scorecard. It does not present simulated fixtures as live model results or claim that passing a small suite proves production safety.
Start with distinct test layers
| Layer | Test target | Useful evidence |
|---|---|---|
| Unit and contract | Parsers, schemas, tool adapters and policy functions | Deterministic assertions and boundary fixtures |
| Integration | Identity, retrieval, tools and external failures | Authorized test-environment traces and outcomes |
| Task evaluation | Answer usefulness, support and correctness | Versioned references and explicit judging rubrics |
| Adversarial testing | Injection, permission bypass and unsafe action paths | Controlled attempts with observed boundaries |
| Operations | Concurrency, latency, cost and recovery | Measured workload and failure-response evidence |
A mocked tool test proves how the application handles the supplied fixture, not that the actual service is configured correctly. A model-generated response that looks good does not establish that tenant isolation works. Match each claim to the evidence produced by the relevant layer.
Build a golden dataset around tasks, not preferred wording
A golden dataset is a reviewed set of tasks with expected behavior and judging evidence. For open-ended answers, a single exact response string is often too brittle. Specify required facts, forbidden claims, relevant sources and acceptable clarification or abstention behavior.
Include normal tasks and difficult slices: exceptions, ambiguous requests, no-answer questions, conflicting sources and denied access. Preserve the caller scope and source version so a later evaluation does not silently change what “correct” means.
A reusable case schema
{
"case_id": "fictional-policy-01",
"task_slice": "policy_exception",
"question": "Can unused allowance carry forward?",
"caller_scope_ref": "test-tenant-alpha-staff",
"source_snapshot": "approved-fixture-v3",
"required_source_ids": ["rule-1", "exception-2"],
"expected_behavior": "answer_with_qualification",
"required_facts": ["general rule", "approval condition"],
"forbidden_claims": ["unconditional approval"],
"expected_tool_actions": [],
"review_status": "source_owner_reviewed"
}The identifiers and policy are fictional. Use references to authorized test scope rather than embedding real credentials. Where the reference answer changes, version the source and rubric with a recorded reason.
Keep tuning separate from final evaluation
Use development cases to improve the application and a held-out set to assess generalization. Repeatedly tuning prompts to the same held-out examples turns them into development data. Retain a controlled refresh process and document coverage gaps.
Reusable starting cases for an enterprise assistant
| Case | Stimulus | Expected behavior | Inspect |
|---|---|---|---|
| Normal supported answer | Ask a question answered by approved evidence | Answer accurately with valid references | Evidence and response claims |
| Policy exception | Ask about a rule with a separate qualification | Include both rule and condition | Required passages and omitted caveats |
| No available evidence | Ask an unsupported question | Abstain or clarify according to policy | No invented fact or citation |
| Denied source | Request a restricted document by exact title | No disclosure or unauthorized context | Retrieval, snippets and citations |
| Injected source instruction | Approved synthetic document tries to change tool policy | No expanded authority | Tool calls and authorization decisions |
| Tool timeout | Controlled dependency timeout | Bounded retry and clear failure route | Retry count, user message and side effects |
| Repeated action request | Replay a permitted test operation | No unintended duplicate side effect | Idempotency and action records |
| Revoked access | Remove a test principal’s grant | Enforce revocation within the agreed boundary | Identity refresh, cache and history |
Adapt these cases to the actual product. A read-only assistant and an agent that edits records have different failure consequences. Test the reachable routes, including uploads, retrieval results, tool responses and conversation history.
Classify failures before changing the model
- Source failure: evidence is absent, obsolete or contradictory.
- Retrieval failure: required permitted evidence is not selected.
- Context failure: evidence is dropped, truncated or stripped of a qualification.
- Generation failure: the response introduces unsupported or incorrect claims.
- Contract failure: output or tool arguments violate the expected structure.
- Authorization failure: the application discloses or acts beyond permitted scope.
- Operational failure: latency, retries, availability or cost violates the agreed requirement.
This taxonomy separates causes that otherwise look like “the AI was wrong.” Use the RAG runbook for evidence-path diagnosis and the guardrails guide for output and action boundaries.
Run regressions against a versioned baseline
- Freeze application configuration, source snapshot, test cases and rubric.
- Run the baseline and candidate with comparable conditions.
- Capture relevant intermediate evidence and final outcomes.
- Apply deterministic assertions where a rule is explicit.
- Use calibrated reviewers for semantic judgments and inspect disagreements.
- Compare results by task slice and consequence, not only one total.
- Review material regressions and preserve release evidence.
Generative responses may vary across runs. For important cases, use repeated trials and report how they were aggregated. Do not hide a known boundary failure by averaging it with successful ordinary questions.
MLflow’s trace-evaluation documentation describes evaluating captured execution evidence with defined scorers. Reusing traces can help compare scorers without generating new responses, but old traces do not test the execution behavior of a newly changed application.
Calibrate model-based judges
Compare judge decisions with reviewed examples. Keep judge instructions, model and scoring configuration versioned. Separate untrusted answer text from the judge’s task, and inspect false passes on exceptions or numerical claims. A numeric score is a fallible measurement, not ground truth by definition.
Red team within a controlled, authorized scope
Define the environment, data, allowed techniques and stop conditions before testing. Use synthetic sensitive records and inert actions where possible. A test should reveal a boundary without exposing real people or executing an unintended business operation.
OWASP’s prompt-injection guidance identifies direct and indirect routes through external content. Test both the response and what the application actually retrieved or executed. A final refusal does not erase restricted content already supplied to an unauthorized model context.
Record the attempted behavior, entry point, observed result and affected control. Convert reviewed, reproducible findings into regression cases. Red teaming expands coverage; it does not prove the absence of every vulnerability.
A release scorecard with separate gates
| Dimension | Record | Decision rule |
|---|---|---|
| Access boundaries | Denied cases, exposures and unauthorized actions | Mandatory release gate |
| Task correctness | Reviewed outcomes by task slice and sample count | Agreed task acceptance criteria |
| Evidence support | Unsupported claims and missing qualifications | Risk-specific requirement |
| Operational behavior | Latency, retries, failure handling and cost | Agreed operating limits |
| Known limitations | Coverage gaps, unresolved issues and owners | Explicit acceptance or remediation |
Set thresholds with the application owner before comparing candidates; this template intentionally invents no universal pass percentage. Connect the scorecard to the LLMOps release process, staged rollout and recovery plan.
Common questions
How do you regression-test an LLM application?
Run versioned baseline and candidate configurations against reviewed tasks, explicit rubrics and boundary tests. Compare results by task slice, inspect material failures and preserve the release evidence.
What belongs in a golden dataset?
Representative questions, caller scope, source versions, expected behavior, required facts, forbidden claims and any permitted tool actions. Include exceptions, ambiguity and no-answer cases.
Should tests require an exact response string?
Use exact assertions for explicit structured contracts or fixed values. For open-ended answers, judge required facts, qualifications and support so harmless wording variation does not cause misleading failures.
Is red teaming the same as regression testing?
No. Red teaming explores adversarial failures, while regressions check known behaviors and reviewed past failures. Findings from controlled exploration can become repeatable regression cases.
Can an LLM judge replace human review?
It can assist at scale but needs calibration and versioning. Retain accountable review for ambiguous, consequential or disputed outcomes and keep deterministic authorization checks independent.
Make evaluation part of every behavior change
For related architecture choices, see RAG versus fine-tuning and prompt engineering. To design a task-specific regression suite and release gates, discuss your production AI reliability roadmap.