Short answer: an LLM regression test suite is a repeatable set of prompts, inputs, expected behaviors, safety cases, scoring rules, and deployment gates that catches quality, safety, cost, and latency regressions before a model, prompt, retrieval index, tool, or orchestration change reaches production.
LLM applications fail differently from normal software. The code may stay the same while a prompt edit, model upgrade, retrieval change, tool schema change, or memory policy alters behavior. A useful regression suite tests the whole application path, not only the model call.
This checklist is written for teams that already have an LLM chatbot, RAG workflow, or agent and need a practical release gate.
1. Define what must not regress
Start with the product risks, not the test framework. List the behaviors that would make a release unacceptable.
Common regression targets include:
- Task success on representative user requests.
- Answer grounding and citation quality for RAG systems.
- Tool-call correctness and argument validity for agents.
- Refusal behavior for unsafe or unsupported requests.
- Prompt-injection resistance.
- Output format compatibility with downstream code.
- Latency, token use, and cost per task.
- Privacy boundaries, tenant boundaries, and secret handling.
For agent-specific release criteria, pair this with the AI agent evaluation checklist.
2. Build a versioned test dataset
A regression suite needs stable test cases. Store each case with the input, context, expected behavior, labels, and the reason it exists.
Include:
- Golden-path tasks that should always work.
- Edge cases from real support tickets or production traces.
- Known previous failures.
- Prompt-injection and jailbreak attempts.
- Out-of-domain questions that should be refused or escalated.
- Ambiguous requests that should trigger clarification.
- Tool-use cases with expected tool names and argument constraints.
LangSmith’s evaluation workflow is built around datasets and experiments for comparing application behavior across versions (LangSmith evaluation docs). OpenAI’s Evals guide similarly treats evaluations as structured tests over model or application outputs (OpenAI Evals guide).
3. Separate deterministic checks from judgment checks
Do not use an LLM judge for everything. Some regressions are better caught with deterministic assertions.
Use deterministic checks for:
- Valid JSON, XML, Markdown, or schema shape.
- Required fields and allowed enum values.
- No banned phrases, secrets, or unsupported claims.
- Tool name and argument validation.
- Presence of citations or source identifiers.
- Latency and cost thresholds.
Use model-graded or human-reviewed checks for:
- Helpfulness and completeness.
- Groundedness against source material.
- Instruction following when multiple constraints interact.
- Tone and escalation quality.
- Subtle safety or privacy failures.
Promptfoo documents assertion-based checks for expected outputs, including exact, contains, regex, JavaScript, and model-graded assertions (Promptfoo expected outputs docs). The practical rule is simple: use code when code can decide, and reserve model judges for cases that genuinely need semantic evaluation.
4. Test retrieval separately from generation
For RAG systems, test retrieval before judging final answers. If the right evidence is missing, the generator is being asked to compensate for a retrieval failure.
Track:
- Whether the expected document appears in top-k retrieval.
- Whether the retrieved chunk contains the answer evidence.
- Whether metadata filters enforce tenant, product, locale, or permission boundaries.
- Whether citations point to the right source.
- Whether stale documents are excluded after updates.
LlamaIndex provides evaluation modules for retrieval and response quality (LlamaIndex evaluating docs). For a deeper RAG-specific checklist, use the RAG evaluation checklist.
5. Add tool-call and agent-path tests
Agents need tests for decisions, not only final messages. A run can produce a plausible answer after calling the wrong tool or passing unsafe arguments.
For each high-risk workflow, capture:
- Expected tool selection.
- Allowed and forbidden tool arguments.
- Approval requirements before side effects.
- Retry and fallback behavior.
- Audit-log fields that must be present.
- Final user-visible response constraints.
For risky tool calls, human approval gates should be tested as part of the regression suite, not treated as a separate manual control. See the human-in-the-loop AI agents checklist.
6. Include security and privacy regression cases
Security tests should be part of normal release evaluation. They should run whenever prompts, tools, retrieval indexes, policy text, or model settings change.
Include cases for:
- Prompt injection in retrieved content.
- Attempts to reveal system prompts or hidden policy text.
- Cross-tenant retrieval attempts.
- Secret, token, or credential exposure.
- Unauthorized tool calls.
- Unsafe instructions embedded in uploaded files.
- Data retention or deletion-policy violations.
The OWASP Top 10 for LLM Applications is a useful source for threat categories to convert into regression cases (OWASP LLM Top 10). For hands-on security cases, use the prompt injection testing checklist.
7. Set release thresholds before running the test
A regression suite is only useful if the team agrees on release criteria before seeing the result.
Example gates:
- No critical safety, privacy, or authorization failures.
- No schema-breaking output failures in production integrations.
- Task-success score cannot drop more than an agreed tolerance.
- Groundedness or citation score cannot drop below the current baseline.
- P95 latency must remain under the service target.
- Average cost per task must stay within budget.
- Known previous failures must continue to pass.
Track cost and latency alongside quality. A change that improves answer quality while doubling cost may still be unacceptable for production. The AI agent cost control checklist covers the budget side.
8. Run comparisons, not one-off scores
Regression testing is about deltas. Compare the candidate version against the current production baseline.
Compare:
- Prompt version versus current prompt.
- Model version versus current model.
- Retriever settings versus current retriever settings.
- Tool schema changes versus current tool schema.
- Memory policy changes versus current memory policy.
- System-level orchestration changes versus current orchestration.
Record the exact model, prompt, retrieval index, tool definitions, system instructions, and evaluation code version for every run. Without versioning, a passing test is hard to reproduce.
9. Feed production failures back into the suite
The best regression cases often come from production. When a user reports a bad answer, an agent takes the wrong action, or a monitoring alert fires, convert the event into a test case.
For each incident-derived test, store:
- The original user input.
- Relevant retrieved context or tool outputs.
- The observed bad behavior.
- The expected behavior after the fix.
- The severity and owner.
- The date it was added to the suite.
For tracing and run capture, see the AI agent observability checklist. OpenTelemetry’s generative AI semantic conventions can help standardize the telemetry fields used in those records (OpenTelemetry GenAI semantic conventions).
Minimum LLM regression test suite checklist
- Define the product behaviors that must not regress.
- Create a versioned dataset of golden paths, edge cases, prior failures, and attack cases.
- Use deterministic assertions for structured outputs, tool arguments, citations, latency, and cost.
- Use model or human judgment only where semantic evaluation is required.
- Test retrieval quality separately from final answer quality.
- Test agent tool choice, tool arguments, approval gates, and audit logs.
- Include prompt-injection, privacy, authorization, and secret-exposure cases.
- Set release thresholds before running evaluations.
- Compare candidate versions against the production baseline.
- Convert production failures into permanent regression tests.