A checklist for buyers evaluating whether an AI agent vendor is ready for production use, not just a polished demo.
The demo is not the product
AI agent demos are often smooth because the path is narrow, the data is prepared, and the presenter knows what not to ask. A buyer needs a different lens. The real question is not whether the agent can complete a rehearsed task. The question is whether the vendor can explain what happens when the input is ambiguous, the tool fails, the policy conflicts, the user lacks permission, the data is stale, or the model is confidently wrong. A buyer readiness checklist helps separate impressive interaction design from production readiness.
Ask what the agent is allowed to do
Start with the permission model. What can the agent read, write, send, delete, approve, purchase, change, or recommend? Which actions are automatic, which need human approval, and which are impossible by design? If the vendor cannot draw the boundary clearly, the buyer should assume the operational boundary is also unclear. Ask for examples of blocked actions and denied tool calls. A system that never blocks anything is not necessarily flexible; it may simply lack controls.
A production AI agent must respect user roles, tenant boundaries, and object-level permissions. It should not infer authority from the user’s wording. Ask whether authorization is enforced outside the model path, how tool calls are validated, whether the agent can access data from another customer, and how role changes take effect. Buyers should be careful with vendors who describe permission checks only as prompt instructions. Prompts can guide behavior, but they are not a security boundary.
Ask how failures are reviewed
Every vendor will say they test their system. The stronger question is how they review failures. Do they keep traces? Can they replay bad runs? Do they label incidents by failure mode? Do they add regression cases after production issues? Do support teams have a path to escalate a suspicious agent action to engineering? If a vendor cannot describe a failure loop, the buyer may become the failure loop after purchase.
Ask about retrieval and source trust
Many agents rely on RAG or connected knowledge bases. Buyers should ask how sources are selected, how stale documents are handled, how conflicting sources are resolved, and whether retrieved text can override system policy. Ask whether the agent cites sources, whether citations are checked, and whether the system distinguishes answer confidence from action permission. A grounded answer is not automatically a safe action.
Ask about logs and auditability
Auditability matters even for small deployments. A buyer should know whether the system logs prompts, retrieved context, tool calls, arguments, approvals, denials, final outputs, user IDs, and timestamps. The vendor should also explain retention, access controls, export options, and redaction. Logs are useful for incident review, but logs can also become sensitive data stores. A mature vendor understands both sides of that tradeoff.
Ask about rollout controls
Production readiness is partly about the ability to start small. Can the buyer enable the agent for one team, one workflow, one customer segment, or one tool before expanding? Can permissions be reduced quickly? Is there a kill switch? Can the vendor support manual review mode, shadow mode, or read-only mode? If the only launch option is full deployment, the vendor is pushing operational risk onto the buyer.
Ask about model and prompt changes
AI systems change underneath the buyer. Vendors may update prompts, switch models, change retrieval settings, or add tools. Buyers should ask how changes are tested, how customers are notified, whether there is a rollback plan, and whether high-risk changes require approval. A vendor that treats every prompt change as a minor internal detail may not understand how much behavior can shift from a small edit.
Ask about cost and rate limits
Agent cost is not only token spend. It includes retries, tool calls, slow escalations, support review, and expensive model routes. Buyers should ask for cost controls, rate limits, usage dashboards, alerting thresholds, and contractual clarity around overages. A vendor should be able to explain cost per completed workflow, not only average tokens per message. Without limits, a mistake can become both a product problem and a billing problem.
Decide based on evidence
The best buying process asks for evidence, not confidence. Ask for a sample audit log, a sample incident review, a sample evaluation report, a security architecture summary, and a clear statement of supported and unsupported workflows. A buyer does not need perfection. They need evidence that the vendor knows where the risks are, has controls for the biggest ones, and can respond when the agent behaves badly in the real world.
How to use this resource
Use this article as a working review aid. It is not a legal certification, formal penetration test, or guarantee that an AI system is safe. The useful next step is to turn each section into evidence: traces, policies, test cases, owners, thresholds, and launch decisions that can be reviewed later.
Implementation notes for teams
The strongest way to apply this checklist is to run it inside the normal operating rhythm of the product team. Assign one owner, set a review date, and collect examples from the actual system rather than from a slide deck. A good review includes a successful run, a failed run, a borderline case, and at least one case that came from a real user workflow. That mix prevents the team from only testing happy paths. It also makes the conversation practical: people can see what the agent did, which tool calls were made, what information was exposed, how long the task took, and where a human would have needed to step in.
Keep the first version deliberately simple. A spreadsheet, issue template, or short internal page is enough if it captures the decision, the evidence, and the owner. The important habit is repeatability. When the model changes, a prompt is edited, a tool permission is expanded, or a customer workflow is added, the same review should be easy to run again. Over time, the team can automate the checks that repeat often and reserve human review for judgment-heavy questions such as customer harm, ambiguous instructions, security tradeoffs, and business impact.
Related IBBS resources
AI Agent Buyer Readiness Checklist: A checklist for buyers evaluating whether an AI agent vendor is ready for production use, not just a polished demo. Read it at https://ibbs.ai/2026/07/08/ai-agent-buyer-readiness-checklist/
For a broader review, start with the AI Agent Readiness Self-Assessment or read the sample audit report.