Short answer: buying an AI agent is not the same as buying a normal SaaS tool. Procurement should review task fit, data boundaries, security controls, evaluation evidence, observability, cost model, human approval design, contractual rights, and exit risk before production use.
AI agent vendors often demonstrate impressive workflows: answer questions, call tools, search documents, update records, and draft customer messages. The procurement risk is that the demo hides the hard questions. What data does the agent see? Which actions can it take? How is authorization enforced? What happens after a prompt-injection attempt? Can the buyer audit failures? What happens if the vendor relationship ends?
This checklist is written for technical buyers, security reviewers, product leaders, and procurement teams evaluating AI agent vendors.
1. Define the job before evaluating vendors
Start with a narrow workflow. “We need AI agents” is not a procurement requirement. A useful requirement is closer to: “We need an agent that can triage support tickets, retrieve policy documents, draft a response, and require approval before sending.”
Document:
- The exact workflow the agent must support.
- The user roles that will use it.
- The systems and data sources it must access.
- The actions it may take autonomously.
- The actions that require human approval.
- The success metrics for a pilot.
This prevents vendor evaluation from drifting toward generic model quality or polished demos instead of operational fit.
2. Ask for evaluation evidence, not only examples
A vendor should be able to explain how they evaluate task success, safety, latency, and cost. Example outputs are useful, but they are not enough.
Ask:
- What test datasets are used for this workflow?
- How are false positives, false negatives, and unsafe actions measured?
- How often are prompts, tools, retrieval, and models regression-tested?
- Can the customer run their own evaluation set?
- Can evaluation results be segmented by workflow, user role, language, or data source?
For your internal evaluation framework, see our AI agent evaluation guide.
3. Review data boundaries and retention
Data handling is usually the most important procurement risk. The buyer should know what data the agent receives, where it is processed, how long it is retained, and whether it is used to improve models.
Ask the vendor to specify:
- Which customer data is sent to the vendor or model provider.
- Whether prompts, outputs, files, embeddings, traces, or tool results are retained.
- Whether customer data is used for training or product improvement.
- How tenant isolation is implemented.
- Which subprocessors and model providers are involved.
- Where data is stored and processed geographically.
- How data deletion and export requests work.
Do not rely on broad marketing claims such as “enterprise-grade privacy.” Require precise data-flow answers.
Agent vendors often describe tool use as a feature. Procurement should treat tool use as a risk surface.
Review:
- Which tools the agent can call.
- Whether tools are read-only or can modify systems.
- How user identity and role are passed into tool calls.
- Whether authorization is enforced by deterministic application logic.
- Whether the agent can access cross-tenant or cross-project data.
- Whether high-risk tool calls require approval.
The model should not be the authority for access control. It can propose an action, but the system should enforce permissions.
5. Require human approval design for risky actions
Many enterprise workflows need human approval before the agent takes external or irreversible actions. Procurement should verify that approval is designed into the product, not improvised by users.
Ask whether reviewers can:
- Approve, reject, edit, or escalate a proposed tool call.
- See exact tool arguments before execution.
- See the evidence or retrieved context used by the agent.
- Require approval for specific tools, users, tenants, or risk signals.
- Export an audit trail of approval decisions.
For implementation details, see our human-in-the-loop AI agents checklist.
6. Review security testing and prompt-injection controls
AI agents that read documents, emails, web pages, tickets, or tool outputs are exposed to indirect prompt injection. A vendor should be able to describe how they test and mitigate this risk.
Ask:
- How does the system treat untrusted retrieved content?
- Can malicious documents override system policy or tool rules?
- Are prompt-injection tests part of regression testing?
- Can the buyer provide adversarial test cases?
- Are tool outputs isolated from instruction-following logic?
- What logs prove that a blocked attempt was actually blocked?
OWASP’s LLM application risks and NIST’s AI risk management materials are useful references because they emphasize governance, mapping risks, measurement, and management rather than relying on model behavior alone.
7. Demand observability and auditability
If an agent makes a bad decision, the buyer must be able to reconstruct what happened. Procurement should require traceability before production.
Verify that the vendor can expose:
- Run IDs and trace IDs.
- Prompt and model versions.
- Tool calls, arguments, outputs, and authorization results.
- Retrieved sources and citations.
- Approval decisions.
- Token usage, latency, and cost per run.
- Incident and escalation records.
For a deeper operational list, use the AI agent observability checklist.
8. Understand cost and scaling risk
AI agent pricing can be hard to compare because vendors may charge by seat, workflow, usage, token, task, or custom contract. The buyer should model costs under realistic usage, not a small pilot.
Ask:
- What usage dimensions affect price?
- What happens if context size, retrieval, retries, or tool calls increase?
- Are there hard budget limits or throttles?
- Can costs be attributed by workflow, team, tenant, or user?
- What are the overage terms?
- Can the buyer export usage and cost data?
For production cost controls, see the AI agent cost control checklist.
9. Check incident response and support obligations
AI agent incidents can involve wrong outputs, unsafe tool calls, data exposure, high cost, or customer-visible messages. Contracts and support processes should reflect that.
Review:
- Security incident notification timelines.
- Operational support hours and escalation path.
- Rollback and emergency-disable capabilities.
- Customer access to logs and traces during incidents.
- Responsibilities for customer notification.
- Post-incident report expectations.
Our AI agent incident response checklist can help evaluate whether a vendor’s process is concrete enough.
10. Reduce lock-in and exit risk
Exit risk is often ignored during procurement. It matters because agent systems can become deeply embedded in workflows, prompts, data pipelines, approvals, and integrations.
Clarify:
- Can prompts, evaluations, logs, traces, and configuration be exported?
- Can embeddings or indexes be rebuilt outside the vendor?
- Can the buyer migrate tools and connectors?
- What happens to data after termination?
- How long does the vendor provide transition support?
- Are there contract terms that restrict benchmarking or migration?
A strong vendor should make exit possible even if they hope customers stay.
Minimum AI agent procurement checklist
- Define the exact workflow and pilot success criteria.
- Require evaluation evidence, not just demos.
- Map data flows, retention, subprocessors, and training use.
- Review tool permissions and deterministic authorization.
- Require human approval for risky actions.
- Test prompt injection, retrieval errors, and unsafe tool calls.
- Require traceability, logs, cost attribution, and audit trails.
- Model real production cost and overage risk.
- Review incident response, support, and notification terms.
- Confirm export, deletion, and exit rights before signing.