Short answer: AI agent cost control requires more than choosing a cheaper model. You need per-run budgets, token accounting, prompt and context caching, model routing, loop limits, retrieval limits, cost alerts, and trace-level attribution. Otherwise, one badly designed workflow can quietly consume a large share of your monthly AI budget.
AI agents are cost-amplifiers. A single user request may trigger several model calls, retrieval steps, tool calls, retries, summarization passes, and final answer generation. That makes the cost profile different from a simple one-prompt chatbot.
If the system is useful, usage will grow. If the system is poorly controlled, spend will grow faster than value. This checklist focuses on practical controls that teams can apply before production and during normal operation.
Why AI agent costs are harder to predict
Traditional software costs are usually tied to infrastructure, storage, database queries, and network traffic. Agent costs are also tied to dynamic behavior:
- The model may call multiple tools before answering.
- Retrieval may add long context blocks to each request.
- Failures may trigger retries or fallback models.
- Long conversations can carry unnecessary history forward.
- Autonomous loops can continue longer than expected.
- High-quality models may be used for tasks that do not need them.
This is why cost control should be designed as part of the agent architecture, not added after the first large bill.
1. Measure cost at the run level
Start by attributing cost to one user-facing agent run. A useful trace should include token counts, model names, cache usage, latency, tool calls, retrieval size, retries, and final status.
Track:
- Input tokens and output tokens per model call.
- Total tokens per agent run.
- Estimated cost per run and per workflow type.
- Cost by model, tool path, user segment, and environment.
- Retry and fallback cost.
- Cache hit rate and cache savings where available.
The OpenTelemetry GenAI semantic conventions are useful here because they define telemetry concepts for generative AI operations, including token usage and model operation metadata. The exact implementation can vary, but the principle is stable: cost must be observable at the same granularity as agent behavior.
2. Set budgets before runtime
Every agent workflow should have explicit limits. These limits should exist at several levels:
- Per model call: maximum input size, output size, and timeout.
- Per agent run: maximum model calls, tool calls, retries, and total tokens.
- Per user or tenant: daily or monthly usage budget.
- Per environment: lower limits for development and staging.
- Per workflow: stricter budgets for low-value background tasks.
Budgets should fail safely. When a run exceeds its budget, the agent should stop, explain the limitation, ask for confirmation, or escalate to a human depending on the use case.
3. Use prompt and context caching deliberately
Many production agent workflows reuse large system prompts, tool instructions, schemas, policy documents, or static context. Caching can reduce repeated processing of those stable parts.
OpenAI documents prompt caching for reducing latency and cost when the same prompt prefixes are reused. Anthropic documents prompt caching for reusing stable context across requests. Google’s Gemini documentation describes context caching for saving and reusing large context. The implementation details differ by provider, but the architectural lesson is the same: separate stable context from dynamic user input.
Good cache candidates include:
- Long system instructions.
- Tool schemas and tool-use policies.
- Static product documentation.
- Compliance or security policies used repeatedly.
- Few-shot examples that rarely change.
Bad cache candidates include user-specific secrets, frequently changing data, and context that must reflect the latest state at the moment of execution.
4. Route tasks to the smallest sufficient model
Not every step needs the strongest model. Use stronger models where reasoning quality materially changes the outcome, and smaller or cheaper models for simpler steps.
Common routing patterns:
- Use a small model for classification, formatting, and short extraction tasks.
- Use a stronger model for complex planning, ambiguous reasoning, and high-risk final answers.
- Use deterministic code instead of a model for calculations, authorization, validation, and formatting where possible.
- Use retrieval filters before generation to reduce unnecessary context.
- Use human approval instead of repeated model attempts for high-risk uncertainty.
Model routing should be evaluated, not guessed. Compare quality, latency, and cost using a fixed dataset before changing routing rules.
5. Limit retrieval and conversation context
RAG systems can become expensive when too many chunks are retrieved or when large documents are passed into every call. More context is not always better. It can increase cost, latency, and distraction.
Control retrieval cost by:
- Setting maximum retrieved chunks per query.
- Using reranking to keep only the most relevant context.
- Summarizing long documents before repeated use.
- Dropping stale conversation history that no longer affects the task.
- Separating retrieval evaluation from answer evaluation.
For a deeper retrieval-specific process, see our RAG evaluation checklist.
6. Stop loops and retry storms
Autonomous agents can retry the same failing action, call tools repeatedly, or keep planning without meaningful progress. This is a reliability problem and a cost problem.
Set hard limits for:
- Maximum tool calls per run.
- Maximum retries per tool.
- Maximum planning iterations.
- Maximum wall-clock time.
- Maximum total token spend.
When a limit is reached, the agent should stop with a clear state: success, partial completion, safe failure, escalation, or manual review. Silent looping is unacceptable in production.
7. Alert on spend anomalies
Cost alerts should be tied to behavior, not only monthly invoices.
Useful alerts include:
- Cost per run exceeds a threshold.
- Token usage per run increases after deployment.
- Cache hit rate drops unexpectedly.
- Retry rate or fallback model usage spikes.
- A tenant or user segment consumes usage abnormally fast.
- A workflow exceeds its expected cost band.
These alerts should connect to traces. If an alert fires, the operator should be able to inspect the exact runs that caused the spike.
8. Make cost part of release evaluation
Cost should be part of the release gate, not a post-launch surprise. Before production, test the agent on representative tasks and record:
- Median and p95 cost per run.
- Median and p95 latency per run.
- Token usage by step.
- Tool-call count by workflow.
- Quality score or task success rate.
- Failure and escalation rate.
The best system is not always the cheapest. The goal is to understand the tradeoff between quality, safety, latency, and cost. For a broader framework, see our AI agent evaluation guide and AI agent observability checklist.
Minimum AI agent cost control checklist
- Attribute token usage and estimated cost to every agent run.
- Set maximum model calls, tool calls, retries, and total tokens per run.
- Use caching for stable prompts, schemas, and repeated context.
- Route simple tasks to smaller models or deterministic code.
- Limit retrieved chunks and conversation history.
- Stop loops with explicit safe-failure behavior.
- Track cache hit rate, retry rate, fallback usage, and cost per workflow.
- Alert on cost spikes, token spikes, and cache regressions.
- Review p50 and p95 cost before production release.
- Convert expensive production traces into regression tests.