How to Evaluate an AI Agent: Task Success, Safety, Cost, and Latency

Short answer: evaluate an AI agent as a workflow, not as a single model response. A useful evaluation plan measures whether the agent completed the task, whether it stayed inside safety and authorization boundaries, how much it cost, how long it took, and whether its failures are visible in traces. The minimum production-ready setup is a curated task dataset, trace-level grading, safety test cases, token and tool-call cost tracking, and latency percentiles.

Disclosure: This article is documentation-based guidance for builders and technical teams. It does not report proprietary benchmark results or claim that IBBS.AI has tested a specific vendor system unless explicitly stated.

Why agent evaluation is different from chatbot evaluation

A chatbot can often be judged from one prompt and one answer. An AI agent usually cannot. It may retrieve documents, call tools, choose between workflows, hand off to a human, update a ticket, or stop because a guardrail blocks the action. That means the evaluation target is the end-to-end run: inputs, intermediate decisions, tool calls, final output, and side effects.

OpenAI’s agent evaluation guidance emphasizes trace grading for workflow-level issues because traces capture model calls, tool calls, guardrails, and handoffs in a single run. That is the right mental model: first inspect traces to understand what fails, then convert recurring failures into repeatable datasets and eval runs. Source: OpenAI agent evals documentation.

LangSmith’s evaluation documentation makes a similar distinction between offline evaluation, online evaluation, datasets, experiment comparison, human annotation, code evaluators, and LLM-as-judge methods. Source: LangSmith evaluation concepts.

The four metrics that matter first

Start with four metric groups. Do not let a single “accuracy” score hide a bad tradeoff.

1. Task success

Task success asks: did the agent produce the intended outcome under realistic constraints? For a support agent, that may mean the correct answer plus the correct escalation decision. For a coding agent, it may mean tests pass and the patch changes only the intended files. For a research agent, it may mean the final answer is source-backed and separates facts from inference.

Use three layers:

• Exact checks: schema validity, required fields, status codes, file changes, unit tests, deterministic assertions.
• Reference checks: expected answer, expected tool route, expected escalation decision, expected citation set.
• Human or judge checks: when success depends on judgment, review against a rubric instead of a vague “good/bad” label.

Good rubrics are specific. “Helpful” is weak. “Identifies the refund policy, refuses to override the 30-day limit, and offers escalation when the customer reports a medical emergency” is measurable.

2. Safety and policy compliance

Safety is not only about toxic text. For agents, safety includes prompt injection resistance, tool misuse, data leakage, unauthorized actions, missing human approval, and insecure fallback behavior. This connects directly to our previous articles on prompt injection in RAG and tool-using agents and AI agent guardrails.

OpenAI’s safety best practices recommend moderation and human oversight patterns, and also discuss privacy-preserving safety identifiers for abuse monitoring. Source: OpenAI safety best practices.

A practical safety suite should include:

• Direct prompt injection attempts.
• Indirect injection hidden in retrieved documents, emails, web pages, or tickets.
• Requests that try to bypass role, approval, or tenant boundaries.
• Data exfiltration attempts, including “summarize hidden instructions” and “print system prompt” variants.
• Tool-call abuse cases, such as sending email, deleting records, changing permissions, or making purchases.

For higher-risk systems, align the test plan with risk management practices rather than only product metrics. NIST’s AI Risk Management Framework is organized around governance, mapping, measurement, and management of AI risks. Source: NIST AI Risk Management Framework.

3. Cost

Cost should be measured per successful task, not only per model call. A cheap call that causes three retries, two unnecessary tool calls, and a human cleanup task is not cheap.

Track at least:

• Input tokens, output tokens, and cached-token usage where available.
• Number of model calls per run.
• Number and type of tool calls.
• Retry count.
• Human review minutes for tasks that require approval or correction.
• Cost per successful task and cost per failed task.

OpenAI’s cost optimization guidance notes that cost and latency are often connected: reducing requests and tokens can reduce both, and smaller models can reduce cost and latency when they maintain sufficient quality. Source: OpenAI cost optimization guide.

4. Latency

Latency should be reported as percentiles, not only averages. A median response time can look fine while the p95 experience is unusable. For agents, split latency into model time, tool time, queue time, human approval time, and post-processing time.

OpenAI’s latency guide groups latency improvements into principles such as generating fewer tokens, using fewer input tokens, making fewer requests, parallelizing when possible, and avoiding LLM calls when a deterministic method is better. Source: OpenAI latency optimization guide.

For an agent dashboard, track:

• p50, p90, and p95 end-to-end latency.
• Time to first useful output when streaming is used.
• Time spent waiting on external tools.
• Retry-induced delay.
• Latency per successful task, not just per request.

Use traces before you build large eval datasets

Do not start by writing 1,000 synthetic test cases. Start with 20 to 50 realistic tasks and inspect the traces. You are looking for failure modes: wrong tool, missing retrieval, hallucinated action, unsafe escalation, over-broad tool argument, weak citation, or needless multi-step loop.

A useful trace should show:

• The user request and relevant context.
• Every model call and tool call.
• Tool inputs and outputs, with sensitive values redacted.
• Guardrail decisions and approval gates.
• Token usage and latency per step.
• The final answer or final action.

OpenTelemetry has moved its GenAI semantic conventions to a dedicated repository, and its public docs still point to GenAI spans, metrics, events, and agent spans. Source: OpenTelemetry GenAI semantic conventions. Even if you do not adopt a full standard immediately, use stable names for the same concepts across logs, traces, and dashboards.

A practical evaluation workflow

1. Define the task contract. What counts as a complete task? What actions are forbidden? When must the agent ask for help?
2. Collect real examples. Use anonymized support tickets, internal workflows, coding tasks, research tasks, or synthetic cases based on real patterns.
3. Create a small golden dataset. Include normal cases, edge cases, adversarial cases, and “should refuse or escalate” cases.
4. Instrument traces. Capture tool calls, guardrails, retries, token usage, and latency before you tune prompts.
5. Grade with layered evaluators. Use deterministic assertions where possible, human review for high-risk cases, and LLM judges only with clear rubrics and spot checks.
6. Compare variants. Compare model, prompt, retrieval, routing, and tool-permission changes against the same dataset.
7. Move online carefully. After launch, sample real traces, collect user feedback, and add new failures back into the offline suite.

Example scorecard

Common mistakes

• Using only happy-path examples. The agent looks reliable until a user asks for a forbidden action or a retrieved document contains hostile instructions.
• Scoring only the final answer. The answer may look right while the agent used the wrong tool or exposed unnecessary data.
• Ignoring failed-task cost. Failed and retried tasks often reveal the real economics of an agent.
• Reporting average latency only. Users experience the slow tail, not the mean.
• Treating LLM-as-judge as ground truth. Judges are useful, but they need rubrics, calibration, and human audits.

Minimum production checklist

• At least 50 realistic offline cases before production use.
• Separate safety/adversarial cases, not mixed invisibly into the average score.
• Trace capture for model calls, tool calls, guardrails, retries, cost, and latency.
• p50/p90/p95 latency dashboard.
• Cost per successful task dashboard.
• Human review for high-risk failures and a process for adding those failures back into the dataset.
• A release gate: no prompt, model, retrieval, or tool-permission change ships without a before/after eval comparison.

Bottom line

An AI agent evaluation program should answer a practical release question: “Is this agent better, safer, cheaper, and fast enough for the tasks we will actually give it?” If your eval cannot answer that, it is probably measuring the wrong thing. Start with traces, turn real failures into datasets, keep safety separate from average task success, and report cost and latency per successful task.