What does a flawless AI-agent demo prove? Only that the agent completed one task under one set of conditions.
It says little about what happens when a tool fails, an interface changes, or the same task must succeed repeatedly. That matters when agents can update records, call APIs, navigate software, and interact with users.
Production readiness depends on evidence about the result, the path taken, and the conditions under which the system fails.
What Trust Looks Like in Practice
A correct result does not establish reliability on its own. Teams should be able to show that the task was completed, the agent’s actions can be reconstructed, and failures will be contained.
Controls may include restricting access to sensitive tools, requiring approval before irreversible actions, limiting retries, or routing uncertain cases to a person. The execution trace is central. It shows which tools the agent called, what information came back, and how the environment changed.
I saw this while co-building a production agent framework. Some orchestration frameworks made it easy to connect models and tools but offered limited visibility into the agent’s internal loop. A run could appear successful until the trace revealed repeated calls, abandoned plans, or a fragile route to the correct result.
That is why agent observability belongs inside the evaluation system.
Last year, when I took Agentic AI (CS294) at UC Berkeley, I also built an evaluation framework for TheAgentCompany benchmark. It combined deterministic checks, trajectory heuristics, and model-based judging because each method exposed different failures. That layered approach is the basis of effective AI agent evaluation.
Layer One: Prove the Task Was Completed
The first layer should use deterministic checks wherever possible. If an agent says it scheduled a meeting, check the calendar. If it updates a record, compare the result with the expected state. If it writes code, run the tests.
These checks are repeatable, inexpensive, and clear. Their weakness is scope. An agent may reach the correct final state after attempting the same write several times. The outcome check passes, but the trace reveals behaviour that could create duplicate transactions elsewhere.
Deterministic checks confirm the result. They do not show whether the execution was efficient, coherent, or repeatable.
Layer Two: Examine How the Agent Got There
The second layer evaluates the trajectory. Useful measures include step count, latency, token use, cost, failed tool calls, repeated actions, and retries.
Teams can also assess coherence. Does each action follow from the latest observation? Does the agent retain the information it needs? When a plan fails, does it adjust or repeat the same mistake?
These signals can expose degradation before completion rates fall. Two agent versions may have the same pass rate, while one uses more tool calls and takes longer to recover from errors. The result is unchanged. The system is slower, more expensive, and less predictable.
Trends across many runs matter more than a fixed threshold. Rising retries, costs, or tool failures can reveal a regression while deterministic checks still pass.
Layer Three: Use Model Judges Carefully
Some questions are difficult to answer with fixed rules. Did the agent follow a detailed instruction? Was its response coherent? Did it stay within the task constraints?
The G-Eval paper describes a model-based method that applies defined criteria and structured scoring steps. Frameworks such as DeepEval make similar methods easier to add to test suites.
For an agent system, a model evaluator can review the task, constraints, tool trajectory, and final output. It can then score instruction-following, policy compliance, or recovery behaviour. This is where LLM-as-judge methods can help. Their scores remain estimates.
Results may change with the rubric, prompt structure, model version, or order of the material. Judges may favour certain styles, produce unstable scores, or miss an operational failure that a direct check would catch.
Teams should use precise criteria, compare scores with human-labelled examples, test known failures, and recalibrate when the judge or prompt changes.
Direct checks should take priority when the relevant state can be inspected.
Multimodal Agents Need Wider Coverage
Agents that work through screens, audio, or video can fail at perception as well as reasoning. A browser agent may click the wrong control. A voice agent may miss a correction. A screen-based agent may complete the task while exposing unrelated information.
Multimodal evaluation should extend the earlier layers. Deterministic checks can confirm the final interface state. Heuristics can measure latency or click efficiency. Model-based evaluators can assess conversational quality or visual grounding.
Combine Evidence Without Hiding Risk
Each method fails differently. Deterministic checks are precise but narrow. Heuristics are scalable but approximate. Model judges can assess context, but their scores may vary.
A single average can hide important failures. A stronger system separates results into:
- Hard gates: failed outcome checks, prohibited actions or policy breaches
- Weighted indicators: cost, latency, efficiency and trajectory quality
- Review triggers: evaluator disagreement, low-confidence scores or unfamiliar behaviour
A direct state comparison should carry more confidence than a subjective coherence score. Some findings should remain outside any average. If the final state is correct but the agent took an unauthorized action, the run should fail. Disagreement between layers is useful evidence.
What Agent Benchmarks Really Show
Public agent benchmarks provide shared tasks and repeatable environments for comparison. TheAgentCompany evaluates agents on professional tasks inside a simulated software company. Its results show that current systems still complete only a minority of these tasks autonomously. τ-bench evaluates interactions between agents, users, and tools. Its pass^k metric measures whether an agent succeeds across repeated trials.
A single successful run shows capability. Repeated success provides stronger evidence of reliability. Benchmarks cannot certify trustworthy AI agents for a specific organisation.
Production environments have their own tools, policies, users and risks. Teams should use benchmarks as a reference point, then build evaluation sets around the real deployment environment.
Why Evaluation Must Continue After Launch
Agent evals should function like regression tests in continuous integration. A change to a prompt, model, tool definition, or orchestration loop can improve one workflow while weakening another.
High-value scenarios, previous failures, and difficult cases should run whenever the system changes. Production failures should become regression tests. New user behaviour should inform new test cases. Tool calls should remain traceable so failures can be reproduced.
This matters because production AI agents operate in changing environments. Interfaces are updated, policies shift, and users introduce requests that were absent from the original test set. A static suite becomes less representative over time.
A Practical Checklist for Production Readiness
Before trusting an agent with important work, teams should be able to answer five questions:
- Can task completion be verified against the environment?
- Can every action, retry, and failure be reconstructed from the trace?
- Are outcome quality, trajectory quality, cost, and policy compliance measured separately?
- Have model-based evaluators been calibrated against human judgement?
- Do realistic evaluations run whenever the system changes?
Trust Comes From Evidence
A successful demo shows that an agent can complete a task once. Production readiness requires evidence across repeated runs, changing conditions, and known failure cases.
This is the basis of agent reliability. Teams need to know what the agent did, how it did it, and when it should stop acting.
About the Author
Felipe Duarte is a software engineer working on applied AI. He’s currently at Sable, a startup building an AI “employee” that can see, talk, and navigate software to run customer calls at scale, where he works remotely and helps build much of the company’s core infrastructure. Previously, he led the AI voice work behind Tako’s AI recruiter, a system that conducts full interviews over video. Duarte writes about applied AI and the realities of building AI products at an early-stage startup. Connect with him on LinkedIn.
