There is a quiet assumption baked into almost every AI product shipping today: agents are going to make mistakes.
We say it in launch posts and disclaimers. We say it to customers when a support agent invents an order number, or a coding agent reintroduces a bug it fixed last week. The industry has settled on a shrug: models are probabilistic, so failure is the cost of doing business.
I think that assumption is wrong, or at least badly incomplete.
AI is often described as the next computer. But we never accepted computers that failed the same way over and over. When a program crashed, we found the bug and fixed it, and it stayed fixed. Somewhere along the way, we stopped expecting that from agents. A failure happens, a user gets frustrated and leaves, and the next day the same agent makes the same mistake for someone else.
Agents will occasionally get things wrong. That part is real. But an agent making the same mistake twice isn’t a property of the model. It’s a property of the infrastructure around it. And right now, most of that infrastructure was never built for agents.
Software was predictable. Agents aren’t.
The tools enterprises use to keep software reliable were designed for deterministic systems. The same input followed the same path and produced the same output. When something broke, an engineer read the stack trace, found the failing line and shipped a patch. Monitoring meant watching for errors, latency spikes and crashes, all of which are loud, well defined and easy to count.
Agents break every part of that model.
An agent doesn’t execute a fixed workflow. It makes a chain of decisions: what to retrieve, which tool to call, how to read the result, what to say next. The same request can take a different path every time, depending on what’s in context, what the retrieval step returned, what a tool responded with, and what the model decided along the way.
The failures change shape too. The most damaging agent failures don’t throw errors. The API call succeeds. The response comes back in 800 milliseconds. Every dashboard is green. And the agent has just told a customer something that isn’t true.
Traditional monitoring has no way to see that. It was built to catch systems that stop working, not systems that keep working while being wrong.
The million-log problem
Ironically, agents produce more telemetry than any software before them. Every run leaves behind a full trace: the prompt, the retrieved context, each tool call and its response, the model’s intermediate steps, and the final answer. On paper, teams have never had more visibility into what their systems are doing.
In practice, almost nobody looks.
The first wave of agent tooling solved storage. It captures traces and lets you search them. That’s necessary, but a trace store is a database, not an answer. When an agent handles a hundred thousand conversations a week, no engineer is going to read a million logs to figure out why a user got angry and left. The traces that explain the failure are in there somewhere, buried among the ones that went fine.
So teams fall back on what they can see: a complaint in a support queue, a thumbs-down, a screenshot shared in Slack. They fix that one case, and they have no idea whether it was a one-off or something hitting thousands of users every day.
The gap isn’t data. It’s analysis: turning raw traces into a clear picture of what is going wrong, how often, for whom, and why.
What a recurring failure actually looks like
Take a customer support agent with an order-lookup tool. A customer asks where their package is. The lookup comes back empty because the order number was mistyped. Instead of saying it couldn’t find the order, the agent pulls details from a similar-looking record and confidently quotes a tracking number and delivery date that belong to someone else.
Nothing errored. The tool call succeeded, it just returned nothing useful. The response was fluent and fast. From the infrastructure’s point of view, it was a perfect interaction.
Now multiply it. The same pattern shows up when a refund lookup misses and the agent confirms a total with no matching invoice, or when a store search fails and the agent lists opening hours for a location that doesn’t exist. Look at these one at a time and they seem like scattered hallucinations. Look at them together and they’re one problem with one cause: the agent fills in gaps after an empty tool result instead of admitting it found nothing. And it affects every user who ever mistypes an order number.
The subtler failures are often the more expensive ones. An agent that tells a customer in Berlin the store opens at 9:00, in Pacific time. A coding agent that keeps treating a harmless “already exists” response as a failure and retries until it times out. A tool call that works in testing but times out under real load. None of these are model problems you can wait for a smarter model to fix. They live in the harness: the prompts, tools, retrieval, memory and logic a team builds around the model.
From observability to harness engineering
If the problem lives in the harness, the fix does too. That’s the shift I think the industry needs to make: from observing agents to engineering them, using production as the source of truth.
Harness engineering treats every production trace as evidence about how the system really behaves, not as a log to archive. In practice, it’s a loop with four steps.
- Find recurring failures and measure their impact. Read every trace, not a sample. Group failures that share a cause into a single problem, and rank problems by how many users they actually affect. A bug hitting 6,000 users matters more than one hitting 200, but you can only know that if you’ve grouped them.
- Understand the cause. Open a problem and see the traces behind it, along with related problems that share the same root. Often what looks like five separate issues is one bad prompt instruction or one tool returning an ambiguous result.
- Test the fix against production before it ships. Replay a prompt change, a tool fix or a model swap against the exact traces where the failure occurred. Compare pass rate, latency and token cost against the current version. A fix that solves one problem and quietly breaks another shows up here, not in front of customers.
- Keep watching. Turn the problem into a standing signal and monitor it. If the failure creeps back after the next model update or prompt edit, the team hears about it within the hour, not from a churned customer.
Each step makes the next one sharper. Every fix produces new traces, and those traces become the evidence for the next round of improvements. That’s what it means for an agent to not make the same mistake twice. It isn’t a promise that the model is perfect. It’s that the system around it learns.
The harness is the moat
It’s tempting to treat all of this as hygiene, a safety net you install and forget. I think that undersells it.
For a growing number of companies, the agent is the product. Everyone has access to the same frontier models through the same APIs. What separates one agent from another is the harness: the prompts, the tools, the way context is assembled, the edge cases the team has learned to handle. That’s where a company’s knowledge of its users and its domain actually lives.
A team that can see exactly where its agent fails, fix it with confidence, and prove the fix worked will compound that knowledge every week. A team that learns about failures from angry customers will keep relearning the same lessons. Over a year, those two teams end up in very different places, even if they started on the same model.
That’s why I see harness engineering as empowerment, not patching. The goal isn’t to put a wall around a fragile system. It’s to give engineering teams control over their agents again, so they can move faster and push their product past what the base model can do on its own. If you care about your product and the people using it, you shouldn’t be content to watch the same failure happen twice.
Where this goes next
The loop I’ve described starts with prompts, tools and evals. It doesn’t end there.
Once you can break production traces into the tasks users are actually trying to accomplish, and know which of those tasks your agent succeeds or fails at, you have something more valuable than a bug list. You have a precise map of what your agent needs to get better at, grounded in real usage rather than synthetic benchmarks.
That map is exactly what you need to go further down the stack. Open-weights models have reached the point where many teams don’t need more general intelligence. They need specific intelligence: a model that is excellent at their tasks, their tools and their users. The same segmented tasks that drive harness fixes can become evaluation suites, training environments and fine-tuning data. Improvement stops being a quarterly project and becomes continuous.
This is where I believe agent infrastructure is heading. Observability, evaluation and training won’t stay separate tools owned by separate teams. They’ll become one loop: production data comes in, failures get found and understood, fixes get verified, and the agent, whether its harness or eventually its model, gets better with every cycle.
The standard we should hold
The AI industry still measures progress mostly by model capability. Inside companies running agents in production, the more important question is simpler: when something goes wrong, does it stay fixed?
We would never accept a computer that crashed the same way every day. We shouldn’t accept it from agents either. The models will keep getting better on their own. Making sure our agents never make the same mistake twice is on us.
Pranav Bedi is the co-founder of Moda, which helps teams running agents in production turn their traces into verified improvements. Moda is backed by Y Combinator and based in San Francisco.
