The demos are impressive. An agent is shown a goal, the browser opens, and within seconds it is clicking through a multi-step workflow with apparent ease. Then the same system is deployed in a production environment and the success rate collapses. Developers refine the prompt, try again, and see marginal improvement before hitting a wall they cannot explain.
I have seen this pattern across agent projects of every scale. The diagnosis applied is almost always the same: the model needs better prompting. The actual cause is almost always different: the system was never designed to handle the structural chaos of a live web environment. Treating these as the same problem is why so many agentic web automation initiatives fail between proof-of-concept and production.
Gartner predicted that over 40% of agentic AI projects would be cancelled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. In web automation specifically, that risk control failure is structural, and it lives at the interface between the model and the browser.
Why Benchmarks Mislead Builders
Academic benchmarks for web agents report headline numbers that sound production-ready. The reality on live websites is considerably harder. Research published in the Online-Mind2Web evaluation tested six frontier agents against 300 tasks across 136 live websites and found that most commercially available systems underperformed a baseline agent released in early 2024. Even the strongest system reached only a 61% success rate on live tasks. That figure would disqualify any software from production in almost any industry.
The gap between benchmark performance and real-world operation is not a measurement artefact. Static benchmarks test agents against controlled, predictable page states. Production web environments are neither. A DOM tree that renders cleanly in one session may be entirely restructured in the next because a developer pushed a front-end change overnight. When agents encounter these conditions they do not fail gracefully: they hallucinate, inferring an action from an outdated context representation and executing it with full confidence. The underlying model has no way to know its picture of the page is wrong, because no component of the standard architecture ever told it.
Three Compounding Failures at the Browser Interface
The current dominant architecture for web agents places the LLM directly in front of the raw browser environment. The model receives a DOM snapshot or screenshot, reasons about what to do next, and issues an action. That design creates three compounding failure conditions.
The first is representational noise. A modern web page’s DOM is not written for machine consumption. It is a layered accumulation of framework-generated markup, third-party scripts, and accessibility attributes. Feeding this structure to a language model is the equivalent of asking a logistics planner to find a route using raw survey sensor files: the information is technically present, but practically unusable without transformation.
The second is action space ambiguity: without a constrained set of valid actions for the current page state, the model must infer them from the full DOM, which introduces variance across runs. The third is context loss across steps. Multi-step tasks require coherent state management throughout the workflow, yet standard implementations begin each step with a fresh inference from a fresh snapshot. Production experience with browser agents confirms that context management is the primary determinant of task success, yet it is the component most implementations leave to the model to manage on its own.
When the Same Task Succeeds and Fails
During the development of an open-source browser action engine I led from its initial architecture through public release, the team ran a controlled test that made the problem concrete. We were automating a multi-step form submission workflow: the kind of task enterprise teams regularly attempt. In a controlled environment with a fixed page version the task completed reliably. We then ran it against the live production site over two weeks.
The failure modes were instructive. On days when the site had no front-end changes the success rate held. On days when the vendor updated their component library, or when a new promotional modal appeared, or when an A/B test variant was active, the agent would complete the early steps confidently, reach a point where its DOM expectation did not match the current page state, and then either take a wrong action or stall.
The prompt was identical across every run. The model was identical. Only the environment had changed. Refining the prompt did not help because the prompt was not the source of variance. Every attempt to describe the page state in better natural language simply moved the problem one level up: the model would produce more fluent descriptions of the wrong thing.
What a Purpose-Built Interface Layer Provides
The structural fix is the introduction of a purpose-built layer between the LLM and the browser. It needs to do three things the raw architecture cannot.
Before the model sees any page representation, it converts the raw DOM into a structured, machine-readable context: irrelevant elements removed, interactive elements indexed, the model given a clean and consistent view of what it can act on. This is not summarisation; it is normalisation. The model’s task changes from parsing an environment to reasoning about a defined action space.
The interface layer then constrains available actions to only those that are valid for the current page state, shifting execution from probabilistic approximation toward deterministic operation. Finally, it maintains a structured record of task state across every step, so that when the environment changes unexpectedly the agent resumes from a known position.
These three properties together shift the failure mode from silent hallucination to handled exception. You cannot diagnose a failure you cannot observe, and you cannot observe a failure the system never registered as one.
Why Prompt Engineering Cannot Close This Gap
There is a reason the industry defaults to prompt engineering as its first response to agent unreliability: it is fast, cheap, and sometimes works. A well-crafted prompt can reduce a specific failure mode on a specific task on a specific page. This creates a misleading signal. The improvement is real; the diagnosis it implies is wrong.
Every gain from better prompting is conditional on the page remaining stable. When the page changes, and it always does, the gain evaporates. Teams that have spent weeks optimising prompts for a target site often find that a single front-end deployment by the vendor undoes all of it. Scaling this approach is not tractable: prompts that work for one site require substantial rework for the next, and every workflow spanning multiple applications multiplies that cost.
The teams deploying web agents at scale have understood this. They are building infrastructure: standardised interfaces, constrained action sets, persistent context management. The model becomes one component of a system designed for reliability, not the single point of accountability for a task the architecture makes nearly impossible.
The Right Question to Ask
For practitioners building or evaluating agentic web automation, the practical question is not which model performs best on the benchmark leaderboard. It is whether the architecture between the model and the browser has been designed for production conditions: non-deterministic environments, changing page states, multi-step context, and the requirement for auditable execution.
The question worth asking of any agentic web automation system is not what it does when everything works. The more revealing question is what it does when the environment changes. If the answer is that it fails unpredictably, the problem is not the model. The problem is the layer that was never built.
