Industry insiders have long pointed to browser agents as the next big consumer channel, following on the heels of the phone, web, and mobile revolutions.
The vision is that autonomous web agents, driven by LLMs, will automate complex workflows like purchasing airline tickets. And early efforts have been impressive in some ways.
After all, today’s browser agents can interpret natural-language instructions, perceive dynamic web environments through structural inputs (like a web page’s DOM) and visuals (screenshots), reason about sequences of actions, and execute interactions with user-like behaviors.
Unfortunately, they don’t work.
At least not well enough. And it’s not their fault.
The problem is that the web is a wild place that’s not designed for machines.
So browser agents are forced into expensive guessing games. Completing a task requires a slow, expensive, and unreliable loop of analyzing visual and structural data, making an LLM call to decide on the next micro-action (e.g., “click the button with the text ‘Submit'”), and repeating this process for every single step, every single time a task is performed.
Accessibility to the rescue
A solution may be emerging from an unexpected corner of the internet: digital accessibility.
That’s because accessibility practitioners have quite a lot of experience working with machines. In effect, the screen readers and voice control software used by many human users depend on just the sort of structure that browser agents need.
Accessible code provides a machine-readable layer of semantics, relationships, and states that describes not just what an element looks like, but what it is and how it behaves.
By fully tapping into this semantic layer, a website can be transformed from an unpredictable visual interface into a predictable, machine-operable one.
- Semantic Description: Natural language descriptions (e.g., accessible labels) allow agents to understand an element’s purpose.
- Predictable Behavior: Accessibility standards define how components like dialogs, menus, and forms should function, reducing ambiguity.
- Stable Identifiers: Roles and labels provide consistent, stable hooks for elements, unlike brittle, auto-generated selectors that can change between sessions.
This allows an agent to move beyond simple “click” and “type” commands to more abstract, deterministic operations.
But more importantly, it also allows a browser agent to learn and generalize from its actions.
A new approach
To prove this idea, we built and tested a “semantic browser agent” that directly leverages accessibility traits. It does this in two ways:
1. Pre-Defined Skills. The semantic agent utilizes a built-in library of pre-defined skills to operate common UI patterns, like dropdown menus, pop-ups, and form fields. By semantically understanding these patterns, the agent can operate them deterministically.
For example, the semantic agent knows how to select an item in even a custom-built dropdown menu, and it does so as a single, atomic operation.
All this without relying on a visual inspection (as a sighted human would) or a roundtrip with an LLM to get advice on how to proceed.
2. Learned Skills. Upon successfully completing a task, the agent extracts the sequence of high-level actions using stable semantic identifiers (e.g., role, accessible name) and caches this as a “learned skill.”
For example, searching for a flight on united.com might initially require ten LLM-driven steps. Our agent condenses this into a single learned skill, find_flights (origin, destination, departure_date, return_date).
On subsequent runs, the agent recognizes the user’s intent, retrieves this skill, and simply asks the LLM to fill in the parameters from the new prompt.
This transforms a multi-step, LLM-intensive process into a single, efficient execution, drastically reducing cost and improving speed.
Testing the semantic agent
To ensure our evaluation was grounded in real-world scenarios, we used the Online-Mind2Web benchmark. It consists of 300 diverse and realistic tasks spanning 136 live websites, making it an ideal testbed for generalist web agents operating “in the wild.”
The benchmark tests a wide range of common user actions, such as navigating e-commerce sites, filling out forms, and interacting with complex web applications.
We tested a reference browser agent versus our semantic agent. For a baseline, we chose Browser-Use, a leading open-source framework that represents the state-of-the-art in agents that rely on visual and DOM-based reasoning.
Each of the 300 tasks was performed five times by both agents. For our agent, each successful run was used to generate a “learned skill,” which was then used in a subsequent run to measure the full efficiency gain.
The results showed overwhelmingly positive results, both in speed and cost.
Performance improvement
Here, we observed the total LLM utilization time (in seconds) to complete each task. With almost no exceptions, the semantic agent is faster to complete tasks (and finish roundtripping to the LLM) than the more generic browser agent.
Across all sites, we observed a 5.4X average improvement in performance.
Some sites, particularly those with HTML that was already executed with correct semantics, saw even larger gains.
Cost improvement
For cost, we calculated the token usage for each task rather than the dollar cost, as token costs vary from model to model
Across all sites, we observed an average cost improvement of 3.2X.
As with the speed results above, sites with HTML that were more accessible experienced even larger cost improvements.
The path forward
The demonstrated improvements available from this approach are not small. A semantically-savvy browser agent like ours could be the bridge between the current, somewhat impractical world of browser agents, and the world that technologists have been hoping for, where agents take care of many of the chores we currently slog through on the internet.
Plus, our research suggests that these benefits, large as they are, will double for sites that score well on standard accessibility checks. In a world of agentic coding, it’s become dramatically easier, verging on automatic, to make websites accessible.
So it’s a perfect storm. And the amazing double whammy will be that by making websites more accessible to machines, we’ll be making them more accessible to humans, too.
