Enterprises are racing to give AI agents real autonomy. The ones that succeed in the physical world will be the ones that solve trust first.
Most agentic AI still works inside a screen. It drafts emails, routes tickets, reconciles invoices, and answers questions. When it gets something wrong, the cost is a bad sentence or a rerun. That is changing quickly. Agents are starting to act in the physical world, in delivery, robotics, field service, and logistics, where a wrong decision becomes a wrong action with a physical price tag.
I build agentic systems for last-mile logistics, and I have become convinced that the physical world, not the browser tab, is where agentic AI will either prove itself or quietly fall apart. It is also where most of today’s enterprise ambition is about to be tested. Gartner has predicted that more than 40 percent of agentic AI projects will be canceled by 2027, and McKinsey’s recent research found that fewer than one in ten organizations has actually scaled AI agents in any function. The gap between a pilot and a production system is wide, and in the physical world it is widest of all. The reason is not capability. It is trust.
A wrong answer becomes a wrong action
Consider the last fifteen meters of a delivery, the walk from the curb to the front door. A human delivery driver handles it without thinking. They step over a low wall, read a confusing entrance at a glance, and find the door. An autonomous delivery robot cannot improvise. It can only move where it has accurate, complete, and valid path data, and it has to know about the steps, curbs, and narrow gaps a person would simply walk around.
This is already real. Companies like Starship and Serve Robotics run sidewalk delivery robots in actual neighborhoods, Nuro operates autonomous delivery vehicles on public roads, and drone programs like Zipline and Alphabet’s Wing navigate to homes where the difference between the backyard and the neighbor’s backyard is a few meters of geospatial accuracy. Autonomous vehicle programs such as Waymo depend on map data being correct before a vehicle moves at all. For any of these systems, two business questions come first. Can the agent be deployed to a given location at all, which is a coverage problem. And will it act correctly once it is there, which is an accuracy problem. Both are the same question wearing different clothes. Can you trust what the system believes about the world.
The real bottleneck is trust, not autonomy
The instinct in a lot of boardrooms is to ask for more autonomy, faster. In my experience that is the wrong order. The harder and more valuable question is whether the agent’s view of the world can be trusted enough to act on, and trust is the thing teams keep trying to bolt on at the end instead of building in from the start. This is the same data accuracy concern that shows up in nearly every enterprise AI survey, and it gets sharper in the physical world, because the agent is reasoning about messy reality from imperfect inputs.
Large language models make the problem concrete. Ask a model to describe a physical scene from an image and it will answer with complete confidence, including when it is wrong. It will call the roof of a parked car a sidewalk. It will produce a location that looks right and is off by enough to matter. A model trained to generate fluent output generates fluent output, and fluent is not the same as correct. For an enterprise putting an agent in front of a customer’s door or a moving machine, that gap is the entire risk.
The encouraging part is that trust here is an engineering and governance discipline rather than a leap of faith. Having built these systems, I would point leaders at four practices that separate an agent you can deploy from a demo you cannot.
Keep the model out of the highest-stakes outputs. A language model is very good at reasoning over evidence and choosing among options. It is not a measuring instrument. Let deterministic systems own the exact values that steer a machine, and let the model do the judgment. The moment a model is allowed to freehand a number that controls a physical action, the guarantee you most need is gone.
Verify every claim against ground truth. Do not trust the model’s assertion that something is true. Check it with something that is not the model. If an agent claims a surface is walkable, a separate system should confirm that against the actual data before anyone acts on it. The reasoning is the proposal. Independent verification is the truth. An agent that grades its own homework is not one you can stand behind in a regulated or safety relevant setting.
Treat any change to shared data as do no harm. Most agents answer a question and stop. A physical world agent usually changes something that other systems already rely on. The bar shifts from “is this change good” to “is this change guaranteed not to make anything worse.” A commit gate that refuses to degrade what already works, even when it means doing less, is a feature and not a limitation.
Design for abstention and human handoff. In the physical world, a missed action is an inconvenience while a wrong action can be a safety event. The system should be biased toward stopping and asking a person when it is unsure, instead of acting confidently on a guess. Most agent evaluation rewards answering. Physical world evaluation should reward knowing when not to.
Why this is an advantage rather than a tax
It is tempting to read all of this as friction that slows you down. The reverse is true. The organizations that build this discipline are the ones that can move agents out of the pilot phase and into the field, while the ones chasing raw autonomy stall in exactly the project cancellations the analysts keep warning about. It helps that the raw materials are increasingly open. Shared map data and open routing tools mean almost any capable team can stand up a physical-world agent today, which is precisely why the model is no longer the moat. In a market where the models themselves are becoming interchangeable, the durable advantage is not the model. It is the quality of your data and the strength of the verification around it. That is what lets you deploy where the stakes are real.
If I were advising a team starting in this space today, my takeaway would be simple. The frontier is not the most autonomous agent. It is the agent you can trust in places where mistakes are physical. Invest in grounding, independent verification, conservative change control, and graceful human handoff, and you can put agents to work in parts of the business your competitors cannot yet reach. Chase autonomy without those, and you are likely to join the projects that get quietly shut down.
The agents that matter next will not be the most autonomous ones. They will be the ones that are right about the world, and that know when they are not.
Vishal Kumar is a Tech Leader at Amazon Maps, where he leads the geospatial data pipelines behind last-mile delivery. He built the infrastructure that powers the visual map tiles and routing artifacts that millions of Amazon drivers depend on every day. He also leads ML and Agent Farm-based learning systems grounded in sensor data and ground-truth aggregation, and he drives the adoption of OpenStreetMap-derived datasets, Overture Maps, and open-source GIS tooling across Amazon Maps.]
