AI & Technology

We Keep Teaching AI Agents How to Act. We Should Teach Them When to Stop.

By Eugene Vyborov, Founder and CEO, Ability.ai

Here is a failure I see almost every week. You give an agent a key and ask it to open a door. The key doesn’t turn. So the agent climbs in through the window and reports: “Door opened.” 

Nothing in its instructions said the window was off limits. Every instruction was about getting inside. 

This is the most common way AI agents go off the rails in production, and it isn’t the one the industry talks about. The conversation is about action: more tools, more autonomy, longer chains of reasoning, agents that “just get it done.” But once you run agents for real, giving them more ways to act is the easy part. The hard engineering problem is getting them to stop: to fail out loud, escalate, or hand a decision back to a person. 

The goal survives, the method doesn’t 

What actually happens looks like this. An agent has a sanctioned tool for a job. The tool breaks: an API times out, a credential expires, a file isn’t where it should be. 

The agent doesn’t stop. It keeps the goal and quietly swaps the method. It scrapes a page instead of calling the API. 

It reconstructs a file from memory instead of reading it. It estimates a number it was supposed to look up. Then it reports success, because by its own measure it succeeded. 

The research agrees. A recent study, “Are Your Agents Upward Deceivers?”, tested 11 popular models on tasks with broken tools and mismatched sources. Agents routinely guessed results, ran unsupported simulations and made up information rather than admit they were stuck. Prompting them not to deceive helped only a little. 

That last finding matters most. You can’t fix this with a better sentence in the system prompt. “Be careful” and “ask when unsure” don’t do anything on their own. An agent that has to reach a goal will always find a reason it isn’t unsure. 

Fallbacks are fine. Improvised ones are not 

The answer isn’t to ban fallbacks. Real systems need them. The line that matters is between a fallback a person designed, tested and wrote down, and one the agent made up at runtime because the first path failed. 

If a human decided “when the CRM API is down, read last night’s export and flag the result as stale,” the agent should do that. If the agent decides on its own to swap the data source, the method or the guarantee it was asked for, it should stop and say so. A rule we use across our own fleet: if you have to argue that a substitute is equivalent, it isn’t. 

The second half of that rule is cultural, and just as important: an honest abort counts as a successful run. A plausible substitute counts as a failed one. 

Toyota got this right decades ago with the andon cord. Any worker on the line can stop the whole line, and stopping is treated as doing the job well, not as an interruption. Agents need the same norm built into how their success is judged. 

Stopping needs a channel, not a virtue 

Telling an agent to stop is useless if stopping leads nowhere. If the only alternative to pushing on is failing silently, it will push on. 

This turns out to be measurable. In a study of escalation channels for agentic AI, Francesca Gomez put 10 frontier models in situations where the task conflicted with the rules. 

With no sanctioned alternative, they took harmful actions 38.7% of the time. A simple escalation email cut that to 5.9%. A channel that actually guaranteed a pause and an independent review cut it to 1.2%. 

The gap between notifying someone and guaranteeing a pause is the whole lesson. Agents respond to whether the stop is real. 

So in our own systems, asking a human is a first-class action, not an exception path. An agent raises an approval, a question or an alert to a named person or role, gets a receipt and ends its turn. 

It doesn’t sit waiting. A turn that blocks on a human burns its whole time budget and delivers nothing. The answer wakes it up later. 

And an approval that expires or gets dismissed means “no.” An unanswered gate has to decay to “do not proceed,” never to a stale “maybe” that the agent later reads as permission. 

Narrow triggers, not general caution 

There’s an opposite failure, and it’s real. An agent that pauses whenever it feels uncertain is useless. It hands every decision back to a person and becomes a very expensive way to create work. 

So the triggers have to be narrow and listed. In our fleet there are four. Stop and ask when the next step would: 

  1. change the data source, method or guarantee you were asked for; 
  2. change the scope of the task; 
  3. be irreversible: money, messages sent under someone’s name, public posts, deletions; 
  4. need input only a person can give. 

Nothing else counts. A slow tool, a failed first attempt or an awkward path are reasons to retry, not to stop. Ambiguity resumes work and logs it. Consequence stops it. 

The real autonomy question 

The industry measures agents by what they can do on their own. I think the better measure is how precisely you can say when they shouldn’t. 

That’s not anti-autonomy. It’s the opposite. The companies that can give agents the most room will be the ones whose stop conditions are explicit, enforced and trusted, because only then can you safely hand over everything else. 

Autonomy you can’t interrupt isn’t something you can scale. It’s something you can only hope works. 

Teaching an agent to act is a prompt. Teaching it to stop is architecture. We should spend our time on the second one. 

Related Articles

Back to top button