When news recently broke of OpenAI agents going rogue, it wasn’t the first such incident, but it was arguably the most alarming. During a cybersecurity evaluation, the agents exploited a zero-day vulnerability to escape their testing environment before compromising the infrastructure of Hugging Face, one of the world’s largest platforms for sharing AI models, to complete their assigned task.
Like something out of a sci-fi novel, the agents “spent a substantial amount of inference compute finding a way to obtain open Internet access.” If that wasn’t remarkable enough, once online, they targeted Hugging Face by combining stolen credentials with further zero-day vulnerabilities to gain access to its production systems. Far from being the only victim, it was then revealed that the rogue AI “found four logins online which allowed it to access four separate, unnamed services” in addition to Hugging Face.
As OpenAI went on to explain in its incident report, the agents were so focused on completing the task they were given that they went to extraordinary lengths to circumvent the controls designed to contain them. In other words, they pursued their objective in ways their developers had neither intended nor anticipated.
Clearly, the incident has enormous implications. OpenAI described this behaviour as “something we expect to become more commonplace with the proliferation of increasingly cyber-capable models”, but another recent case strongly suggests the threat is already taking shape. In April, US startup PocketOS faced an existential crisis when an AI agent operating in a Claude development environment autonomously deleted the company’s entire production database and its backups in just nine seconds.
The company was able to recover, but it took sixty hours and reportedly cost “$15.6M+ in losses across the sector.”
An unprecedented resilience problem
Among the many questions these incidents raise is how organisations can protect themselves against the risks. Behind the “rogue agent” headlines, what has been revealed is a very serious resilience problem, and one that is quite different from the security and disaster recovery scenarios organisations have traditionally planned for.
The first thing that urgently needs to change is how agents are viewed from a security perspective. They have already gone way beyond their role as mere productivity tools and, as we have seen, are now more analogous to operational actors with delegated authority. From a security and resilience perspective, therefore, access rights should be more closely aligned with specific tasks and regularly reviewed, based on the principle of least privilege. Limiting permissions can help contain the scope rogue agents have to operate and, by definition, reduce the potential blast radius of any single failure.
Resilience planning should also assume that agents will make incorrect decisions and, as the PocketOS agent later explained, “violate every principle I was given”. If an organisation detects that an agent is behaving unexpectedly, it must be able to limit the damage and restore any affected systems.
This means backup isolation is absolutely essential. In practical terms, recovery environments should be evaluated separately from production so that the same agent cannot alter or delete both production data and the backups needed to restore it. Those backups must remain available even when production systems have been compromised or accidentally altered.
Then there is the question of recovery. Even organisations with stringent access controls and backup processes should assume that some failures will still occur, so resilience must also be measured by how quickly and reliably they can recover. This means identifying which systems must be restored first to keep the organisation operational, along with the clean data and trusted recovery points needed to restore them.
As PocketOS discovered, when an agent can interfere with the backup process and delete recovery data, the resulting disruption becomes far more difficult to contain. Organisations must therefore be confident that they can recover even when their preventive controls fail.
Recovery readiness
The speed at which autonomous agents can operate makes it increasingly difficult to rely on conventional recovery plans that are tested only periodically. PocketOS lost its production database and backups in nine seconds, yet recovery took sixty hours. That disparity illustrates how quickly the gap between damage and recovery can widen when machines are responsible for both making and executing decisions.
Organisations should therefore validate their recovery capabilities regularly in an isolated recovery environment. This allows them to establish whether critical systems can be restored from clean data without reconnecting compromised components or reintroducing the conditions that caused the incident.
In the case of a rogue agent, restoring the data may only be part of the task. Organisations must also determine whether the agent’s credentials, permissions or underlying workflow contributed to the failure. Unless these are addressed before systems are brought back online, the same behaviour could simply resume after recovery.
This is where defining the Minimum Viable Company (MVC) becomes critical. An MVC is the leanest version of an organisation that can continue operating and serving customers during a disruption. It typically includes the essential systems, datasets and services needed to maintain core operations, such as identity and access management, communication platforms, financial systems and critical business applications. By identifying these priorities in advance, organisations can avoid trying to restore everything at once and instead focus on recovering the trusted environment needed to keep the business running.
Mean Time to Clean Recovery (MTCR) then provides a more meaningful measure of resilience. Rather than simply tracking how quickly systems are restored, it measures how long it takes to recover the MVC into a clean, trusted and operational state. This complements traditional recovery metrics by accounting for the time needed to verify data integrity, remove compromised access and validate that the conditions behind the incident have been addressed.
For organisations operating at machine speed, the objective should therefore be clear: define the MVC, test how it will be restored and continually reduce the MTCR. That is what will determine whether recovery plans can withstand an incident caused not only by a human attacker, but by an autonomous agent capable of repeating the same damaging actions in seconds.
