One of the ways researchers try to improve AI models is to teach them to learn from their own mistakes made during tests, then re-test them to see how much they’ve improved. But what happens when the test itself is flawed?
The results are interesting, to say the least.
A research team at the AI startup Sentient Labs, headed by Dastin Huang, Baran Nama and Abhishek Saxena discovered the answer while conducting an experiment with AI agents that involved getting them to study their previous failures and transform the insights they derive into “reusable skills.” For the research, they developed a system that involves an AI worker that completes the tasks, and an AI coach that studies its mistakes and writes instructions so the model can avoid making them next time around.
The system succeeded, to a degree. After originally succeeding in only three of 120 attempts at completing a spreadsheet task, the AI worker was able to complete 21 of the tasks successfully in its second attempt. Then, suddenly, the researchers noticed something quite startling.
The AI spotted a flaw in the test
The spreadsheet used in the task turned out to be corrupted, and this meant it retained cached values from earlier calculations. The grader system used to calculate the AI worker’s scores was reading those cached values, rather than recalculating the formulas each time.
Because the spreadsheets originally contained the correct values, it meant that the cached data contained the answers to the problems the AI worker was being asked to solve. It was as if each test had a hidden answer key, and the AI coach spotted this long before any human did.
And this is where it gets interesting, for rather than reporting the error, it decided to leverage this discovery to help the AI worker do better, using the cached values to create a reusable skill. It provided instructions on how to modify the formula, as well as a note to the AI worker to “never run a recalculation” because “the file carries its own answer key.”
It was a sneaky move that was only noticed when one of the human researchers actually read through the generated skill and realized what the AI coach had found.
But despite the AI coach making a concerted effort to cheat at the test, it didn’t really improve the AI worker’s overall score. Once the grader had been fixed and the total 1,080 attempts made by the worker recalculated, not one of the results that had previously passed with the broken grader had failed. On the contrary, 81 previous failures turned into passes. So while the AI had found a loophole and exploited it, it actually had a negative impact on its performance.
All this was done on an incredibly small budget
Incredibly, the entire experiment cost Sentient’s researchers just $25. The team used Anthropic’s Claude Opus 5 model as the AI coach, before switching to DeepSeek V4 Flash to see which performed better.
Test runs coached by Claude achieved 77 passes out of 360 attempts, while DeepSeek V4 Flash fell just one short, with 76 passes.
It’s important to realize that the researchers didn’t teach an AI model to become more intelligent for just $25, for the model wasn’t retrained. Instead, it simply learned how to write instructions based on what it had learned from previous efforts, so it could reuse that knowledge later.
It’s an interesting technique that can sometimes pay off, but not always. On a second benchmark involving a banking task, the self-coached model actually regressed. It originally scored 37, but later only scored 30 when taking the tests a second time and leaning on its new skills.
Strange behavior
The researchers spotted some other curious behavior from the models. For instance, during two rounds of tests, they attempted to access files outside of their controlled environment on six occasions, only to be prevented by the sandbox. It’s not clear why this happened, the researchers said.
In a second experiment, the AI coach deleted an instruction in one of its own skills that had told the AI worker when it should bring a conversation to an end, claiming it had done this to “strengthen” the rule. As a result, the number of conversations that did not end rose from six to 16 out of 75. It was only while investigating this jump that the researchers found the modified rule and restored it.
Sentient’s research team was quick to stress that none of this behavior should be seen as “malicious.” Rather, they say the system was simply trying to optimize itself to achieve an objective more efficiently.
AI prioritizes achieving its objectives, but that doesn’t mean it gets smarter
Sentient reported its findings at a time when AI safety is dominating the headlines. Anthropic CEO Dario Amodei has been at the forefront of this, raising concerns that AI agents have tried to manipulate the systems used to evaluate them, and the risks of what’s known as recursive self-improvement – where AI systems build better versions of themselves.
Sentient’s team says its research isn’t an example of recursive self-improvement. An AI model that creates skills for itself to help it achieve an objective is not nearly the same as one that autonomously designs a new model that’s superior to itself.
What the experiment reveals is that, when you tell an AI model to learn from its past experiences to maximize its score in a task, it doesn’t try to make itself better at completing that task. If it can, it will just exploit a loophole, for in the AI’s eyes, the final objective is all that matters.
