In 2012, IBM set out to build a system that would grow wiser with every case it saw. Watson Health was deployed to help oncologists recommend cancer treatments, and the promise was that it would learn continuously as it absorbed clinical data and expert input. What happened instead is one of the most instructive episodes in the short history of enterprise AI. Experienced physicians found the system’s recommendations at odds with their own clinical judgement, and they began to override it rather than work through its reasoning. The corrections that would have taught the system were never captured. The oncology division was eventually closed. The system had recorded a great deal and understood almost nothing of it.
Learning Through Deciding
An organisation becomes intelligent through the act of deciding. A capable person makes a judgement, observes what follows, and adjusts. That loop, repeated across thousands of decisions and thousands of people, is how a firm accumulates the instinct that competitors cannot buy. It is slow, it is expensive, and it is the most durable asset most organisations will ever hold.
AI agents change the location of the decision. Earlier tools automated tasks and left the judgement to the human. An agent executes the whole chain, choosing and acting across steps that a person used to own, and it does so at a speed and scale no individual could match. The efficiency is genuine. The consequence is that the moment where a person used to learn now belongs to a machine, and the machine keeps no memory of why the choice was right.
Institutional Memory and Institutional Intelligence
Institutional memory is what an organisation has recorded, the data, the documents, the models trained on years of history. Institutional intelligence is what an organisation knows how to do, the judgement embedded in its people and its routines. Machines are extraordinary custodians of the first and incapable of the second.
Michael Polanyi (1966) gave us the phrase that explains why. We know more than we can tell. The expert underwriter, the seasoned clinician, the operations manager who senses a supplier problem from the tone of a phone call, all of them apply knowledge that was never written down and could not be. Kogut and Zander (1992) showed that this combinative knowledge, held in relationships and shared experience, is the most difficult of all resources for a rival to copy. It is the thing that passes the VRIN test that Barney (1991) set for genuine competitive advantage, valuable, rare, hard to imitate, and without substitute.
When an agent takes over the decision, the recorded memory continues to grow while the living intelligence begins to thin. The people who held it stop exercising it. The routines that transmitted it fall quiet. The loss is invisible for a long time, until the day a judgement is needed and no one in the room can make it.
The erosion carries a financial cost as well. Every step an agent takes is another inference call, and inference now accounts for the majority of the energy a model consumes across its working life. Global data-centre electricity demand reached around 415 terawatt-hours in 2024 and is projected to more than double by 2030, with AI as the primary driver.1 An organisation running agents at scale is paying a rising bill to automate the decisions through which it used to learn. It spends more each quarter and understands less.
The Industry That Automated Its Judgement
Wealth management traced this path a decade ago. Firms that deployed algorithmic portfolio tools believed they were building a technological edge. Today the major platforms offer near-identical strategies, rebalancing logic, and fee structures, and the edge has become the floor. The firms that kept genuine human advisory relationships and real client knowledge now command premium pricing that the automated platforms cannot reach. What looked like advantage in 2015 became the minimum cost of participation. The firms that dismantled their human judgement to get there were left with neither differentiation nor an easy way to rebuild it.
Three Questions Before You Deploy
- When this agent runs, who in the organisation is still learning? If the answer is no one, the agent is consuming the capability that made the work valuable. A senior partner at a global consulting firm described the pattern to me after a difficult client presentation. “We produce better-looking work than we used to,” he said. “I am not sure we produce better thinking.” A deployment that improves this quarter’s efficiency while emptying next year’s judgement is a poor trade, however it appears in the numbers.
- What does the organisation know how to do today that it will have forgotten in two years? Name the specific expertise that will fall out of use once the agent is running. If you cannot describe how that knowledge will be preserved and exercised, the deployment is retiring a capability under the appearance of automating a task.
- Where the decision is now the machine’s, where does human judgement still get practised? An agent that routes the difficult cases to expert people and captures their reasoning strengthens the organisation. An agent that runs high-stakes decisions end to end, with no one working through the logic, leaves nothing behind. The distinction determines whether the intelligence you are building remains yours.
What No Agent Can Learn
Firms endure by continuing to learn. Agents will remember every transaction, every document, every output, with a completeness no human institution could achieve. The impressive memory is exactly what makes the loss hard to see. A firm can watch its stored memory grow larger every quarter while the intelligence that made it worth trusting thins out of view.
Every leader deploying agents faces one real choice, and it concerns whether the organisation continues to learn once the agents are running. Build agents that route judgement to people and capture what those people know, and the firm compounds an advantage no competitor can purchase. Build agents that decide alone, and the firm scales its own forgetting.
A machine can remember everything a company has ever done and still leave it unable to make the one judgement its future depended on.
_____________________________________________
Dr. Mark Khater Dr Mohamed Khater is Head of the Centre for Strategy and Performance at the University of Cambridge, Director of the YNOT Institute for Digital Finance, and a Fellow Commoner at Queens’ College, Cambridge. He has been building artificial intelligence systems since 1994 and advises governments, central banks, and enterprises on cognitive strategy and human-centred technology. His research explores how AI and machine learning are reshaping competitive advantage, decision-making, and the boundaries of the firm.
1 International Energy Agency, Energy and AI (Paris: IEA, 2025). iea.org
