Every previous wave of computing asked enterprises to transform before they could benefit. This one doesn’t and that’s exactly why it’s dangerous.
I’ve spent fifteen years building systems that answer one question: Why did it do that?
The question always arrives the same way. Someone from the business walks in, points at a screen, and asks why the system did what it did. For most of those fifteen years, I had a way out: if the software couldn’t answer, a person could, because a person made the decision and a person remembered why. The database recorded what was true; people recorded why.
That arrangement worked for decades because the person who made the decision was still part of the system. There was always someone who could explain what happened, even when the software itself could not.
Agents change that. They remove the person who could explain the decision while leaving behind systems designed to record what happened, not why it happened.
That creates a problem we have not had to solve at this scale before: when an AI agent makes a decision, where does the explanation live?
Technology transformations used to make their costs visible
Think about what the previous transformational waves of computing asked of us: client-server, the web, and above all, the cloud. You couldn’t lift and shift your way into elasticity. You re-platformed, rebuilt for statelessness and horizontal scale, hired consultants to run a program, and somebody called it a digital transformation.
Say what you like about that, it was honest about the price. The cost was a line item: a migration budget, a delivery plan, a board paper with a number on it. Painful, but visible, and visible before anyone signed.
This wave asks for none of that. Point an agent at the API you already have, and it works on Tuesday afternoon. No program, no re-platforming, nothing to approve. That’s real, and it’s good news.
The absence of an upfront cost does not mean there is no cost; it means the total cost of ownership is deferred and distributed elsewhere. It shows up the first time someone asks about a decision made months ago and nobody can answer. By then, it isn’t a budget line you can argue about. It’s an inability.
That’s the hardest kind of cost to manage, because nothing forces you to decide about it. A re-platforming program demands a yes or a no from someone senior. This one adds up quietly while every dashboard stays green.
Every hour an agent makes a decision without recording what it saw is another decision you’ll eventually be unable to explain. You can migrate a data center late, but you can’t back-fill a reason.
Retrofitting doesn’t just get more expensive; it gets impossible as the evidence decays. The customer record has been overwritten twice. The pricing table changed in April. The model version that made the call has been retired. Six months on, “what did this system know at 09:14 on the third of March” isn’t a hard question, it’s unanswerable.
Three forms of explainability
Ask most organizations how they’re handling AI explainability and you’ll hear about the model, and the tools that come with it. Attention maps, feature attribution, chain-of-thought traces. Something that explains what the model concluded.
That’s one third of it. And frankly, the third least likely to be asked about.
The inputs. What did the agent actually see, in the state it was in at that moment? Not what’s in the database today. What was in it then, including the parts since corrected, superseded, or deleted.
The reasoning. What it concluded, under which model version, which prompt, which policy, which tools it was allowed to use.
The effects. What happened because of it. Agents don’t just decide. They act. And their actions trigger other decisions, sometimes made by other agents. Nobody has ever asked me how a model weighted a feature. They ask what it did to them. If you can’t trace the consequences, you can’t answer that, and you can’t unwind the decision either. By the time an agent has got the same thing wrong four hundred times, your options are to do nothing or to review everything. Both are expensive. Only one of them is embarrassing in public.
So why does everyone measure the middle one? Because that’s the part vendors sell.
The first and the third are where the liability sits. They’re also the two your current architecture is busy destroying. Current-state storage overwrites the inputs by design. That’s what it’s for. Fire-and-forget messaging drops the effects the moment delivery succeeds.
You end up able to explain the reasoning behind a decision whose inputs are gone and whose consequences can’t be traced. That is not explainability. It is a thorough account of a conclusion, with no way to establish whether the conclusion was sound.
From nice to have to legal requirement
The EU AI Act is worth a glance even if it does not apply to you. Article 12 doesn’t ask for a report, it asks that high-risk systems automatically record events over their full lifetime, meaning the system produces the record itself, from deployment to decommissioning. That’s not a compliance checkbox, it’s a legislature deciding that explainability is a property of the system, not a document assembled afterward.
It won’t stay exclusively European. South Korea’s AI Framework Act, borrows the same high-risk categories. The United States has no federal equivalent yet, but its growing patchwork of state law keeps landing on the same short list: notice, documentation, record-keeping, retention. We’ve watched this movie before with GDPR, and I don’t know many companies that still treat data protection as a European concern.
You probably don’t need event sourcing everywhere
Event sourcing is not the answer to every traceability problem, and pretending it is doesn’t help anyone.
In event sourcing, the sequence of events is the state, rather than a report about it. The record can’t drift from reality, because there’s nothing else for it to drift from. Every question about what the system knew, when, and what followed is answerable by construction.
Not every decision needs event sourcing. For plenty of them, a reliable log of actions alongside your existing systems will do. The database keeps doing its job. You add a record beside it. I didn’t much enjoy reaching that conclusion, for reasons you can probably guess, but it’s the honest one.
Now, “we have logs” is not the same as having a record, and this is where most of us are fooling ourselves. What teams usually mean is application logging. Mutable, rotated after thirty days, written in language only the engineer who added the line can read, produced on a best-effort basis. It fails every test that matters. And I’ve written those lines myself, with the best of intentions.
What does a log need to be before you can rely on it? Append-only and tamper-evident, obviously. Beyond that, three properties decide whether you’re holding a record or just a lot of text.
First, the record has to be written in the same transaction as the action. If the action can succeed while the record fails, you now have two versions of history and no way to tell which one is true. A useful test: kill the process between the action and the record. What survives? If the answer is the action, fix that before anything else.
The inputs are captured by value, or by immutable version. “Evaluated customer profile 4711” is worthless the moment profile 4711 changes. Record the values you used, or a reference to a specific, frozen version of them. This is the most common mistake I see, and the one that quietly turns a complete-looking archive into an unanswerable one.
Causality is spelled out, not inferred. Correlation and causation identifiers, so you can walk forward from a decision to every action it triggered, and back from an effect to the decision that caused it. Wall-clock timestamps from six services are not an ordering. Without ordering, you’re back to holding the reasoning and none of the consequences.
That isn’t the whole list. Retention that matches how long people might ask, entries your domain expert can read without an engineer translating, and the ability to replay history through corrected logic. They all matter, and I’ve watched each of them sink an otherwise sound design. But a log that gets these three right is doing the work of a record instead of impersonating one.
What’s left is whether that log is authoritative, as it is in event sourcing, or a companion to something else that remains the source of truth. Choosing the companion is a perfectly good engineering decision. Just make sure you chose it, instead of discovering it later.
Start with a list, not a program
Nobody gets to start over. The systems that run your business are running right now, holding accounts, moving inventory, processing payments. They won’t be rebuilt because an architect drew a nicer diagram, nor should they be.
So don’t start a program, start with a list.
Write down the decisions that get made without a human in the loop, starting with the ones you’ve already handed to agents. Decisions, not systems. The list is usually shorter than people expect. Dozens, not thousands.
Then sort it by one question. Who will ask about this, and how long from now? A regulator. A board. An auditor. A customer. A counterparty’s lawyer. Much of the list has no answer to that question, and those decisions can carry on exactly as they are.
Take the top of the list and give those decisions a record that holds up against the properties above. Draw the boundary at the decision, not at the system. You’re not converting the payments platform. You’re recording the decision to make the payment, along with what was known and what followed.
Then stop. Come back when the next batch of decisions moves to an agent.
That’s a quarter of work for a first meaningful slice, not a two-year transformation. Which is why “we’ll do it when the regulation lands” is such a weak argument. It isn’t a big enough job to justify waiting for a deadline.
Do you know what your systems actually decide?
All of that assumes you can write the list. Almost no organization can tell you which decisions its systems actually make. Not from lack of caring, but because the knowledge was never in one place: old code, stray configuration, a workflow model nobody has opened in years. Ask which services feed a downstream decision and you’ll get guesses that differ depending on who you ask. That ignorance, not necessity, is what turns these efforts into full rewrites. If you can’t tell which parts of a system carry consequence, you end up treating all of it as if it does.
Here’s what I find quietly satisfying. Reading a large body of existing software and reporting what’s in it happens to be the one thing this technology is genuinely good at. I’ve been publicly skeptical about AI and software development, and I stand by it. These models behave like a knowledgeable intern who reads everything quickly and understands nothing fundamentally. They are excellent at summarizing and transforming knowledge that already exists.
Disappointing the moment the job is to create something new. But mapping what already exists, which decisions a system makes, which data feeds them, where state gets overwritten without a trace, is exactly that kind of work. The technology that removed the person from the loop turns out to be good at mapping what that person used to cover.
One warning. Ask it what’s there. Don’t ask it what to do about it. Deciding which boundaries matter and what a decision means to the business is design work, and design is where these models still miss the point while producing something that looks right.
Can we agree on some explainability upfront?
We have a choice about when, not about whether. Explainability becomes structural the moment consequential decisions get made with nobody in the room. That’s already happening. The only variable is whether you build for it on purpose, or retrofit it under a deadline, across a much larger surface, from evidence that has partly decayed.
What I find encouraging is that this wave asks less of us than the last one did. The cloud wanted a transformation before it paid out. This one doesn’t. It wants attention. A short list of decisions that carry real consequence, and a record beside them that keeps what was known, what was concluded, and what happened next. And the mapping that used to make all this feel impossible is now work we can get real help with, from the same technology creating the pressure.
For fifteen years, being able to explain your systems was a quiet virtue that few people asked about. It’s about to become one of the defining advantages of the companies that lead in AI.
Before we hand over the next decision, one ask. Can we agree on some explainability upfront?
Allard Buijze is the Founder and CTO of Axoniq and the original author of Axon Framework, the open-source event-sourcing framework downloaded more than fifty million times.
