The AI question buyers are asking, that most vendors can’t answer

When the Department for Science, Innovation and Technology was abolished in July, it renewed a familiar debate about which parts of government AI adoption actually depends on. But that debate risks overstating how much of the adoption problem sits with government at all. What is holding AI back in regulated sectors such as financial services and insurance can be partly attributed to a lack of policy, but more broadly, it is that buyers cannot defend the decisions these systems produce. That ability to defend a decision is what full adoption depends on.

What a regulated buyer needs before deployment

Sit in enough procurement conversations in financial services or insurance and the same questions come back. What is an adequate explanation of how this output was reached, and what proof is there that the system gets the right answer? How is the system’s reasoning reconstructed six months later when a decision is challenged? And who is accountable when it turns out to be wrong?

These are engineering questions with commercial consequences. Grant Thornton’s 2026 AI Impact Survey found that 78% of senior leaders lack strong confidence they could pass an independent AI governance audit within 90 days, and 46% name governance failures as a leading cause of AI underperformance. Among organisations still piloting, only 7% were very confident of passing that audit, against 74% of those running AI in full production. The data points to the conclusion that the organisations that can show how a decision was reached are the ones getting past pilot.

The pattern is consistent across sectors where a decision has to be justified to somebody, as it can prove difficult to provide a defensible account of how a particular output came about when using Large Language Models (LLM’s). 

Why generative architectures struggle with these questions

The difficulty is structural rather than a matter of engineering effort. An LLM predicts the most likely next words based on patterns in its training data. That’s a different task from reasoning through a problem to reach a valid conclusion. When you ask why a particular answer appeared, there is no reasoning trail to inspect, because no reasoning trail was ever constructed.

Instead, the industry has grown accustomed to building a second pass into the process, in which the model describes what it might have been doing. That is useful for debugging, but unsuitable as evidence. A story spun by a possibly hallucinating AI model about a decision is not the same as an undisputable record of which sources were used, what reasoning was applied, and why the system landed where it did.

This is manageable when the output is a first draft and much less so in a decision about someone’s mortgage application or insurance settlement. The same property that makes these systems fluent makes them difficult to hold to account.

Where hybrid approaches come in

The architectures most likely to clear that bar combine neural methods with symbolic reasoning – an approach known as neurosymbolic AI. In practice that means using language models for areas they excel in, including handling unstructured input and ambiguous language, while the decision itself is made by an explicit rule system that produces a traceable path from input to output.

Symbolic components need rules written by people who understand the domain, and kept up to date as policy changes. That is slower and more expensive than training a model on historical data, but it buys you decisions that can be reconstructed, tested against edge cases and corrected without retraining. 

For a majority of enterprise problems that trade-off is well worth making. If a decision has to satisfy a regulator, an ombudsman or a court, an auditable path is essential evidence in your corner. 

Auditability is moving into the RFP

The shift I have found most notable over the past year is that this has stopped being a conversation between engineers.

Explainability and audit requirements  are increasingly appearing in procurement documents written by risk and compliance teams. Much of that is preparation for regulation. The EU AI Act’s high-risk deadline has slipped to December 2027, but it slipped because the standards needed to prove compliance weren’t ready, not because the requirements eased. Buyers are using the extra time to work out what they will need to evidence, and to press vendors on whether they can provide it.

This means vendors have to be able to show buyers how the system reached a specific decision – confidence scores just aren’t cutting it anymore. Regulation may force the same thing eventually, but procurement is getting there first, because it comes with money attached.

It also means buyers should be more specific in what they ask for. If you ask a vendor whether their system is explainable, you will get a slide saying it is. But, if you ask them to take a decision the system made last year, reproduce it, and show you the path it followed, you’ll get a much better look at the technology you’re buying into, and how it’s going to help you get ahead of compliance. 

The standards vacuum is now a competitive variable

The practical advice for anyone deploying AI in a consequential decision is to stop waiting. Write your own explainability and audit requirements, make vendors evidence them against real cases rather than benchmarks, and treat the ability to reconstruct a decision as a functional requirement alongside accuracy and speed.

As standards like the EU AI Act begin to arrive, the organisations that have set a high bar for themselves already will find themselves compliant. The ones that waited will be retrofitting auditability onto systems that were never built to provide it, which is considerably more expensive than choosing differently at the outset.