The AI Governance Gap in Financial Services
JD Vance recently remarked that “the AI future will not be won by handwringing about safety.” He was speaking in the context of the international race to produce the fastest generative AI models: a race run by OpenAI, Anthropic, Google, and others. Yet, intriguingly, trustworthy, explainable, or risk-resilient AI models are not part of the finish line many are racing toward. Safety, it seems, is a second-order concern.
That may be acceptable when deploying chatbots for benign tasks like FAQs or menu planning. But when those same models are deployed inside compliance- and regulation-heavy industries such as financial services, the stakes change instantly. Banks and insurers must be able to explain how their systems work. Increasingly, they cannot.
Even the builders acknowledge the risk. Sam Altman has warned that biometric authentication is already broken, and banks face a deepfake-driven fraud crisis. Yet safety is too often cast as “slowing innovation.”
In financial services, ignoring AI governance isn’t naïve. It’s career-ending. Reputational fallout can be swift and significant. Fraud, mis-selling, AML breaches: these are not hypothetical. They are already here.
The Critical vs. Non-Critical Myth
Traditional governance divides systems into “critical” (like fraud detection, AML, credit scoring) and “non-critical” (chatbots, marketing automation, HR tools). That distinction is fast becoming obsolete.
Take the Air Canada chatbot incident in 2024. A bereaved customer interacting with the airline’s chatbot was misled about refund eligibility. The chatbot promised a policy that did not exist. When Air Canada refused to honour it, the British Columbia Civil Resolution Tribunal ruled the airline was responsible and ordered it to pay damages.
Now ask: if this had been a financial institution, would the regulator have let it slide?
Closer to home, the FCA has issued fines linked to automated marketing tools mis-selling investment products. Generative AI in marketing, often seen as “non-critical,” can cross into regulated territory in an instant.
Moral: Any AI system can go critical the moment it touches customers, compliance, or fraud vectors, especially when that system is lacking context, guardrails, or oversight.
Invisible Risks: Drift, SaaS, and Poisoned Context
Even for systems meant to be “critical,” control is slipping:
- Credit risk drift: A retrain at JPMorgan reportedly led to a 7% mis-scoring spike before being caught.
- AML drift: National Australia Bank’s AML systems produced false negatives that led to millions in remediation costs.
Then there are the hidden threats:
- Silent SaaS updates: OpenAI shifted all users onto a new ChatGPT model without notice in 2025, altering behaviour overnight before partially reversing course.
- Context poisoning: Microsoft 365 Copilot was recently patched after a zero-click prompt injection vulnerability (dubbed “EchoLeak”) allowed attackers to exfiltrate sensitive data.
Put drift, SaaS opacity, and poisoned context together, and you have a compliance nightmare.
Black Box vs Glass Box
Modern generative AI often functions as a black box: results appear without clear logic trails or auditability. Regulators will not accept “the AI told us so.”
Some institutions are experimenting with explainability:
- Capital One has invested in in-house tools for credit scoring transparency.
- HSBC has launched AML explainability initiatives.
Methods like SHAP and LIME help, but they are post-hoc fixes and not governance. In highly regulated industries, the only defensible path is glass box AI: traceable, explainable, and reproducible.
If you can’t explain what your AI did, you can’t defend it.
Context as Governance (and Performance)
Generative AI rarely fails from lack of computing power. More often it fails from lack of context.
Jennifer Bemert highlights “context engineering” as the missing discipline:
- Static: rules and policies
- Dynamic: transaction feeds
- Latent: customer histories
- Temporal: shifting fraud patterns
Without orchestrated context such as filtering, versioning, securing, AI systems fail. One corrupted document or poisoned feed can tip the balance. Governance collapses without explainable context flows.
Your Organisation and AI
Ask yourself (and be honest):
- Do you know every AI model in use (including SaaS or employee-adopted tools)?
- Are those models’ version-locked and monitored for drift?
- Could you explain every AI decision to a regulator tomorrow?
- Who owns AI governance? And do they have authority across your stack?
Few financial services firms can answer “yes” to all four. That alone should spark urgent action.
Vance may be right that the AI race isn’t won with handwringing. But in financial services, it will be lost through hand-waving.
Perhaps a wiser note belongs to DeepMind’s Demis Hassabis: “I would advocate not moving fast and breaking things.”
In financial services, that isn’t caution. It’s survival.
