Much of the debate on AI safety in healthcare focuses on the dangers of misinformation, rogue AI agents and the very real threat of increasingly autonomous systems acting beyond their intended purpose and remit. Important as those concerns are, they can potentially distract us from a more immediate and pressing concern: clinicians reaching for general-purpose AI tools that nobody has clinically approved or even checked.
Whilst there’s nothing wrong with using AI tools in principle – in busy clinical environments, AI can help clinicians swiftly find evidence and guidance – its use in practice can have serious implications. In order to make informed decisions, clinicians need to understand what tools they are using, how they work, and where they draw evidence from. Human-in-the-loop oversight remains essential so that technology augments clinicians’ decision-making, rather than redefining it by stealth.
Whipping out ChatGPT to quickly look up a clinical question is understandable, but undocumented uses risk creating a ‘shadow AI’ problem, where technology covertly enters clinical workflows before governance and vetting processes have had a chance to catch up. As AI tools become more accessible, informal adoption is likely to increase, particularly where workload, fragmented systems, and time pressure make the shortcut feel rational.
But if there’s one thing the healthcare sector does not need, it is another high-level warning about why AI is dangerous. Clinicians are using AI (whether we like it or not), so as an industry we need to come together, agree on what validated alternatives look like. As part of this, we need to seriously consider how current generic models fall short in providing healthcare professionals the support they need.
The accountability gap: Why generic AI fails in clinical settings
Generic AI does not fail in healthcare because the underlying models are inherently useless; it fails when it is trained on, and surfaces, generic data that lacks the review and accountability trails that are essential in a clinical setting.
While generic AI tools do provide sources, these sources often do not reflect the high-level evidence that clinicians need to a) ascertain the information is correct and b) make informed decisions off the back of that data.
In the heat of the moment, when busy clinicians rush to be efficient, it’s easy to forget that AI tools can be wrong, use weak evidence, and offer biased recommendations. People tend to trust these tools too easily, and one of the most worrying developments is that they are beginning to trust AI tools more than they trust people. This misplaced judgement risks overlooking the value of human oversight and creating a systemic breakdown in evidence-based practice.
This process also raises an accountability problem: if an unvalidated tool contributes to an unsafe decision, who takes responsibility for the error? The answer, quite simply, is that the clinician remains responsible for the decision made, while the employing organisation may be exposed for allowing ungoverned technology into the workflow. This is one reason it is so important to understand AI tools before we use them, and why “human in the loop” must mean informed, active clinical oversight.
Healthcare doesn’t just need a model that can generate quick, easy answers; it needs tools suited to a defined clinical purpose. The practical response to these challenges should therefore not be to ban AI, but to replace opaque, ungoverned tools with validated alternatives, and train staff to use AI with intention and integrity.
What it really takes to build a credible AI tool
In my experience, building a clinically credible AI tool or ‘assistant’ begins with deciding where your data comes from. Using a clinically validated dataset, often curated for a specific profession or speciality, means choosing not to use the infinite resource of generic data available on the internet. In the short term, this choice can feel restrictive and make the product feel narrow; it can limit the questions an assistant can answer safely, and mean accepting that it will occasionally be less superficially impressive than a chatbot willing to answer anything.
However, these initial constraints also provide long-term protection. A closed, curated system makes it possible to identify the sources behind an answer and remove outdated material, replacing the simplistic “chatbot says” approach with robust answers grounded in verifiable sources. In a healthcare setting, this is critical to underpinning trust, upholding confidence and ultimately empowering clinicians in their work.
We’ve been fortunate in that respect: our organisation has spent nearly two decades building a professionally reviewed knowledge base, so the foundations were already in place. Most organisations won’t have that head start, but that doesn’t mean defaulting to the open web. Partnering with professional bodies, licensing established clinical content or starting with a narrower, well-evidenced scope are all ways to build credibility and validation from the get-go.
But accuracy is only part of the picture; just as important is defining what a tool is, and is not, permitted to do. AI tools certified as medical devices are a strong example of AI supporting clinical decisions safely, while information-only and educational tools must be careful never to stray into that territory. The same clarity should apply to data: what personal information a tool accepts, and whether the company behind it shares, sells or uses inputs to train other models. The more useful a tool becomes, the greater the pressure to blur these lines, so defining and maintaining clear guardrails must be an active, ongoing choice.
Who will win with AI in healthcare?
AI tools must fit the setting in which they are used, and the healthcare organisations best placed to benefit from AI will not necessarily be those that move fastest, or those that try to keep it at arm’s length. Rather, success will depend on scoping and validating tools properly before adoption. That begins by asking the right questions:
- Where does an AI tool’s knowledge come from, and can you name the source?
- Has it been reviewed by people qualified to judge clinical accuracy in your profession or speciality?
- Can the tool show its working, with every answer tracing back to a checkable source?
- Who is accountable if it’s wrong, and has that been agreed before the tool is adopted rather than after?
- What happens to all inputted data? Is it used to train other models?
Healthcare organisations must be crystal clear on the answers to these questions before they even consider deployment.
As AI becomes more embedded in everyday clinical work, it’s tempting to think the right tool is the whole answer; it isn’t. Even the best-designed AI tool or assistant, one that satisfies each of the above questions, depends on the workflow and team that it sits within.
Looking forward, to ensure these tools remain fit for purpose, healthcare leaders must move beyond broad AI policies and vague commitments. A new standard of AI safety in healthcare must include clear governance, robust testing, structured staff training, meaningful human oversight, and a willingness to scrutinise tools on an ongoing basis, long after deployment.
The organisations that pull ahead will be those with empowered teams who understand the tools in their hands, trust where the answers come from and know when to push back. That confidence doesn’t just come from the software; it comes from the processes and standards built around it.
