In a 2026 study in Nature Medicine, researchers at Oxford tested language models on ten medical scenarios. Working alone, the models identified the correct condition in almost 95% of cases. When 1,298 members of the public used the same models to work out what was wrong, they got it right in fewer than 35% of cases, a worse result than a control group relying on their usual sources.
The model knew the answer; the system around it did not.
That study was about the public rather than clinicians, but the gap it exposes is the one that matters most in healthcare AI: the distance between what a system achieves in testing and what it delivers in use. It is where most healthcare AI decisions go wrong.
In more than 25 years of building technology for clinicians, I have watched the same gap reappear with every new wave, from dictation and transcription to speech recognition and now generative AI. Closing it is less a procurement exercise than an organisational skill, and it is the skill I would urge every healthcare leader, and everyone entering this field, to develop.
Know What the Model Is Really Predicting
Every evaluation should start with a clearly defined problem, but the more revealing question comes next: what, precisely, has the model been trained to predict? In a 2019 study in Science, Obermeyer and colleagues examined an algorithm used across US health systems to decide which patients needed extra care. It predicted future healthcare costs as a proxy for need, and because less had historically been spent on Black patients who were equally ill, it systematically underestimated how sick they were.
The algorithm was accurate against its own target; it was simply measuring the wrong thing. No benchmark would have caught that. Only a close look at the gap between what a model optimises for and what the organisation actually needs would have done so, and that look is the first job of any evaluation.
Treat Performance Claims as Hypotheses
Headline accuracy figures are usually produced under conditions the developer chooses, and regulatory clearance does not settle the matter either. The useful question for any vendor is not whether a product is cleared but what kind of evidence sits behind its claims: a retrospective test on curated data, a prospective evaluation in live care, or a randomised trial.
With generative systems, the question has to go a step further, because there is no single correct clinical note to score a system against. Ask instead what it gets wrong, and how.
In ambient documentation, the errors that matter are quiet ones: a negative that is dropped so that “no chest pain” becomes “chest pain”; a drug name or a dose misheard; the wrong side recorded; a plausible finding that was never said. They arrive inside a confident, well-formatted note, in front of a clinician with a queue outside the door. So the numbers to ask for are the rate of omissions, the rate of fabrications and the proportion of errors a clinician would judge clinically significant, all three measured in your specialty rather than in a general sample.
National assurance schemes are a floor, not a verdict. Even NHS England’s 2026 registry for ambient voice technology states that assurance and decision-making remain with each local organisation. Having taken our own technology through the process, I would add that the registry tells a buyer what a supplier has evidenced, not how the product behaves in their clinics. It asks for device registration, a DTAC assessment, a monitoring plan and evidence of benefit; it does not ask what the note gets wrong, how often, and whether anyone would notice.
Those are the questions I would want a buyer to put to us. A supplier who cannot answer them with numbers, and show how those numbers moved after the last model update, has not yet done the work the registry assumes has been done.
Assume Performance Will Not Travel
A model’s performance belongs to the setting in which it was measured. Change the patient population, the languages spoken, the documentation habits or the workflow, and the numbers can change with them. Yet local validation remains the exception even in wealthy systems.
Speech and language systems are unusually sensitive to this, and I have watched the same product behave like two different products in two different rooms. The variables are ordinary ones: regional accents, a patient who switches language mid-sentence, the vocabulary of a subspecialty, a consultation held against the noise of a ward, and the shape of the encounter itself. A ten-minute general practice appointment and a forty-minute outpatient clinic produce different conversations, and a system tuned on one will not automatically understand the other.
For health systems in India, the Middle East and much of Asia and Africa, where clinicians routinely work across several accents, languages and local documentation conventions in a single day, the problem is sharper still. Most clinical AI is developed and validated on patients, languages and care models that look very different from theirs. Whether a model will transfer should be the first question in any evaluation, not a footnote, and the only reliable answer comes from running it on your own consultations before the contract is signed.
Test the People, Not Just the Model
Human oversight is usually presented as the safeguard, but it is also a point of failure. In a 2023 study in Radiology, radiologists reading mammograms with what they believed was an AI assistant saw their accuracy collapse when it suggested the wrong category, with inexperienced readers falling from 79.7% to 19.8% and even the most experienced from 82.3% to 45.5%.
Evaluation should therefore observe how real users behave when the system is wrong, rather than simply confirming that they are able to override it. For a documentation tool, that means watching clinicians review notes at the end of a real clinic, not in a demonstration, and counting how many seeded errors they catch.
Leaders should also ask the reverse question: what happens to the team when the AI is not there?
The evidence here is new and uncomfortable. A 2025 study in The Lancet Gastroenterology & Hepatology followed 19 experienced endoscopists, each with more than 2,000 colonoscopies behind them, across four centres in Poland. In the three months after AI-assisted polyp detection was introduced, their adenoma detection rate in procedures performed without the AI fell from 28.4% to 22.4%. Routine use of the tool appeared to erode the very skill it was built to support. If the system goes down, or a new clinician joins who has never worked without it, can the team still perform safely? That is a question to settle before deployment and to keep asking after it.
Evaluation Does Not End at Go-Live
A pilot shows that a system can work; it does not show that it will keep working. This is truer of generative AI than of anything that came before it. Ambient products are built on foundation models that their vendors update, and that the model’s own provider may change underneath the vendor. A note that was excellent in March may be produced by a different model in June.
Before signing, leaders should settle who monitors performance, whether the vendor will notify them of every model change and let them hold a version while they re-test, how a random sample of notes will be reviewed each month, and how a wrong note enters the organisation’s incident reporting so that patterns can be seen rather than single events explained away.
The same discipline applies to value. Success should be defined against the original problem, whether that is time returned to clinicians, errors avoided or faster turnaround, and measured before and after deployment. Be wary of time savings that exist only in surveys; clinicians who like a tool tend to overestimate what it saves them. Consultation length, time from consultation to closed note, after-hours documentation and coding completeness can all be measured against a baseline taken before go-live, and should be.
In the UK, there’s another point that is easily lost when a product arrives with national paperwork attached. The supplier’s clinical safety case (DCB0129) covers the product; the deploying organisation must build its own (DCB0160), owned by a named clinical safety officer, alongside the data protection impact assessment, the integration with the patient record, and clear answers on where audio is processed, how long it is kept and whether it is ever used for training. Leaders should also ask who will support the system locally: who understands their information governance, their record systems and their clinicians’ workflows, and who will be there when something changes.
Evaluation Is the Capability That Lasts
I have spent most of my career on the supplier’s side of this table, and every wave I have lived through has followed the same arc: the technology improves faster than the organisations’ ability to judge it, and demonstrations fill the gap. The demonstrations are now extraordinary. That makes the discipline more necessary, not less.
None of this is an argument for caution as an end in itself. The cost of a badly evaluated system is paid in the same currency as the cost of no system at all: clinician time, patient safety and trust. Evaluating well is what allows an organisation to adopt more, not less.
Models will keep improving, and today’s differentiators will become tomorrow’s baseline. What will not become a commodity is an organisation’s ability to tell the difference between what an AI system does in a demonstration and what it will do in a busy ward, in a different language, for a different population. The practical version of that ability is small. Before the next vendor meeting, write the problem down in one sentence, decide which number will tell you it has been solved, and ask to be shown how the product fails.
To anyone entering this field on either side of the table: the suppliers worth working with are the ones who welcome that request. The leaders who build this capability may adopt AI more slowly than some of their peers, and more successfully than most. The models will get better on their own. Whether they get better for your patients is a decision your organisation makes, and keeps making, long after the contract is signed.
Sources
- Bean AM, Payne RE, Parsons G, et al. Reliability of LLMs as medical assistants for the general public: a randomized preregistered study. Nature Medicine, 2026. doi:10.1038/s41591-025-04074-y. Figures used: 94.9% (models alone), fewer than 34.5% (participants with models), 1,298 participants; the control group identified a relevant condition in 47.0% of cases.
- Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019;366(6464):447–453.
- Dratsch T, Chen X, Rezazade Mehrizi M, et al. Automation bias in mammography: the impact of artificial intelligence BI-RADS suggestions on reader performance. Radiology, 2023;307(4):e222176. Figures used: 79.7% to 19.8% (inexperienced), 82.3% to 45.5% (highly experienced).
- Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology & Hepatology, 2025. doi:10.1016/S2468-1253(25)00133-5. Figures used: 19 endoscopists, four centres, more than 2,000 colonoscopies each, ADR 28.4% to 22.4%, three months before and after.
- NHS England, Ambient Voice Technology self-certified supplier registry, published 16 January 2026: digitalregulations.innovation.nhs.uk. Registry criteria (Class 1 medical device, DTAC, evidence of benefit, performance and monitoring response): Find a Tender notice 009501-2026.
