Imagine returning to a digital therapy programme that helped you through a difficult month. Its name and interface are unchanged. Yet the voice responds differently.
An exercise moves faster. A question that once invited reflection now produces immediate reassurance. Somewhere between your last session and this one, the software was updated.
For the developer, that release may represent progress. For the person using it, the experience may have changed in ways that matter psychologically. For a healthcare organisation relying on published results, it raises a question that should accompany every significant update: how closely does this version resemble the intervention that was evaluated?
We need to become more precise about what we mean when we describe an AI product as supported by evidence. A study examines a particular intervention, delivered under particular conditions, to particular people. Its findings deserve to travel with that context.
The product name alone cannot tell us whether those conditions still hold. That connection needs to remain visible as products evolve.
When an update changes the experience
This matters because psychological support depends on how an interaction unfolds. A response can be factually accurate while arriving too quickly, asking too much or closing a conversation prematurely. In immersive environments, the relevant experience also includes sensory intensity, the sequence of activities and the person’s ability to control what happens next.
Consider a hypothetical programme that helps someone practise remaining with manageable uncertainty. Its developers introduce a model that gives warmer, more reassuring responses. Users may prefer the new version.
However, if reassurance now arrives whenever uncertainty appears, the programme may offer fewer opportunities to practise the skill it was designed to develop. Whether that change helps or hinders requires assessment.
The same question applies when an immersive programme changes its instructions or adjusts how quickly an environment responds. A modification that appears minor on a development schedule may change the demands placed on attention or the degree of control a person experiences. We should examine those possibilities before assuming the earlier findings remain equally applicable.
Deciding what needs further assessment
I see this as a responsibility shared across product development, research and purchasing. Developers need room to improve their systems. Patients need improvements too. The difficulty is deciding which changes preserve the intended intervention and which alter it enough to require a different level of scrutiny.
Requiring a new clinical trial for every correction would be impractical. A spelling correction and a change to crisis escalation rules clearly deserve different treatment. Between those examples sits a large area of judgement: changes to conversational prompts, memory, exercise selection, pacing and the underlying model. Teams need a process for assessing their likely impact before release.
That process should begin with an explicit description of what the intervention is meant to do. Which psychological task does it support? What are its boundaries?
What should happen when the user struggles? Without that description, it becomes difficult to distinguish an improvement in delivery from a change in purpose.
Testing should then examine the experience across an interaction, including difficult moments. A polished answer to a single question reveals little about how a system behaves after repeated requests for reassurance, contradictory information or an expression of discomfort. Reviewers should compare the previous and proposed versions in situations relevant to the intended users, documenting where their behaviour diverges.
Some changes may be justified through technical testing and expert review. Others may warrant usability work, additional outcome measurement or clinical investigation. The appropriate response depends on the intended use and the significance of the change. Claims should remain proportionate to what has actually been established.
Questions for healthcare buyers
Healthcare buyers also have a role. When considering a product, they should ask which version the supporting research evaluated, what has changed since and how those changes were assessed. They should establish who receives update notices and what happens if a release creates problems. These questions belong in purchasing conversations before deployment.
Users deserve explanations that help them make sense of changes they can feel. A generic promise of improved intelligence says little to someone who finds a familiar exercise unexpectedly demanding. Where a change affects the experience, communication should explain what is different, how to raise a concern and what support is available. Technical disclosure alone may leave the person’s actual question unanswered.
The human issue here is continuity. Someone seeking support may invest effort in learning how a programme works, deciding what to disclose and recognising its limits. An update enters that developing relationship. Treating the transition carefully acknowledges the work the person has already done.
We also need humility about what an unchanged result means. If satisfaction remains stable after an update, that is useful information, but it does not settle whether the intervention continues to support its intended outcome. Evaluation should follow the purpose of the care being offered, with enough time to notice effects that are not immediately visible.
AI makes revision easier. Healthcare needs a clear account of what each revision changes for the person receiving support. Confidence should rest on the connection between the current experience and the evidence behind it. When that connection becomes uncertain, recognising the uncertainty is part of responsible innovation.