Millions of people manage their health conditions via apps on their phones. A person with diabetes may get a glucose reading from a wearable sensor every few minutes and get an alert if their levels go too high or too low. Many of these apps are regulated as medical devices. Some are standalone medical devices, i.e., software that can monitor, diagnose, or help treat a patient on its own. Others are part of a regulated device system and, in most cases, the FDA reviews the software before it’s released to the public.
The phones these apps are designed to run on are evolving. In September, Apple announced a redesigned Health app to collect, summarize, and provide users with their own health-related data and insights, i.e., trends in their sleep, steps, and other metrics, through AI.
Regarding the Personalized Glucose Management System, if Apple releases these insights and trends to users of the Apple Health app, this could definitely impact and reshape the insulin management space. Google also integrated a Gemini-powered health-coaching AI into Fitbit and rebranded the Fitbit app as Google Health in 2026.
This has created a situation the industry has not fully addressed. Now, a patient’s medical data can be interpreted by two kinds of AI. One comes from the device maker, which is subject to regulatory oversight for the medical device. The other is built into the phone and interprets information generally. I believe that generally purposeful AI built into phones should not interpret the safety alerts or treatment recommendations from the medical device. General-purpose AI should only add context, and if it does so, it should state that it is doing so.
Where AI already sits in medical devices
AI in medical devices is not new. The FDA authorized 331 AI-enabled devices in 2025, the most in a single year, and has authorized more than 1,600 since 1995. About three-quarters of these devices are in the field of radiology, while other specialties, including cardiology and neurology, are steadily increasing.
What is relatively new is the use of Generative AI that explains health data to patients. Generative AI creates new text or images, rather than sorting or ranking data. In 2024, Dexcom launched a Generative AI platform, in partnership with Google’s Gemini models, that provides users of its OTC Stelo glucose sensor with personalized health recommendations (i.e., nutrition and exercise) based on their health data.
The trend is clear. AI is progressing to tools that explain health data to patients rather than aids that clinicians use to generate health data.
Two AI Systems, One Patient
Imagine that a patient’s continuous glucose monitor (CGM) sends an urgent low glucose alert. The patient hears the alert and checks their phone, but the phone’s AI can also group recent notifications and display a summary on the lock screen. The summary may say “several glucose updates,” creating a lack of sense of urgency for the patient. Additionally, the phone’s health assistant can read the same glucose data and recommend dietary changes, while the device suggests that the patient reach out to their care team.
Errors like these have occurred with non-healthcare applications of AI. In 2024, Apple’s AI for summarizing news notifications wrongly reported a BBC News alert. After the summary, Apple removed the AI and paused the news summary feature.
The FDA’s guidance is behind the times. The FDA’s revised general wellness policy in 2026 states that some features of wearables can remain outside of medical device regulation, as long as they do not make disease or diagnostic references, provide medical opinions, or suggest treatment or changes to patients. The FDA’s policy states that software is considered a medical device based on the intended use and purpose, and not if it incorporates AI. Thus, a wellness feature of a phone that reads data from a medical device (e.g., CGM) may not fall under medical device regulation.
There are questions around who controls changes with AI. Many generative AI tools are designed using foundation models, large general-purpose models which can be customized to perform various tasks. The FDA’s discussion paper on generative AI-enabled devices, released in August 2026, mentioned foundation model changes (rather than device modifications) as change control issues. It is not guidance, but it gives an indication of the direction of the agency’s thinking. Phone AIs create similarly challenging change control questions outside of a device. Phone AIs also update on the phone manufacturer’s schedule. Device manufacturers perform many tests on newly released OS versions and determine if their device is compatible. An update to the phone’s AI may change how it interprets or rewrites the output and is not always detected in a compatibility test.
The source of recommendations
Regardless of whether the recommendations come from a medical device or a general AI feature on the phone, patients should be able to discern the following. If the recommendation is based on averages or generalities of the user’s past behavior (e.g., “your glucose levels have been above 200 mg/dL for the last week”), patients should understand that the recommendation does not mean it is certain the user’s glucose levels will be above 200 mg/dL for this week. Patients should be able to determine the specific recommendation the AI is giving and the time at which they should call their clinician. In cases where the AI recommendation is to call their clinician, patients should be assured that the recommendation is clear and they will not be criticized for following the AI’s instructions.
Compare two messages. One says, “Your glucose is trending well. Keep it up.” The other says, “Based on your sensor readings over the past week, your glucose was in your target range more often than the week before. This is a general summary and is not medical advice. Follow your device alerts and your care plan.” The second message, while longer, is clear to the patient what it means and what it is not.
Message length is important because of automation bias. Patients tend to trust messages given to them by a computer, regardless of whether that message answers their question. Sometimes a confident, correct answer by a computer can give the impression that it answers all the patient’s questions.
Most direct risks associated with changing the messaging framework are associated with loss of meaning. Phrases generated by AI can be grammatically correct but wrong or (in the best-case scenario for the designer) just poorly phrased, and can adversely affect the safety of the device.
A second direct risk is conflicting messages. If the phone and the device give different messages, the patient should not need to resolve the discrepancy. AI messages should show their source and prioritize the device’s message.
There is also the uneven performance of AI. For instance, one AI may perform worse for a particular condition or age group. Tracking performance against real-world scenarios should include different conditions and ages as well as geographical regions.
There is the potential for over-reliance. The patient and care team should always be in control of making decisions about the patient. Clear and effective framing should bring the clinician into view when more in-depth input is needed.
Community can collectively advocate for implementation of guardrails on AI
First, advocate for “protected outputs” where safety alerts and treatment recommendations marked by the device manufacturer would not be hidden by the AI. Apple has already demonstrated it can effectively manage AI summaries by hiding them in certain categories. After pausing summaries for news and entertainment, Apple restored them with a warning that summarization may change the original meaning of the headlines. Of higher importance than headlines are medical alerts, and therefore, they should be excluded from summaries.
Second, advocate for source labels. All AI-generated health messages should indicate whether they were generated from a medical device or general AI.
Lastly, advocate for joint testing. Prior to release, phone and medical device AI should be tested together.
The FDA has requested public comment on its draft paper concerning Generative AI until Oct 19, 2026. They usually solicit feedback within 60 days of the publication of a draft document.
Where this is heading
In a few years, it is likely that virtually all individuals who use a connected medical device with a smartphone app will have their device data interpreted by an AI built by a third party with whom the medical device manufacturer has no control.
Regulators seem to expect more change after launch, not less. The FDA’s discussion paper posits whether the agency should accept more uncertainty prior to authorization to enable greater monitoring after the product is launched. The paper notes generative AI can evolve and change over time and can depend on 3rd party models.
Health care can benefit from generative AI. One positive example is the use of generative AI to explain health data to patients in language patients can understand. But the regulated device is one that is tested for the job. Phone AI could help patients understand what their medical device said. It should not change what their medical device said.
Isha Sharma is a product development project specialist in digital health, leading development and global launch of multiple patient-facing applications used by patients across 120 countries. She previously worked in business development at an early-stage drug delivery startup and as an analyst at Bain & Company on healthcare engagements. She serves as an advisor to the Midwest Pediatric Device Consortium and as an expert evaluation panelist for the HHS TOPx Tech Sprint, and reviews journals including Journal of Translational Medicine and PLOS One. She holds an M.Eng in Bioengineering from UC Berkeley and an MBA, and is an IEEE Senior Member.
