This audio is auto-generated. Please let us know if you have feedback.
National healthcare groups representing hospitals, health systems, physicians and telehealth companies are telling the government they don't want to be held responsible for the safety and performance of medical devices that use generative AI, arguing that developers should shoulder that burden.
In comments submitted to the Food and Drug Administration, the American Hospital Association, American Medical Association and American Telemedicine Association said that monitoring devices that use generative AI — including tools like chatbots and those that synthesize medical information — would place too heavy a load on providers, particularly small and rural practices.
Rather, developers should carry the responsibility of monitoring AI devices used in healthcare, and the FDA should create ways for providers to report problems with devices.
“Manufacturers should remain responsible for the safety and performance of the devices they sell,” according to comments submitted by ATA Action, the lobbying arm of the American Telemedicine Association.
As AI tools advance at warp speed, the healthcare industry is grappling with who’s responsible for the tools’ mistakes. For providers, taking on responsibility could increase their exposure to malpractice lawsuits, while tech companies could face their own negligence claims. Neither party is eager to take on responsibility for the tools.
One problem, doctors say, is a lack of transparency into the AI models themselves. Hospitals told the FDA that, while health systems usually have established governance to review AI-backed devices, a lack of understanding into how the models actually work would make it hard for providers to identify flaws in the models.
“While hospitals and health systems support evaluation and monitoring activities, vendors and developers must be responsible for the ongoing performance, maintenance and integrity of the tools they market,” the AHA said.
The comments came in response to the FDA’s request for feedback from the industry on its proposed approach to regulating generative AI-enabled medical devices, which have unique characteristics compared to other types of AI. For instance, they accept open-ended inputs and produce different outputs to similar inputs, the FDA noted. Various trade associations, including the AHA, AMA, ATA and Workgroup for Electronic Data Interchange, submitted comments to the FDA.
Providers orgs question human supervision safeguard, ask for AI labels
Beyond the question of responsibility, provider groups generally agreed with other elements of the FDA's framework, like its proposal for assessing risk for generative AI-backed devices.
The agency is proposing to assess risk based on the actions a device performs. For example, a device that provides information that doesn't require immediate action, such as a prediction of a future cardiovascular event, has limited consequences and is low-risk, whereas a device that makes decisions entirely on its own could have severe consequences.
Still, some groups had suggestions on how to tweak the framework.
The AHA noted that activity or action alone may not be enough to predict risk. Some activities carry several risks, such as AI tools that recommend emergency care, which could cause patients with non-emergency conditions to seek care, or fail to recommend emergency care to a patient who needs it.
The framework should include more variables to account for these possibilities, such as the likelihood of error or negative outcomes from using a specific device, rather than only assessing the tool’s activity level: whether it is purely informational, results in an action or the action occurs under the supervision of a human clinician.
The AMA had recommendations on the human-supervision aspect of the framework, noting that having a human-in-the-loop – a commonly employed safeguard for AI tools – does not necessarily mitigate risk. Clinicians may make errors when reviewing AI suggestions, or they may become too reliant on the tools and may not be able to effectively oversee them, so evaluating the humans too should be part of the risk assessment, according to the AMA.
The discussion paper also describes approaches to evaluating the devices before they hit the market and monitoring them afterward.
WEDI further suggested that the FDA create a process for reporting adverse events and develop “nutrition” labels to give patients and clinicians insights into a tools’ functions and its potential risks.
This is not the first time a nutrition label-type card has been suggested for health AI tools. The Biden administration had issued a rule for developers to create these types of cards, but the Trump administration rescinded it last year.
Trust in health AI is critical to encourage adoption. Physicians say they want AI that is validated by a trusted entity and is continuously monitored. They also want a clear liability framework and more oversight of AI-enabled medical devices by governing bodies, like the FDA.
Physicians may get what they want soon enough. Rick Abramson, director of the FDA's Digital Health Center of Excellence, said at a conference earlier this week that the agency plans to post draft guidance on its approach to regulating generative AI-enabled medical devices in the coming months.