AMA releases governance framework for agentic AI in healthcare

An article from

Dive Brief:

• The American Medical Association has released a new governance framework to ensure the safe and effective use of AI agents in healthcare.
• The framework is based on the principle that for agentic AI solutions to perform clinical functions, they must be backed by evidence and maintain high-quality performance.
• It provides four questions focused on which tasks AI tools should be allowed to perform and in which situations, as well as who is accountable for the tools' actions.

Dive Insight:

The document offers insight into how the AMA plans to govern agentic AI as the tools are increasingly used in clinical care, including for ambient documentation, diagnostics and patient monitoring.

Agentic AI solutions can operate autonomously, completing complex tasks that generative AI cannot. Since the tools can complete multi-step tasks without human oversight, an inaccurate or biased decision could cause significant patient harm before a human clinician can intervene. 

Industry groups are grappling with how best to contain the risks of agentic AI, with the AMA doubling down on the physicians’ central role in the technology’s development and deployment. Its new framework will guide its work with other organizations to ensure AI agents are used safely and responsibly in tasks that touch patient care. 

The framework lists four questions the AMA will consider when assessing clinical AI agents:

• What clinical function is the AI performing?
• What demonstrates that it can perform that function, and what happens if it fails?
• What should the AI be permitted to do, and under what conditions?
• Who is responsible for the risks and outcomes?

“As AI capabilities advance, evidence, authority, oversight and accountability must remain aligned in service of better patient care,” the framework states.

The new guidance comes amid an explosion of agentic AI tools in healthcare. In a recent survey of 250 U.S. healthcare leaders, about 28% said their organization has already deployed agentic AI tools, and 44% reported conducting pilots or proof-of-concept projects. A majority of respondents expect agentic AI to have a "transformative" impact on clinical workflows.

But 72% of respondents say agentic AI tools are occasionally implemented without IT approval, highlighting a gap in supervision. The lack of oversight could result in major challenges for healthcare leaders, with more than half citing potential compliance or regulatory violations and unauthorized access to protected health information as major concerns.

Health AI is evolving so quickly that regulations are struggling to keep up. In the absence of substantive AI regulation from the federal level, the healthcare industry has established industry groups to provide guidance and guardrails. 

At the state level, a patchwork of laws and regulations has emerged, with some states reining in the technology and others experimenting with AI-based care. According to legal and consulting firm Manatt, 47 states introduced over 250 AI bills impacting healthcare in 2025.