Google’s multiagent artificial intelligence system outperformed comparison AI systems on several personal health tasks, but expert reviewers rated 17% of health summaries from one of its agents as carrying at least mild potential harm, according to a study published Oct. 21 in Nature.
Here are seven things to know from the study:
- The system, called Personal Health Agent, uses three specialized agents built on Google’s Gemini 2.0 models. One analyzes wearable and blood test data, one supplies medical context and one coaches users on wellness goals, with an orchestrator coordinating them. Alphabet funded the study, and most authors are current or former Alphabet employees.
- Researchers tested the system across 11 benchmark tasks, with more than 7,000 human annotations and 1,350 hours of assessments from health experts and users.
- Each agent beat the base Gemini model at its own task. The data science agent passed 75.5% of expert-written coding tests on its first attempt, compared with 58.4% for the base model. It also flagged insufficient data about 67% of the time, compared with about 5%.
- Clinicians rated 65% of the health-domain agent’s summaries as useful or very useful, compared with 30% for the base model. They also judged 15% of the summaries as presenting mild potential harm and 2% as presenting moderate-to-severe potential harm. These ratings reflect expert assessments, not documented patient injuries.
- The health-domain agent identified the correct leading diagnosis in 46.1% of 2,000 retrospective consumer health cases, a modest gain over an earlier Google system.
- In a test with 45 volunteers role-playing real study participants, the integrated system was the top choice 46.7% of the time, compared with 28.9% for a parallel multiagent system and 24.4% for a single-agent system. Three health experts ranked it best in about two of every three conversations. It was also slower, averaging about 3.5 minutes per reply compared with 36 seconds for the single-agent system.
- The system is a research prototype for consumer wellness, not clinical diagnosis or triage. The study relied mostly on Fitbit data and blood tests rather than full EHRs, and it measured short-term conversation quality rather than health outcomes. Researchers said future versions need several additions before real-world deployment:
- Safeguards to flag high-risk questions and direct users to clinicians.
- Broader testing across demographic groups.
- Stronger privacy protections.
The post Google’s health AI agent tops Gemini but carries safety gaps: 7 things to know appeared first on Becker's Hospital Review | Healthcare News & Analysis.