A 60-year-old man who asked AI ended up in hospital after following dietary advice from ChatGPT.
The man had been looking for ways to reduce chloride in his diet and replaced table salt with a chemical compound called sodium bromide. He later developed hallucinations and paranoia and required hospital treatment. However, the chatbot conversation could not be retrieved, doctors were unable to determine exactly what advice the man had received.
The case detailed in a paper, authored by researchers from Binghamton University, Stanford University, Texas A&M University and Indiana University, argues that the lack of transparency surrounding AI-generated health advice can make such cases difficult to investigate.
"I think AI can be very helpful for health questions," Kaicheng Yang, one of the paper's authors, told Newsweek. "But AI makes mistakes, and it could state wrong answers with the same confidence as correct ones. The consequences might include delaying treatment and taking the wrong medication."
...
Growing Use of AI for Health Questions
The man isn't alone when it comes to using AI for health questions, according to a Gallup survey of more than 5,500 U.S. adults conducted between October 27 and December 22, 2025, which found that about 14 million adults reported skipping a visit to a health-care provider after using AI.
The researchers argue that current AI systems operate with little visibility into the health advice they provide, making it difficult for clinicians, researchers and regulators to identify harmful recommendations after the fact.
An OpenAI spokesperson told Newsweek: "ChatGPT is not a doctor and should never be used as a substitute for medical care, diagnosis, or treatment.
"We're clear in our terms that ChatGPT isn't intended to be used for medical diagnosis or treatment. People still use it for health information, and we take seriously the need to make those outputs as reliable and safe as possible."
The authors argued that harms linked to AI-generated health advice are often hidden from clinicians, researchers and regulators who would otherwise be able to identify and address them.
Yang, an assistant professor in the School of Computing at Binghamton University's Thomas J. Watson College of Engineering and Applied Science, told Newsweek that nobody knows "how dangerous" it is right now.
Researchers Call for Greater Transparency
Researchers recommend that users ask AI systems for their sources and verify information before making changes to medications, diets or treatments.
Yang told Newsweek: "We are not asking users to stop using AI for health questions.
"Our advice is to treat the answer as a starting point, not a diagnosis. For example, users can leverage AI to understand terms, prepare questions for a doctor, or interpret test results."
To improve accountability, the paper proposes several measures aimed at increasing transparency. Among them are giving users access to records of their health-related chatbot conversations so they can share those records with health-care providers investigating adverse events.
The authors also recommend creating systems that allow AI-generated health guidance to be flagged, reported and examined.
"AI companies should give users access to records of their health conversations and a way to report harm, and independent third parties should be able to monitor and evaluate the health advice these systems give," Yang explained.
The OpenAI spokesperson told Newsweek: "We believe AI can make that experience better by helping them find clearer answers, organize their questions, and prepare for conversations with medical professionals.
"At any given time, ChatGPT is powered by OpenAI's latest underlying models. These models continuously improve in recognizing when a situation may need urgent attention, handling uncertainty with better judgment, and giving people clearer, more useful guidance about what to do next.
"This progress is driven by close partnership with hundreds of physicians from 60 countries and representing dozens of specialties who help us define, measure, and improve health responses in ChatGPT."
The researchers further suggest that social media companies improve labeling of AI-generated health content and that search engines use medically vetted information in AI summaries. They also argue policymakers should consider extending physician malpractice liability to AI chatbot companies.
"We do not think this is something we should rely on the companies to do, because their incentive is always to make more money," Yang said. "Building such a system goes against that incentive. So we have to have some kind of third-party monitoring system, an independent evaluation."
AI Errors Can Reach People Who Aren't Looking for Them
The paper also highlights concerns about the quality of health information generated by chatbots. Researchers note that AI systems can provide incorrect answers with confidence, oversimplify complex medical information or draw on outdated material.
Those risks may extend beyond people actively seeking medical advice. With AI-generated summaries increasingly appearing in search engines and AI features embedded in social media platforms, users may encounter health information even when they are not searching for it.
"Sometimes I'm not even looking for health information, but just by browsing my social media feeds, it's there. It just shows up, and we believe that could have undesirable outcomes, especially if there's medical misinformation or state actors trying to manipulate the online discussion," Yang said in a statement.
The authors argue that whether people intentionally seek out health guidance or encounter it incidentally, harmful outcomes often leave little evidence that can be independently reviewed or verified.
Reference
DeVerna, M.R., Yan, H.Y., Yang, KC. et al. The invisible risks of AI-generated health information. Nat. Health (2026). https://doi.org/10.1038/s44360-026-00206-x