Artificial intelligence systems designed to help Emergency Departments (ER) decide which patients should receive care first may be influenced by the information in a patient’s medical records, potentially causing some higher-risk patients to be deprioritized, according to new research.
The peer-reviewed study, published in NPJ Digital Medicine, examined whether the AI models used to assess emergency patients changed their recommendations when researchers introduced information that can carry clinical stigma, such as frequent emergency-department use, psychiatric emergencies, and substance-use histories.
The researchers found that this type of information could shift an AI model's assessment away from a patient who was actually at greater risk of their condition deteriorating.
How the AI Experiment Worked
Rather than allowing AI to make decisions about real patients' treatment, the researchers conducted a controlled experiment using records from two academic emergency departments.
Researchers compared patients who appeared similarly ill when they first arrived at the ER but had different outcomes. One patient later deteriorated while the other didn’t, allowing researchers to test whether AI correctly identified who should receive priority.
The researchers tested models from three prominent AI families: Gemma, developed by Google; Qwen, developed by Chinese technology giant Alibaba; and DeepSeek, developed by the Chinese AI company of the same name, to see which of the two patients should receive priority attention.
Researchers examined 18 different conditions, testing whether demographic, social or potentially stigmatizing details, including a history of frequent emergency-department visits, psychiatric emergencies, or substance abuse, changed which patient the AI prioritized.
This allowed the researchers to investigate an important question: Would adding potentially stigmatizing information cause an AI model to switch its preference away from the patient who ultimately proved to be at greater medical risk?
What the Study Found
...
The results indicate that the answer can be yes.
Researchers found that telling the AI that a patient frequently visited the ER could make it less likely to prioritize that patient, even when they were the one who later became more seriously ill. Information about a history of psychiatric emergencies had a similar effect in some tests.
Crucially, that meant information beyond the immediate severity of a patient's condition could affect which patient an AI system considered the higher priority.
The findings, therefore, suggest that using AI doesn’t necessarily eliminate bias from medical decision-making, because the models can still be influenced by how patients and their medical histories are described.
What Does This Mean for ER Patients?
For patients, the concern is straightforward: if AI tools eventually play a significant role in emergency-room triage, information in a medical record that isn’t central to the immediate medical emergency could potentially affect where an individual sits in the priority order.
That matters because emergency departments use triage to determine which patients require the most urgent attention, rather than simply treating people in the order in which they arrive.
The research particularly raises questions about patients whose records contain information that may already expose them to stigma in health care, including those with psychiatric histories, substance-use disorders or repeated emergency-department visits.
Does This Mean AI Is Already Delaying ER Treatment?
...
No. That’s an important limitation of the findings.
The study didn’t test an AI system that was independently determining the order in which real patients received treatment. Instead, researchers used existing clinical data and controlled comparisons to determine whether particular pieces of information could change the models' prioritization decisions.
The findings, therefore, identify a potential risk that should be addressed before such systems are solely relied upon for high-stakes clinical decisions.
Why the Findings Matter
Hospitals and researchers are increasingly examining how AI can help clinicians process large amounts of medical information and make decisions about patients.
The new study demonstrates why simply measuring whether an AI model is generally accurate might not be enough.
In emergency medicine, where determining which patient needs attention first can have serious consequences, the findings suggest AI triage systems would require careful testing for bias before being incorporated into clinical decision-making.
The study ultimately highlights a paradox surrounding AI in health care: technology could help doctors interpret complex patient information more efficiently, but the language contained within that information may itself influence what the technology decides.
Newsweek has contacted Dr. Philip Jarrett, the co-author of the study and Assistant Professor of Emergency Medicine at UT Southwestern Medical Center, for comment.
Newsweek’s reporters and editors used Martyn, our AI assistant, to produce this story. Learn more about Martyn here.
Contact Newsweek editors on this story: John Fitzpatrick and Gray R. Thomas