Nebraska Medicine set out to use AI on a job its nurses spend thousands of hours on each year: pulling data from patient charts for its cancer registry. Two months later, the Omaha-based health system pulled the plug.
The registry takes in about 6,000 new cases a year, and each one takes a person roughly 2.5 hours to abstract, according to Michael Hasselberg, PhD, RN, chief transformation and digital officer of Nebraska Medicine. About 6.5 full-time equivalents, most of them nurses, do nothing else.
The plan was to have an AI tool handle the abstraction from start to finish. Nurses would sign off on the results and spend the time they got back on quality improvement instead of data gathering.
“If we can take that 2.5 hours that it takes you to do this case and take it down to a half an hour, and you’re just essentially validating the results, that gives you two hours back to again do the quality improvement part of it, and that’s what we really want them to be doing,” Dr. Hasselberg said.
The tool didn’t get there. Registry staff wanted at least 95% accuracy before the AI could run without a human checking its work, and it landed at a little over 80%. Some sections of the registry, which runs to hundreds of questions, cleared the bar. Too many fields didn’t.
So the team scaled back. If the AI couldn’t fill out the registry, maybe it could do the searching, pulling the relevant data from the chart into a summary so nurses wouldn’t have to dig through the record themselves.
The summaries were good, Dr. Hasselberg said. The nurses checked them anyway.
“The human wanted to go double, triple, quadruple check the source data, so they were going back into the record, and even if they saw where we pulled it, the source, they wanted to go into other parts of the chart just to double quadruple check,” he said.
The fallback saved little time, and in some cases it made the work take longer.
The experiment cost about two months of a data engineer’s time on nothing else, plus hours from registry staff who validated the tool and gave feedback on its design. The decision to stop went to Nebraska Medicine’s AI strategy steering committee. The committee is made up of the system’s C-level executives and votes on which AI projects move forward and when to move on from one. It shut the registry project down.
The stakes reached beyond one registry. Dr. Hasselberg’s team builds about one AI tool a month in-house, and this one was supposed to be a template: If it worked, the same approach could have automated other registries across the system that staff still complete by hand.
Instead, it produced a lesson the team has since applied to other tools it builds.
“To be able to leverage AI to pull data out and provide summaries is very different than fully automating a workflow,” Dr. Hasselberg said.
The right accuracy bar depends on the workflow, he added, and sometimes on the specific fields within it.
The project is shelved, not scrapped. Dr. Hasselberg said he still thinks registry automation is a good idea and doesn’t rule out coming back to it.
“After two months in it, we said, ‘All right, we’ve learned a lot. Let’s shut it down. Let’s go focus on other problems,'” he said.
Editor’s note: This is the first installment of a recurring feature. Email gbruce@beckershealthcare.com with other examples.
The post The pilot we killed: Why Nebraska Medicine shelved its AI cancer registry tool appeared first on Becker's Hospital Review | Healthcare News & Analysis.