The Pentagon's guidelines for "responsible AI" aren't always followed, report shows

The Pentagon is building weapons and projects with artificial intelligence capabilities that officials say don't always closely adhere to the agency's own guidelines for AI, according to a congressional watchdog report published in October.

AI researchers and international organizations have long advocated for strict regulation of AI when used in weapons that could become unpredictable or function with little human control, arguing that those tools could endanger the lives of soldiers and civilians and risk quickly escalating wars. But Defense Secretary Pete Hegseth has repeatedly dismissed previous caution of AI tools and encouraged rapid autonomous innovation at the Pentagon.

The Pentagon issued a memo in January saying the department would become "an 'AI-first' warfighting force across all components, from front to back." Last month, Hegseth announced a new warfare command focused on autonomous weapon development, including drones.

The watchdog report, published by the U.S. Government Accountability Office on Oct. 14, grants a rare look inside how those kinds of projects are being developed.

The GAO surveyed 10 officials involved with building 10 weapons and tools for the Pentagon that use AI to automatically identify targets on the battlefield, process surveillance and improve tactical decision making, and other tasks. The projects are among more than 300 being built with AI in the Pentagon.

When responding to the GAO survey, six of the 10 officials said their teams had "identified risk(s)" with using the projects "in an operational environment."

To address some of those sorts of risks, in 2021, the Pentagon sent a memo outlining six basic tenets to ensure that weapons built with AI are created responsibly. Those six guidelines mainly stress the importance of Pentagon-wide oversight, verification and "real-time monitoring" of the technology, and cooperation with allies "to advance global norms grounded in shared values."

Alissa Czyz, who directed the GAO review, said the tenets evolved from earlier ethical principles, and while they weren't rules, they were understood to be important by people involved with building AI tools at the Pentagon.

"It's a memo, right, at the end of the day, so it's not a department instruction," said Czyz. "It's something that the department has signaled that should be implemented."

Most of the people involved with making the 10 weapons and projects reviewed by GAO had taken "at least initial steps" to follow the six tenets outlined in the memo, according to the report, and some of the projects were in early stages, Czyz clarified.

But some of the responses indicated officials weren't always closely following the guidelines.

Four officials told the GAO that requirements for the weapons and projects they were working on weren't "aligned with Responsible AI tenets." And three of the 10 officials told the GAO they "don't know" if the projects will be continuously monitored after they're deployed, though the six tenets encourage monitoring.

Officials told the GAO that more access to expertise about the agency's guidelines or staff who know about them would increase their ability to incorporate AI responsibly. But the group that helps manage education on the Pentagon's guidelines, the Department of the Responsible AI Division, is understaffed. Although the group was authorized to hire 10 people in 2023, as of February, only three of those positions were filled and the Pentagon "is unlikely to fill vacant positions or increase staffing," the report states.

"That's a very small number of folks tasked with a large responsibility of responsible AI throughout the department," said Czyz.

The GAO issued two recommendations: The Pentagon should more clearly outline the responsibilities and meeting schedules of the office charged with overseeing AI regulation, and address understaffing. Although the Pentagon agreed with those recommendations, as of Oct. 22, the agency had not implemented those changes, a GAO representative confirmed to NPR.

NPR asked the Pentagon when it planned to make those changes, but did not receive a response.

When delivering a speech at SpaceX in January, Hegseth indicated he believed AI development at the Pentagon should not be slowed by those sorts of steps.

"Our enterprise currently operates on staffing and committee cycles measured in months and years, and that's unacceptable," said Hegseth. "Military AI is going to be a race for the foreseeable future where the risks to U.S. national security of moving too slowly outweigh the impacts of imperfect alignment."

NPR would like to hear from more people who work with federal agencies or AI companies and have information to share about how agencies are using AI. You can email the reporter of this article, Chiara Eisner, at ceisner@npr.org or contact her on Signal here, username ceis.78.