The UK government will extend a programme using frontier AI models to find vulnerabilities in its own systems, and will consider legislating on AI incident reporting and loss-of-control risk. The pilot behind it found 407 weaknesses across nine government organisations in a month for £13,000 in tokens.
Britain will expand a programme that points frontier AI models at the government’s own code to find flaws. It will also look at legislating for AI incident reporting and for systems that escape human oversight.
The package came on Monday from the Cabinet Office, the AI Security Institute, the National Cyber Security Centre and the culture department, according to the announcement.
Kanishka Narayan, who in July became the first AI minister to attend cabinet, set it out at Imperial College London. He wants Britain to be “one of the hardest countries in the world to attack using artificial intelligence“.
The government says two pilots have found and fixed more than a thousand potential vulnerabilities. The case study it cites gives a smaller figure, because it documents only one of them.
That pilot produced 407 findings across nine government organisations over a month of weekly hackathons, at a cost of £13,000 in tokens.
Teams chose their own methods. One built a six-stage agent pipeline in which each stage challenges the last, another layered model stages over conventional scanners, and a third distilled an audit of hundreds of services into five Claude Skills.
The worst finding sat in legacy GitHub Actions on a repository behind a key government service. A crafted comment on an open pull request could run arbitrary code on the build machine and reach its secrets.
The warnings come from the government too. Agents produced candidate findings faster than people could check them, and the case study notes that “finding isn’t the same as fixing“.
Narayan had a message for the labs as well, that a company unable to control a system’s risks should not release it. In September, though, the White House asked OpenAI and Anthropic to withhold new models from the AI Security Institute.
Anthropic held back Mythos 5.1. Britain is urging restraint on labs whose newest systems its own institute cannot reliably see.
The institute is still evaluating the current generation. In July it found that every model tested for the behaviour tried to cheat, and that one attacked its evaluation systems.
Ministers will also explore strengthening the Cyber Security and Resilience Bill so hospitals and utilities are covered against AI risk, including loss of control, and how near misses get reported.
The next phase moves from public repositories into closed government systems. AI will also sit at the centre of Britain’s G20 presidency next year.