New ‘First Proof’ exam reveals how AI-powered math is a pay-to-play sport

Artificial intelligence models are getting good at math—but only the corporate AIs.

The First Proof project, which aims to objectively evaluate the mathematical reasoning of publicly available AI models, released its third batch of results today. They underline what recent headlines have shown—that some large language models (LLMs) are becoming powerful research tools. They also suggest that proprietary subscription-based models from so-called frontier labs far exceed the math skills of more transparent and accessible “open-weight” models.

“The gap between the open-weight models and frontier models is very large in research mathematics—much larger than in other domains,” says Mohammed Abouzaid, a mathematician at Stanford University and a member of the First Proof team.

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

In February a team of top mathematicians started First Proof as a rigorous, independent exam for cutting through the proliferation of imperfect “benchmarks” for scoring different models. The AI race to conquer math has accelerated dramatically since those quaint days, when its greatest successes concerned problems that were scarcely connected to actual research. OpenAI’s “slop drop” last week has put to rest scores of more pressing questions, many of which had stood unresolved for decades. And the company’s solution of the Navier-Stokes problem last month convinced many that LLMs are on the verge of superseding human mathematicians—if they have not already.

But these breakthroughs are far from controlled experiments. For the slop drop, OpenAI didn’t disclose the thousands of problems that the company’s new model attempted, most of which it didn’t solve. And that internal model remains completely inaccessible to outside mathematicians, preventing a direct comparison of its performance with others.

First Proof’s mathematicians solicited problems from experts in a broad range of subdisciplines and then selected 10. They asked each model under evaluation to answer the problems as written, with no further human input. Then they had three graders per problem evaluate the answers in a mock “peer review.”

First Proof’s rules require any entrants to be openly available, so OpenAI’s blockbuster new model was excluded. The team tested OpenAI’s GPT 6 Astra, Anthropic’s Claude Opus 5.5, an aggregate model called “ProofCouncil” that combines an Astra agent with other agents to check its work and a similar “harness” built on Kimi K3, the leading open-weight model—one whose inner workings are made completely public, among other entrants. Producers of other major models, including Google, did not participate.

The best performer—ProofCouncil—solved nine out of the 10 research problems in a manner the judges deemed acceptable with “minor revisions” for publication in a peer-reviewed journal. This helped quantify the models’ rapid ascent at math over the course of 2026. (In the round one results that First Proof released in February, the top model got just a few problems correct. And in the round two results released in June, the winner got six or seven.)

For the main skill that First Proof tests—proving well-defined statements—“the models have become very good,” Abouzaid says, “so good that testing the frontier models on this task might not be that relevant in the future.”

Notably, the best open-weight model, Kimi K3, solved exactly zero problems. “It really fell flat on its face,” says Lauren Williams, a mathematician at Harvard University and a member of the First Proof team. “The open models are really, really far behind, so we would urge the community to invest a lot into improving them.”

The First Proof team has received many requests to test open-weight models from mathematicians concerned about privacy after recent high-profile controversies. “I know people who use open-source models exclusively because they’re so paranoid about data leakage with the frontier models,” Williams says.

“We have a lot of questions from mathematicians telling us, ‘I would like to look for alternatives to using OpenAI’s or Anthropic’s models, but I don’t know whether they would work,’” Abouzaid says .

For now, it seems, they don’t.