AI Benchmark Scores Don’t Tell the Whole Story. Here’s What You Need to Look For.

AGI, artificial general intelligence, means a system that can match or beat human performance across essentially any cognitive task, not just the narrow ones it was trained for. That distinction matters because it’s exactly what’s being argued about right now.

Jensen Huang has a habit of declaring victory at the exact moment his company needs the market to believe it. So when OpenAI shipped GPT-6 Astra on Sept. 3, Huang posted on X that “AGI has arrived,” adding that “400K GPUs” are “coming online next.”

I’m an AI engineer, and I’m currently building ChatMakr, an AI-powered platform that creates multi-agent chatbots in seconds. I spent the last year watching every model release get relabeled as the one that finally changes everything. Here is the truth.

What Astra actually does

Astra isn’t a chatbot upgrade. OpenAI positions it as an operator — a system that browses, codes, runs multi-step workflows, models in Blender and handles cybersecurity tasks with minimal supervision. The benchmark numbers are genuinely striking: 97.6% on FrontierMath Tier 4, 99.9% on ARC-AGI-3, and a perfect 100% on ExploitBench, up from 78.5% for its predecessor, GPT-5.6 Sol. Those are not incremental gains.

But OpenAI itself, notably, stopped short of calling Astra AGI, choosing instead to emphasize “speed and versatility.” That restraint from the company that built the model is the tell.

Why people aren’t convinced

The pushback isn’t really about whether Astra is impressive. It clearly is. It’s about the word AGI itself. Wharton professor Ethan Mollick has argued for months that what we’re seeing is “jagged AGI:” systems that are superhuman in some narrow domains and unreliable in others, which is a very different thing from general intelligence.

There’s still no agreed upon test for AGI, so declaring it “arrived” is closer to a subjective judgment call than a measurement. Augmented Mind and Manolo Remiddi put it plainly: “AGI is the point where you can trust the output without checking. We are not there.”

The Fable 5.1 comparison and why scores lie

This is where it gets interesting for anyone actually building with these models. On paper, Astra and Anthropic’s Claude Fable 5.1 look neck and neck on several coding and reasoning benchmarks. In practice, engineers report meaningfully different output quality. Fable tends to produce more production-ready code with fewer output tokens, while Astra’s strength shows up more in computer use and multi-step automation. Snorkel AI’s testing revealed that Fable fails through a fundamentally different pattern than how it succeeds, losing valid work mid-trajectory rather than running out of budget, which matters far more to a shipping team than a leaderboard percentage.

The reason similar scores can hide such different outputs comes down to something most people outside AI engineering have never heard of: the harness.

What a “harness” actually is

A harness is the infrastructure wrapped around a model: the prompt assembly, tool access, memory management and error recovery that determine how the raw model is actually put to work. As MindStudio describes it: “A model is the engine. The harness is the car. Most benchmarks test the car.” Swap the harness and the same model produces wildly different results. Cursor’s own research found one model scoring 46% accuracy with one harness and 80% with another, with no model change at all.

Astra’s own benchmark history makes the point better than any explainer could. On ARC-AGI-3, Astra scored 62.7% using a standard harness, essentially tied with its predecessor, and 99.9% using a “Provider Adapter” harness that let it retain and reuse prior reasoning instead of restarting cold on every step, according to Augmented Mind and Manolo Remiddi. Same weights. Same model. A 37 point swing from infrastructure alone. That’s not a small footnote. It means a huge share of what gets reported as “model improvement” is actually harness improvement, and leaderboards that don’t control for it are telling a story, not reporting a fact.

Why Cursor just became the most important acquisition of the year

This is exactly why SpaceX’s $60 billion acquisition of Cursor matters more than it looks. Cursor isn’t a model company. It’s one of the best harness companies in the world, the layer that decides how coding models actually get used. Folding it into SpaceX’s GPU fleet is a bet that owning the layer is as valuable as owning the model, maybe more. OpenAI has already cut Cursor’s access to its models over trust concerns tied to Musk’s companies, which tells you the model providers see the same thing: Whoever controls the harness controls the product experience, regardless of which model sits underneath it.

The bottom line

Nobody serious disputes that GPT-6 Astra is a major model. What’s contested is the leap from “major model” to “AGI has arrived.” If you’re evaluating these systems for your own business, ignore the AGI framing entirely. Look at the harness a vendor wraps around the model, ask how it fails, and test it on your actual workflow. The benchmark gap between Astra and Fable 5.1 already proved the leaderboard alone won’t tell you.

AGI, artificial general intelligence, means a system that can match or beat human performance across essentially any cognitive task, not just the narrow ones it was trained for. That distinction matters because it’s exactly what’s being argued about right now.

Jensen Huang has a habit of declaring victory at the exact moment his company needs the market to believe it. So when OpenAI shipped GPT-6 Astra on Sept. 3, Huang posted on X that “AGI has arrived,” adding that “400K GPUs” are “coming online next.”

I’m an AI engineer, and I’m currently building ChatMakr, an AI-powered platform that creates multi-agent chatbots in seconds. I spent the last year watching every model release get relabeled as the one that finally changes everything. Here is the truth.