AI benchmarks plateau when the top models all cluster within a point or two of each other near the top of the scale, so the test stops telling you anything about which model is actually better. It's already happened to some of the most-quoted scores in AI marketing, and it's a big part of why two tools with "nearly identical" benchmark numbers can feel completely different the moment you use them for real work.
Short answer: AI benchmarks plateau when top models cluster near a test's ceiling, leaving no room to show real differences between them. MMLU and GPQA Diamond are already near-saturated, and OpenAI's o3 effectively solved ARC-AGI-1 in December 2024 — which is exactly why ARC Prize launched the harder ARC-AGI-2 three months later.
Last updated: August 5, 2026 · By Vishal Swami, Founder & Lead AI Reviewer, AISagely
I test AI tools for a living, and for the last year I've noticed vendors leaning harder on benchmark percentages right as those percentages stopped being useful. When three chatbots all claim 90%+ on the same test, that number has quietly gone from "signal" to "marketing footnote." Understanding why that happens — and which benchmarks it's already happened to — is the difference between picking a tool off a leaderboard and picking one that actually does the job.
In my testing, I picked two current models that both cite MMLU scores within a point of each other and gave them the same graduate-level word problem, one that also resembles the style of question GPQA Diamond uses. Both produced confident, fluent answers. Only one caught its own arithmetic slip when I asked it to double-check the steps; the other defended the wrong number until I pointed out exactly where it went wrong. The benchmark said these two models were nearly identical. The task said otherwise, which is the whole reason a plateaued benchmark stops being useful for a buying decision.
What "benchmark plateau" means, in plain terms
A benchmark plateaus when it stops separating good models from great ones. Early on, a new test spreads models out — some score 30%, some score 60%, and the gap is meaningful. As labs train specifically to beat that test, scores climb toward the ceiling, and eventually every frontier model is bunched up in the top few points. At that point the benchmark isn't measuring capability anymore; it's measuring who optimized for the test hardest.
The clearest public admission of this came from the team behind Humanity's Last Exam, a 2,500-question benchmark released January 24, 2025. Their own paper states the reason for building it plainly: popular tests like MMLU had already reached the point where "LLMs now achieve over 90% accuracy," which the researchers said was "limiting informed measurement of state-of-the-art LLM capabilities," per the Humanity’s Last Exam paper. In other words, one of the field's own benchmark teams built a harder test specifically because the old one had stopped working.
How three real benchmarks actually plateaued
1. MMLU: the 90% ceiling
MMLU (Massive Multitask Language Understanding) covers 57 subjects, from law to anatomy to abstract algebra, and for years it was the standard "how smart is this model" number. That's exactly why it broke. Once frontier labs started training against it directly, scores kept climbing until, by the numbers cited in the Humanity's Last Exam paper itself, the entire top tier of models sat above 90% — a gap of a point or two, not the double-digit spreads that used to separate a strong model from a mediocre one.
2. GPQA Diamond: built Google-proof, still under pressure
GPQA Diamond was designed to resist exactly this problem. Its 198 graduate-level science questions were built so that skilled non-experts, given unrestricted web access and over 30 minutes per question, average only about 34% accuracy, while PhD-level domain experts land around 69.7%, according to the original GPQA paper. That gap was meant to hold for years. Instead, frontier models started closing in on the expert ceiling within about two years of the test's release, and the benchmark is now doing less to separate this year's top models than it did in 2024.
3. ARC-AGI: solved once, then rebuilt harder
ARC-AGI is the sharpest example of a benchmark actually being retired mid-life. On December 20, 2024, OpenAI's o3 scored 87.5% on the ARC-AGI-1 semi-private evaluation in its high-compute configuration (75.7% in the efficient configuration) — a result ARC Prize itself called "a genuine breakthrough" and "a qualitative shift in AI capabilities," while creator François Chollet added that "o3 is not AGI yet," per ARC Prize’s own write-up. That score was high enough that the original test had lost its purpose. Three months later, on March 24, 2025, ARC Prize launched ARC-AGI-2 specifically to restore headroom. On the new version, o3's low-compute configuration scored just 4%, while the human baseline — defined as at least two people solving every task in under two attempts — sat at 100% for a human panel and roughly 60% for an average individual, according to ARC Prize’s ARC-AGI-2 announcement. Solving a task also cost humans about $17 versus roughly $200 for the AI systems that attempted it — proof the new test still had real distance left to cover.
Questions to ask instead of trusting the leaderboard
These are the prompts and checks I actually run before I believe a benchmark claim in a vendor's marketing page:
- Ask which version of the benchmark: "Is this the original ARC-AGI-1 score or ARC-AGI-2? Is this MMLU or MMLU-Pro?" A saturated version and its harder successor get reported side by side more often than you'd expect.
- Ask for the gap, not the score: "What's the spread between the top five models on this benchmark?" A 90.1% versus 89.8% headline isn't a real advantage.
- Ask what the human baseline is: Benchmarks with a stated expert baseline (like GPQA's 69.7%) tell you whether a model score of 75% is genuinely superhuman or just decent.
- Run your own task instead: "Take this exact question from my actual job and answer it the way I'd need to ship it." A model's leaderboard rank doesn't always predict how it handles your specific work.
Common mistakes people make reading benchmark scores
The mistake I made for longer than I'd like to admit was treating a 2-point benchmark gap as a real capability difference, the same way I'd treat a 20-point gap on a benchmark that wasn't saturated yet. It isn't the same thing, and vendors know it — that's why saturated-benchmark comparisons show up in marketing decks more than the harder, less-flattering ones. Second, people compare a model's score on ARC-AGI-1 against a rival's score on ARC-AGI-2 as if they're the same scale; they aren't, and the second one is dramatically harder. Third, a lot of buyers skip the human baseline entirely, so a 40% score on Humanity's Last Exam sounds low when experts themselves would only hit somewhere in the neighborhood of that range on the hardest questions. Fourth, nobody re-checks whether the benchmark that impressed them a year ago is still meaningful today — plateaued benchmarks don't get a warning label when they stop mattering.
Saturated vs. still-useful: a quick comparison
| Benchmark | Launched | 2026 status | What it tells you now |
|---|---|---|---|
| MMLU | 2020 | Saturated — top models cluster above 90% | Little; use MMLU-Pro or a harder test instead |
| GPQA Diamond | Nov 2023 | Nearing the expert ceiling (~69.7%) | Some signal, less than in 2024 |
| ARC-AGI-1 | 2019 | Effectively solved (o3: 87.5%, Dec 2024) | Retired for frontier comparisons |
| ARC-AGI-2 | Mar 2025 | Wide open (o3: ~4% at launch vs. 100% human panel) | Strong signal — big gaps remain |
| Humanity's Last Exam | Jan 2025 | Early — built specifically to outlast MMLU-style saturation | Strong signal today |
Tools that make this easier to judge
Once you stop trusting the raw benchmark number, the honest fallback is testing the tool yourself on work you actually do — which is the same approach behind my best AI models roundup, where I ran the same email rewrite, the same buggy function, and the same live-news question through ChatGPT, Claude, and Gemini rather than ranking them by leaderboard score. My AI tool ratings page and how we test AI tools guide both explain the method in more detail if you want to run a similar comparison on your own shortlist. If you're weighing two specific models against each other, Claude vs. ChatGPT is a closer, task-by-task version of exactly this exercise, and Claude Opus 5 covers what changed in Anthropic's most recent frontier release. This same skepticism applies outside benchmarks too — I found a nearly identical gap between vendor claims and measured reality when I dug into AI productivity gains being closer to 10% than 10x, and the pattern shows up again in the AI productivity illusion.
My take
When AI benchmarks plateau, the mistake isn't in the test — it's in still quoting it like it's news. A saturated benchmark isn't dishonest by itself: MMLU was a genuinely useful test for years, and GPQA Diamond still tells you something. The problem is that vendors keep citing them past the point where the numbers mean anything, because a 92% still sounds more impressive on a slide than "we're within noise of every other frontier model." My honest verdict: treat any benchmark where the top models are separated by a point or two as retired for comparison purposes, and give real weight only to tests — like ARC-AGI-2 or Humanity's Last Exam right now — where a meaningful gap between models and against the human baseline still exists.
Frequently Asked Questions
What does it mean when an AI benchmark "plateaus" or is "saturated"?
It means the top models have all converged near the top of the scale, usually within a point or two of each other, so the test no longer separates a genuinely stronger model from a slightly weaker one. The benchmark still runs; it just stops producing useful information.
Which AI benchmarks are already saturated in 2026?
MMLU is the clearest case, with frontier models clustered above 90%, per the Humanity's Last Exam paper's own framing. GPQA Diamond is close behind, nearing its PhD-expert baseline of about 69.7%. ARC-AGI-1 was effectively solved when OpenAI's o3 scored 87.5% on it in December 2024.
Why did ARC-AGI need a second version, ARC-AGI-2?
Because ARC-AGI-1 stopped separating frontier models once o3 scored 87.5% on it. ARC Prize launched ARC-AGI-2 on March 24, 2025, and the same o3 configuration scored only about 4% on the harder version, restoring a real gap between AI systems and the 100% human-panel baseline.
Can I still trust a benchmark score when I'm comparing AI tools?
Only if there's still a meaningful gap between the models you're comparing, and ideally a stated human baseline to anchor the number against. If two tools are within a point or two of each other, the benchmark isn't the deciding factor — testing both on your actual task is.
How long before today's hardest benchmarks, like Humanity's Last Exam, plateau too?
There's no fixed timeline, but the pattern in this piece — MMLU, then GPQA, then ARC-AGI-1 — suggests it happens once frontier labs start training specifically against the test. Humanity's Last Exam and ARC-AGI-2 were both built with that failure mode in mind, which is why they still show real separation between models today.