Frontier Benchmarks & Evaluation Trends

Companion page to AI Model Benchmarking, which covers the composite indices used for model and provider selection. This page covers the individual benchmarks those indices are built from — what each one actually measures, how to read a leaderboard without being misled, and where evaluation is heading.

Public benchmarks and your own evaluations answer different questions, and confusing the two is the most common evaluation mistake:

A benchmark score is evidence about a model’s general capability. It is never evidence that the model works for your workload.

The Saturation Treadmill

Benchmarks have a life cycle. A new one separates frontier models cleanly; two years later, everyone scores in the high nineties and the differences at the top are noise. Each generation has been built to escape the ceiling of the last:

    flowchart LR
    A["Knowledge recall\nMMLU, HellaSwag"] --> B["Hard reasoning\nGPQA Diamond, AIME"]
    B --> C["Frontier difficulty\nHLE, FrontierMath, ARC-AGI-2"]
    C --> D["Environments & agents\nSWE-bench, Terminal-Bench, OSWorld"]
    D --> E["Economic value\nGDPval, domain suites"]
    style A fill:#f1f5f9,stroke:#64748b
    style E fill:#f5f3ff,stroke:#7c3aed
  

A saturated benchmark is not worthless — it becomes a regression smoke test. A model that drops on MMLU has broken something. It just no longer ranks the top of the field.

ARC-AGI — the fluid-intelligence line of benchmarks

Most benchmarks test what a model knows. ARC-AGI tests whether it can acquire a new skill on the spot: every task is a handful of input/output grid pairs illustrating a rule the model has never seen, and it must infer the rule and apply it to a held-out grid. Because each puzzle is novel by construction, memorizing the training set does not transfer — which is exactly why it stayed unsolved long after knowledge benchmarks fell.

Two things make ARC-AGI worth watching beyond the headline score:

• Cost per task is reported alongside accuracy. A system that brute-forces a puzzle with thousands of sampled programs is not doing the same thing as one that solves it in a single pass, and the leaderboard makes that visible instead of hiding it. This is the clearest mainstream example of efficiency-normalized evaluation.
• A human baseline is measured, not assumed. Tasks are validated on real people, so “AI vs. human” comparisons on ARC are grounded rather than rhetorical.

ARC-AGI-2 has since followed its predecessor up the curve — the snapshot below puts the top of that leaderboard above the benchmark’s own grand-prize threshold, roughly eighteen months after release. ARC-AGI-3 is the more informative track today. It is the shift the whole field is making in miniature: from answer this question to operate in this environment and figure out what the goal even is.

OSWorld 2.0 — evaluating computer use

Computer-use evaluation asks a different question from every text benchmark: not can the model answer, but can an agent operate a machine. The agent is dropped into a real virtual machine, sees screenshots, and acts with mouse and keyboard; grading is execution-based, meaning a validator script inspects the final state of the machine — the files written, the rows in the spreadsheet, the setting actually changed — rather than reading what the agent claimed it did.

OSWorld is the reference benchmark for this, and its three releases track how the whole field’s evaluation practice has matured:

What makes 2.0 worth studying as an evaluation method, separate from its scores:

The reason to care about the internals: on OSWorld 2.0 the same run can be reported two ways. At a 500-step budget, published results put the leading agents near 31% binary completion but roughly 68% partial score — the agent gets most of the way through most workflows and then fails to finish. Both numbers are honest; quoting either without saying which is not. This single benchmark is the clearest available demonstration of why “what does this number mean” beats “whose number is higher”.

For infrastructure teams, the practical cost of this style of evaluation is real: every task needs a fresh VM snapshot, isolation between runs, and a budget for hundreds of model calls per task. Running a computer-use suite internally is a platform project, not a script — which is the same conclusion the trends section below reaches from the other direction.

Benchmark map by capability

When a model is announced, the scores quoted are chosen by the vendor. This is the fuller map to check against:

Snapshot — where the numbers stood in September 2026

Scores below were compiled on 2026-09-06 from the public leaderboards linked in each row. They are included to show the shape of the field, not as a citable ranking: frontier numbers move weekly, aggregators disagree, and the entry at the top of any row will likely be stale by the time you read it. Follow the link before quoting a figure.

Four things this table shows better than any argument:

• The treadmill is fast. ARC-AGI-2 was built in 2025 to be near-zero for models and is now above its own 85% grand-prize threshold, against an average individual human score of 66%. FrontierMath Tier 4 — the hardest tier of a benchmark designed to last — reads the same way at the top.
• Protocol matters more than model at the top. Humanity’s Last Exam spans roughly 46% to 65% for the same model family, depending on whether search, browsing, and code execution are allowed. A number without its protocol is not a result.
• Aggregators disagree. SWE-bench Verified appears as 96% or 97% depending on whose harness ran it, and the top several models sit within about one point — inside the noise of the harness itself.
• The unsaturated benchmarks are the interactive ones. ARC-AGI-3 and long-horizon computer use still have real headroom — on OSWorld 2.0, agents finish under a third of the workflows outright even while scoring around 70% on partial credit, and the gap between first and second place there dwarfs anything on the coding leaderboards.

Snapshot — a five-model comparison on the newest suites (October 2026)

The September snapshot above follows single benchmarks across public leaderboards. This one is a different kind of artifact: one table, seventeen benchmarks, eight capability areas, assembled by whoever made the chart. It is worth reading twice — once for the numbers, once for how the table is built.

Reading the table

• Fifteen of seventeen rows go to Gemini 4 Carbon, but several of those leads are thin. On four rows the margin is a point or less — SWE-sweep (+0.1), Vals Index (+0.2), Chartography (+0.9), AA Index (+1) — small enough that a different harness or a rerun could reorder them. The gaps large enough to matter are ExploitGym (+12.4 over the next model), GraphWalks (+12.0), Terminal-Bench 4.0 (+5.6), and Agents’ Last Exam (+4.3).
• The two rows it trails are the two the chart itself highlights: FrontierSWE 2, where GPT-6 Astra leads 65.5% to 61.2%, and the OSWorld-2.1 offline subset, where Claude Opus 5.5 leads 87.2% to 81.5%.
• The newest suites are the ones with headroom. SWE-sweep tops out at 9.6%, AutomationBench at 53.9%, ExploitGym at 54.8%. On Vending-Bench 2 the best ending balance is $16,484, against Andon Labs’s own estimate that roughly $63K is achievable over the simulated year — the leader is nowhere near the ceiling.
• Benchmarks get harder by versioning, and the version is part of the score. Nine of the seventeen rows carry a version or revision in their name. The September snapshot has Terminal-Bench 2.0’s leader at 91.9%; the best score on Terminal-Bench 4.0 here is 65.2% — a harder test, not a worse model, and likely a different harness, since “(AA)” presumably marks Artificial Analysis’s own run. Going the other way, the OSWorld-2.1 offline subset tops out at 87.2% while the OSWorld 2.0 leader above sits at 72.6%: a subset of a newer revision, not the same test. A score without its benchmark, version, subset, and harness is not reproducible.
• The comparison set is the chart author’s choice. Models that lead rows elsewhere on this page — GPT-5.6 Sol, Claude Mythos 5, Claude Opus 5 — are not in the table. Five cells are also blank, four of them for Claude Fable 5.1; a dash means not reported, not zero.
• Reasoning effort is part of the result, and the table does not state it. An article citing a different version of the Artificial Analysis index (v4.3.2) reports Gemini 4 Argon at 52.56 on high effort and Claude Opus 5.5 at 57.62 at max effort — the same model scores differently at different settings. The table’s 50 and 58 for those two models are in the same order at a different version, which is mildly reassuring and proves nothing. The source is an aggregator, so weigh it accordingly.

What the newer suites measure

For nine of the seventeen rows, no primary description could be found. That is worth knowing before quoting them: a score on a benchmark whose task, grader, and subset you cannot read is a number, not evidence.

If you copy a number from a chart like this into a decision, copy its benchmark, version, subset, harness, and reasoning-effort setting beside it — and re-run the part that matters on your own tasks.

Reading a leaderboard without being fooled

The practical rule is triangulation: a static academic benchmark, a human-preference arena, and an agentic suite. Where all three agree, the ranking is probably real. Where they disagree, the disagreement is usually the interesting information.

Where evaluation is heading

Applying this in practice

    flowchart TD
    A["Frontier leaderboards\n(capability shortlist)"] --> B["Filter by task type\ncoding / agentic / reasoning"]
    B --> C["Cross-check independent eval\n+ cost and latency"]
    C --> D["Run your own eval set\n20-50 real cases"]
    D --> E{"Beats what we ship\ntoday?"}
    E -->|"Yes"| F["Ship behind a version pin"]
    E -->|"No"| B
    F --> G["Keep the eval set running\nas a regression gate"]
  

Checklist

• Identify which two or three benchmarks actually correspond to your workload — ignore the rest of the announcement
• Confirm the score is pass@1 under a harness comparable to the one you will run
• Record the benchmark, version, subset, harness, and reasoning-effort setting next to every score you copy into a document
• Check the same model on an independent evaluator, not only the vendor’s own numbers
• Read cost and latency alongside accuracy before shortlisting
• Verify against a contamination-resistant benchmark if the task resembles a well-known public dataset
• Replace the leaderboard with your own eval set before the release decision
• Re-run that eval set on every model and prompt version change — see Prompt & Context Design

Related Categories

• 📊 AI Model Benchmarking — composite indices, provider comparison, openness
• ⚙️ Model Selection & Tuning — turning a shortlist into a deployed model
• 🛡 AI Governance Overview — hallucination monitoring and regulatory evidence