There are now more AI benchmarks than any one person can hold in their head — and nobody knows where they all are. A single evaluation might have a paper on arXiv, a dataset on Hugging Face, a harness on GitHub, a row in one leaderboard, and a mention buried in a model card. Finding the right benchmark for a task, or checking whether the one you're designing already exists, means rummaging through half a dozen disconnected places. This paper's answer is Benchmark Radar: a search engine for benchmarks themselves, refreshed daily from 37 public sources.
The system runs on two streams that deliberately never merge into one number. A catalog keeps 1,283 benchmark records from four sources (LLM Stats, OpenCompass Hub, Artificial Analysis, and model reports), preserving every record's source identity, citations, and score history — even when a record has no score at all. A discovery feed collects dated snapshots of new papers, repos, datasets, and releases (11,068 observations across 46 daily snapshots). The two are joined by shared benchmark IDs and a transparent BM25F search ranking that shows you exactly which words matched, in which fields, and how the score was composed — no semantic black box.
The most quietly alarming result is the census itself. Of 790 records with numeric scores, only 82 declare a percentage unit with a known direction and values in range — the minimum bar for even defining "headroom to 100." And 668 of 1,283 records have no recorded release date, while not a single one of the 12,916 score observations carries an actual evaluation date. The paper's lesson: before you compare two benchmark scores, you have to be able to find the settings behind them. Now, for the first time, you can.
GPT-6 Astra and Claude Fable 5.1 are built for coding, research, and multi-tool work. Comparing them requires benchmarks — but first you have to find the benchmarks.
The paper opens with a problem every evaluation researcher knows: finding a benchmark, its dataset or repository, and reports of its use means searching paper servers, code hosts, dataset hubs, vendor releases, blogs, and catalogs. Existing resources — LLM Stats, OpenCompass, Artificial Analysis — each hold a piece of the picture. Connecting a newly released benchmark to its task materials, and then to its later use in model reports, "can still require consulting several separate resources."
Benchmark Radar's answer is to ingest all four and refuse to flatten them. The v0.11.0 catalog holds 1,283 source records — one record per benchmark per source — carrying 12,916 numeric score observations on 790 records. Crucially, the 493 records without numeric scores are not deleted: 464 of them still link to at least one paper, repository, or dataset. In the full catalog, 475 records link to papers, 506 to repositories, and 293 to datasets.
| Rs | source records from source s (one benchmark entry from one contributing source) |
| |r| | 687 + 461 + 25 + 110 = 1,283 — a record remains if a score, date, or citation is absent |
The four sources contribute in opposite directions. OpenCompass Hub brings 461 records and zero scores; Artificial Analysis brings 25 records and 7,050 scores — a median of 245 scored models per benchmark, peaking at 586 for GPQA Diamond. Any "average benchmark" statistic that mixes these populations without care is meaningless.
The paper's discipline is to keep the units separate: records ≠ benchmarks ≠ scores ≠ models. Twenty-two records carry scores for at least 100 distinct models each.
A daily radar and a stable catalog pull in different kinds of evidence. The whole design lives or dies on keeping them separate — then joining them by ID.
Figure 2 of the paper separates two input paths. The benchmark catalog is populated by registries and model reports: 1,283 source records with normalized names, identifiers, artifact links, score observations, model identities, and cited documents. The discovery history is a stack of dated snapshots — mentions, releases, and updates found across public sources, linked to artifacts by exact identifiers such as DOIs, arXiv IDs, and repository URLs.
The join happens in a shared evidence layer: stable source-record IDs, scores, models, citations, and links back to discovery evidence. Clients — the web dashboard and the offline CLI — consume the same benchmark IDs and the same JSON response format. A shared query service supplies both interfaces, so "what you see on the web" and "what you query locally" cannot drift apart.
| o | a discovery observation (one collected record: a mention, release, or update) |
| a | an artifact: a paper, repository, dataset, release, or page |
| o ∼ a | linked by exact identifier — never by fuzzy match; 59 of 6,546 artifacts have multi-source observations |
The same six reader questions route through different evidence scopes: Today (what appeared or changed), Search (which benchmarks match this task — the complete catalog, unscored included), Leaderboard (which evaluations combine lower scores with broader use), Saturation (score histories and settings), Blog (a daily brief), and CLI (the same evidence, offline).
Display filters change only what a view shows; search and exports always retain the full catalog. The census in the paper applies none of the display filters.
Every day, 37 sources get swept in a 48-hour window. Most of what comes back is thrown away — and the run only publishes if three core sources are healthy.
Daily discovery searches a rolling 48-hour window across 13 direct connectors and 24 first-party research and engineering feeds, recording counts and errors by source, removing future-dated rows, and requiring healthy core sources before publication. At the cutoff the core three were arXiv, the Hugging Face Hub, and GitHub Search — all must be healthy before a snapshot can publish. Optional routes may fail without blocking: at the cutoff, Semantic Scholar returned a malformed payload and Brave Search lacked an API key, both recorded in the snapshot.
The 2026-09-07 snapshot shows how brutal the filter is: 1,003 fetched rows became 954 candidates after duplicate removal; only 366 qualified for publication, and 138 met the recommendation threshold. Cumulatively the feed holds 11,068 observations and 6,546 artifacts across 46 snapshots (4 of them simulated historical backfills). Five source labels account for 88.0% of all observations: Hugging Face 3,487 · GitHub 2,470 · arXiv 1,945 · Semantic Scholar 1,039 · OpenAlex 802.
| R | relevance — match to benchmark-tracking intent |
| E | evidence — does the observation link artifacts (paper, repo, dataset)? |
| C | recency — freshness within the 48-hour window |
| A | adoption — observed traction of the underlying artifact |
| P ≥ τ | meets the recommendation threshold τ; at the 2026-09-07 cutoff, 138 of 366 qualified items cleared it |
Source caps and collection failures shape the mix — GitHub Search was at cap (300 rows) at the cutoff, and five labels carry 88% of everything. The system's honesty about this is the feature: failed optional sources are reported in the visible coverage notice rather than silently dropped.
The discovery feed's job is evidence, not totals — those 11,068 observations add exactly nothing to the 1,283-record benchmark count.
Most search engines ask you to trust the model. Benchmark Radar uses BM25F — a 2004 formula where every point of every score can be traced to a word, a field, and a weight you can inspect.
Catalog retrieval is lexical: BM25F, the field-weighted extension of BM25 from Robertson, Zaragoza & Taylor (2004), "with bounded boosts for name and phrase matches." Each result exposes matched and missing query words, the fields they occur in, and the score components. Source membership does not change the ranking — a record from a small source competes on equal terms with one from a big one.
The design stance is that candidate retrieval precedes suitability judgment. An analyst or agent tries focused query variants, inspects a record's tasks and score settings, and follows its citations. The engine's job is to surface candidates with their evidence attached — not to decide relevance for you.
| D, Q | a benchmark record (its concatenated weighted fields) and a query |
| wf | per-field weights — the "F" in BM25F; name matches should dominate |
| k1, b | term-frequency saturation and length-normalization constants |
| βname, βdesc | bounded boosts — capped additive bonuses for full-phrase matches (the paper's "bounded boosts for name and phrase matches") |
| N, df(t) | corpus size and document frequency of term t |
Lexical matching has a known cost, and the paper says so plainly: it can miss paraphrases and renamed tasks, and evaluating semantic retrieval "will require reviewed relevance judgments." In the worked example (Chapter 9), the agent supplemented local queries with web searches for exactly this reason.
But auditable retrieval is the right default for prior-art search, where a silently missed candidate is a false "we're first" claim. When the engine can't find your words, it shows you that too.
Is SWE-bench a coding benchmark or an agent benchmark? Force one label and you lose 117 agentic benchmarks from every count of agentic evaluation.
The census classifies 1,279 of 1,283 records into 11 top-level capability domains and 63 sub-domains, using the publishers' own fields — OpenCompass Hub dimensions, LLM Stats categories, Artificial Analysis categories, and model-report registry domains. The 4 unclassified records are LLM Stats community rows whose crawl supplied no description, category, or modality; the paper records that reason rather than guessing from titles.
The design tension: interaction paradigm and input modality are recorded as facets — properties held beside a record's domain, not inside it. Every record carries exactly one Level-1 class and, independently, any number of facet values, so facet counts don't sum to the population. The paper's example is exact: "a benchmark that resolves repository issues is a coding benchmark run as an agent, not an agent benchmark." Of 345 records with the agentic facet, only 128 take Agentic & Tool Use as their Level-1 class; 117 sit under Coding & Software Engineering, with the rest spread across 6 further classes.
| nagentic | 345 records carry the agentic facet (any Level-1 class) |
| one-axis count | 128 — records whose Level-1 class is Agentic & Tool Use |
| hidden agents | 117 under Coding & Software Engineering + 100 spread across 6 further classes |
Every "how many agent benchmarks shipped this year?" statistic you've read depended on this labeling choice, and most datasets make it silently. With a facet scheme, the claim becomes inspectable: you can list the 117 coding benchmarks that are run as agents and check them yourself.
The paper is equally careful about the limits: task-capability classification "remains unvalidated in this rebuild," and a deterministic null extractor assigns no capability levels to its 5,863 discovery-derived tracks.
668 of 1,283 records have no release date. And of 12,916 score observations, not one carries the date the evaluation actually ran.
Benchmark release dates are known for 615 records. The dated mass clusters heavily in recent years: n=49 records in 2023, n=179 in 2024, n=279 in 2025 — with the remaining 108 dated records spread across all other years. Figure 6 keeps the 668 undated records in a separate column with its own scale rather than leaving the figure, because a year's share "is read only where the evidence supports one."
Even that isn't enough. The OpenCompass Hub crawl stops at the discovery cutoff, so the truncated final year is drawn mostly from model reports — its share would measure a change of catalog, not a change in the field. So the paper additionally requires a year's source mix to stay close to the pooled mix, and checks: reweighting each year to a common source composition moves the agentic share by at most 1.1 percentage points over the reported years. The rise in agentic benchmarks is real, not an artifact of who supplied which year.
| y | release year with n2023=49, n2024=179, n2025=279 dated records |
| ̄ws | the pooled source composition (fraction of dated records from each of the 4 catalog sources) |
| ≤ 1.1 pp | the paper's empirical bound: the agentic rise survives reweighting to a common source mix |
Any trend claim about benchmarks — "agentic evals are exploding," "multimodal is saturated" — is a claim about dates. This catalog makes the evidence base visible: which half is dated, which mix each year draws from, and which date basis each score carries.
The punchline for saturation research: measuring it "requires comparable test versions and settings, with dates tied to score reporting or evaluation." In this catalog, that last part doesn't exist yet.
12,916 numbers, and only 82 records' worth can even define "distance from 100." The rest are scores without a scale you can verify.
Section 3.5 sets a deliberately strict eligibility rule. To summarize a record on a percentage scale, the census requires: a declared percentage unit, a known score direction, and numeric values within 0–100. Rescaling a displayed value or reading an aggregator's declared maximum does not establish the unit. And matching scales still don't establish matching test versions, prompts, tools, attempts, or evaluators.
Under that rule, only 82 of 790 scored records qualify. The remaining 708 keep their numeric observations — they're not deleted — but they don't support a shared percentage-headroom calculation, and the 493 unscored records have unknown headroom by definition. Headroom itself is a narrow calculation: $100 -$ a record's highest value, only when the three gates pass.
| unitr = % | declared in the source — a displayed multiplier or an aggregator's maximum does not establish it |
| direction | higher-is-better vs. lower-is-better must be recorded; a numeric maximum need not be the best result |
| 0 ≤ v ≤ 100 | every numeric observation in range |
| 82 / 708 / 493 | eligible / other-or-unverified-scale / no numeric score — all 1,283 records remain accounted for |
| Record | Max |
|---|
When the next paper claims "benchmark X is saturated — only 4 points of headroom," this census is how you check whether those 4 points mean anything. A small gap to a ceiling can describe a reported setup; catalog-wide headroom estimates require the eligible measurement coverage alongside the statistic.
The deeper point is evidentiary: scores are claims, and claims have scopes. Test versions, reasoning budgets, tools, attempts, and evaluators must still be checked before comparing results — the census tells you which records even start that conversation.
Scores alone mislead. The Leaderboard's answer: plot reported score against measured use — how many distinct models have actually been scored — and trace the Pareto frontier.
The Benchmark Frontier view (Figure 3) combines reported scores, scored-model counts, and benchmark release dates or first-score proxies. Gold rings mark Pareto candidates: benchmarks no other record beats on both axes. Hollow marks flag unverified scales or counts — those stay visible but are kept outside the Pareto calculation. Dotted outlines flag model-release date proxies. The view requires a numeric score, excludes known pre-2024 benchmarks, and starts from a score cutoff of 70.
Why "measured use"? Because a benchmark's score in isolation says nothing about whether anyone relies on it. The Artificial Analysis record for GPQA Diamond carries scores for 586 distinct models; Humanity's Last Exam and SciCode for 577 each; CritPt for 492. Median scored-model counts tell the same story of skew: 245 for Artificial Analysis records, 3 for LLM Stats, 2 for model reports. Twenty-two records have ≥100 distinct scored models.
| sr | reported score of record r (highest numeric value on the source's native scale) |
| mr | distinct scored models (measured use) — counted by source model ID within the record |
| filter | numeric score required · known pre-2024 excluded · initial cutoff s ≥ 70 · unverified scales excluded from Pareto (shown hollow) |
The frontier is the paper's alternative to the naive leaderboard: it answers "which evaluations combine lower scores with broader measured use" — the benchmarks where headroom and evidence actually coexist. It also encodes the honesty rules from Chapter 7 as geometry: unverified scales hover at the edge, visible but unable to define the frontier.
Note the frontier's own date caveat: positions use "benchmark release dates or first-score date proxies" — and we know from Chapter 6 how thin dated coverage is.
A contributor is about to design a new evaluation of credit assignment in agentic training. Before building it, a coding agent interrogates the catalog: does this already exist?
Section 5 walks through a complete prior-art search. The requirement: small Qwen-series models as reproducible baselines, work published in August 2026. A coding agent installed the Benchmark Radar client and its public Skill, downloaded the corpus, and searched locally — inspecting recorded paper, repository, and dataset links, then supplementing with web searches for work described in different terms (lexical matching can miss paraphrases). Five works surfaced: SRPO, ContextPilot, SkillGate, CIPO, and MoRSE — each a different attack on credit assignment, each using different agentic benchmarks.
The workflow's shape matters more than the specific hits: candidate retrieval is separated from comparison. Benchmark Radar retrieves candidates and exposes their evidence; the researcher or agent judges relevance and compares designs. The paper also shows the counterfactual: for an earlier benchmark-design effort, one contributor hand-assembled an 11-row prior-art comparison table (MLE-Bench, MLGym-Bench, EXP-Bench, ResearchCodeBench, MLR-Bench, PaperBench, AstaBench, InnovatorBench, AIRS-Bench, COMPOSITE-Stem, ScienceBoard) — "more effort than any part of that project except producing the benchmark data."
This is the whole system compressed into one use case: discovery found the papers, retrieval surfaced them with visible word matches, evidence links let the agent read the sources, and the judgment — is this the same idea? — stayed with the human. The paper is careful not to overclaim: retrieval precision, task suitability, and time saved "remain to be evaluated"; the example has no controlled baseline.
But the shape of the claim is right: the way to prevent duplicate benchmarks is not better memory — it's a searchable, source-preserving, auditable index.