Intelligence Measurement

A benchmark is three things wearing one name: a fixed set of items, a rule for scoring an answer, and a protocol for administering it. A score is a statistic computed from a sample of items under a protocol. It is not a property of the system, and it becomes a statement about the world only when the protocol travels with it. Strip the protocol and what remains is a number that cannot be checked, compared, or falsified — which is the condition most published capability figures are in.

This brief is the Institute's instrument for the rest of Category IV. When artificial general intelligence, artificial scientists, artificial creativity or multi-agent intelligence systems quotes a benchmark number, the questions that decide whether the number means anything are the ones set out here: which evaluation split, how many attempts, at what inference compute and what dollar cost per task, who ran it, and whether the items existed publicly before the system was built. Five questions. Most reported figures answer none of them.

The field's signature failure mode is not fraud. It is under-specification, compounded by the fact that the organisations producing the most-quoted numbers are frequently the organisations that built the systems being measured. What follows takes the measured record seriously, sets out what contamination does and how it is detected, asks whether a score measures the construct printed on its label, and ends at the exotic question the category actually turns on: whether machine general intelligence is the kind of thing that admits a scale at all, and what an instrument that survived contact with a new model would have to look like.

Established An evaluation is an experiment, and the field reports it as though it were an observation. Miller's 2024 methods note, written at Anthropic, states the problem in one sentence: “Fundamentally, evaluations are experiments; but the literature on evaluations has largely ignored the literature from other sciences on experiment analysis and planning” (arXiv:2411.00640). Established The paper supplies what is missing — treat the eval's questions as a sample from an unseen population, compute a confidence interval, use paired-difference estimators when comparing two systems on the same items. Established The overwhelming majority of published benchmark tables in this cluster report point estimates with no interval at all, so the reader cannot tell an eight-point gap that is real from a two-point gap that is noise. Established And the people best placed to know say the same thing about reproducibility: Biderman, Schoelkopf, Sutawika, Gao and twenty-six co-authors — the maintainers of the Language Model Evaluation Harness — name three obstacles by title, “sensitivity of models to evaluation setup, difficulty of proper comparisons across methods, and the lack of reproducibility” (arXiv:2405.14782).

Established MMLU defined the era and has an error floor that bounds what any score on it can mean. Hendrycks, Burns, Basart, Zou, Mazeika, Song and Steinhardt introduced it in 2020 across 57 subjects; at publication the largest GPT-3 model beat random chance by almost twenty percentage points on average, with “lopsided performance” and models that “frequently do not know when they are wrong” (arXiv:2009.03300). Established Four years later Gema, Leang, Hong and colleagues re-annotated it and found that 6.49% of MMLU questions contain errors, rising to 57% of the analysed Virology subset, releasing 5,700 corrected items as MMLU-Redux and reporting “significant discrepancies with the model performance metrics that were originally reported” (arXiv:2406.04127). Established MMLU-Pro, built as the harder successor, supplies the other half of the diagnosis: accuracy falls 16% to 33% relative to MMLU, and prompt sensitivity falls from 4–5% in MMLU to just 2% (arXiv:2406.01574). Established That second figure is the more consequential one, because it quantifies how much of an MMLU score was never about the model: up to five points were harness artifact. Established An MMLU number quoted without its shot count, its prompt template and its answer-extraction method — log-likelihood ranking or free generation — is not a measurement of anything.

Established Human baselines are the weakest link in the chain, not the model scores. GPQA is the clearest case because its authors did the work properly: 448 questions, PhD-level experts scoring 65% (74% after discounting mistakes they identified in retrospect), non-expert validators scoring 34% “despite spending on average over 30 minutes with unrestricted access to the web,” and GPT-4 at 39% at publication (arXiv:2311.12022). Established The 65% figure is the one that matters and is almost never quoted alongside model scores. Established Elsewhere the baselines are wildly heterogeneous: humans reach 92% on GAIA (arXiv:2311.12983), while ARC-AGI-2 reports 407 participants across 515 sessions producing 13,405 test-pair attempts of which 8,277 (62%) were solved, or 75% when aggregated by task. Established Those are three different populations under three different time budgets, and comparing a model to “human performance” without saying which is an equivocation, not a result. Established The canonical illustration is the bar exam: OpenAI's GPT-4 technical report claimed a simulated bar exam score “around the top 10% of test takers” (arXiv:2303.08774), and Martínez's re-analysis in Artificial Intelligence and Law recomputes roughly the 62nd percentile against first-time takers, roughly the 48th against licensed or license-pending attorneys, and on the essay components roughly the 15th percentile against those who actually passed, concluding that the original “estimates of GPT-4's UBE percentile are overinflated.” Established The error is not that the number was wrong. It is that a percentile was published with no reference population named.

Established Contamination is the field's signature failure mode, and the detection methods do not close the question. Yang, Chiang, Zheng, Gonzalez and Stoica showed that “simple variations of test data (e.g., paraphrasing, translation) can easily bypass” string-matching decontamination, that a 13B model can then reach GPT-4-level scores on MMLU, GSM8k and HumanEval by overfitting to rephrased test items, and that 8–18% of HumanEval overlaps with major pre-training datasets (arXiv:2311.04850). Established Li and Flanigan attacked it from the release-date side: models perform “surprisingly better” on datasets released before their training cutoff, and — the sharper finding — “for classification tasks with no possibility of task contamination, LLMs rarely demonstrate statistically significant improvements over simple majority baselines, in both zero and few-shot settings” (arXiv:2312.16337). Established The counterweight is a purpose-built clean benchmark: Zhang and colleagues matched GSM1k to GSM8k on human solve rate, solution length and answer magnitude, and found accuracy drops of up to 8% with systematic overfitting in several model families and a Spearman r² of 0.36 between a model's probability of emitting a GSM8k example and its GSM8k–GSM1k gap — but also that “many models, especially those on the frontier, show minimal signs of overfitting” (arXiv:2405.00332). Established Both results are credible and they point in opposite directions. Frontier The Institute declares the tie and states the terminal position it supports: benchmark scores for closed models are biased upward by an unknown amount, not noise, and no currently available method establishes the size of the bias for a model whose training corpus is undisclosed.

Established The worked example every reader should carry: o3 on ARC-AGI, with both numbers documented inside one paper. The ARC-AGI-2 technical report (Chollet, Knoop, Kamradt, Landers, Pinkard, arXiv:2505.11831) reports, in its Table 1 for the semi-private evaluation set: o3 (Medium) at 53.0% on ARC-AGI-1 and 3.0% on ARC-AGI-2; the 2024 prize-winning ARChitects system at 56.0% and 2.5%; o4-mini (Medium) at 41.8% and 2.4%; o3-mini (High) at 34.5% and 3.0%; Claude 3.7 with an 8K budget at 21.2% and 0.9%. Established The body of the same paper records that the December 2024 o3 preview “achieved scores of 76% (low-compute; estimated cost: $200 per task) and 88% (high-compute; estimated cost: $20,000 per task)” on the ARC-AGI-1 semi-private set — the identical evaluation split on which the shipped model scores 53.0%. Established Nothing about the benchmark changed between those two rows. What changed was the system: a preview configuration that was never shipped, run at a compute setting whose cost per task differs by a factor of a hundred between its own two figures. Established A third number is routinely mixed in: the ARC Prize target of 85% is set on the private set under a compute cap, and the 2024 competition moved private-set state of the art from 33% to 55.5% (arXiv:2412.04604). Established Three different evaluation sets, two different models, two compute regimes, one headline. Frontier Note also that the widely circulated pair “75.7% / 87.5%” does not appear in the ARC-AGI-2 paper, which reports 76% and 88%; the ARC Prize announcement that is the presumed origin of the more precise figures could not be obtained for this brief, and the discrepancy is unresolved. Speculative And the question that decides what the result means at all — whether the ARC-AGI public training set was in o3's pre-training corpus — is asserted confidently in commentary in both directions and established by no source this brief could read. It is open. It is also the whole ballgame, because it determines whether the number measures generalisation or preparation, which is precisely the distinction Chollet built ARC to draw (arXiv:1911.01547).

Established Agentic evaluation is worse than model evaluation on every axis its own critics measure. Kapoor, Stroebl, Siegel, Nadgir and Narayanan document four shortcomings: benchmarks score accuracy while ignoring cost, so agents become “needlessly complex and costly”; the benchmarking needs of model developers and downstream developers have been conflated; many agent benchmarks have no holdout set at all, producing agents that “take shortcuts and overfit to the benchmark”; and there is “a lack of standardization in evaluation practices, leading to a pervasive lack of reproducibility” (arXiv:2407.01502). Established The launch numbers are worth having, because they are the floor the treadmill started from: SWE-bench opened with 2,294 issues across 12 Python repositories and Claude 2 resolving 1.96% of them (arXiv:2310.06770); GAIA opened with GPT-4 plus plugins at 15% against a human baseline of 92%. Established Then the integrity results. Aleithan and colleagues found that 32.67% of successful SWE-bench patches involved solution leakage — the answer present in the issue report or comments — and 31.08% passed only because the tests were weak; excluding both, one measured agent's resolution rate fell from 12.47% to 3.97%, and over 94% of the issues predate the models' knowledge cutoffs (arXiv:2410.06992). Established Independently, Liang, Garg and Zilouchian Moghaddam found models identifying buggy file paths from issue text alone at up to 76% on SWE-bench versus up to 53% on repositories outside it, and 35% versus 18% consecutive-5-gram function reproduction (arXiv:2506.12286v4). Established SWE-bench Verified is the single number most often cited as evidence of agentic progress, which makes this the most consequential eval-integrity finding in the category.

Established The most serious attempt at a durable scale measures time, not accuracy — and it renamed itself to say what it covers. METR's paper proposes the “50%-task-completion time horizon”: the time humans typically take on tasks the model completes with 50% success. Established In v2 (arXiv:2503.14499v2, NeurIPS 2025) it reports that “current frontier AI models such as Claude 3.7 Sonnet have a 50% time horizon of around 50 minutes,” that the horizon “has been doubling approximately every seven months since 2019, though the trend may have accelerated in 2024,” and that the gain is “primarily driven by greater reliability and ability to adapt to mistakes.” Established Cite the version. The v2 title is “Measuring AI Ability to Complete Long Tasks”; the v4 title, dated 2026-07-10, is “Measuring AI Ability to Complete Long Software Tasks,” with an abstract identical to v2's. Established The scope narrowed in the title and nowhere else, and the paper's own extrapolation was always conditional: “if these results generalize to real-world software tasks.” Established The human baselines underneath it are the best in the category: HCAST supplies 189 tasks and 563 human baselines totalling over 1,500 hours under conditions identical to the agents', finding agents succeed 70–80% of the time on tasks taking humans under an hour and under 20% on tasks taking over four (arXiv:2503.17354). Frontier That last pair of numbers is the shape of the whole problem: the agent advantage is concentrated at short horizons, and the instrument that measures it is calibrated on software tasks alone.

Established A score without an attempt count and a reliability metric describes an event, not a capability. Brown and colleagues showed that coverage — the fraction of problems solved by any sample — scales log-linearly with sample count over four orders of magnitude, taking DeepSeek-Coder-V2-Instruct from 15.9% at one sample to 56% at 250 samples on SWE-bench Lite, above the then single-sample state of the art of 43% (arXiv:2407.21787). Established In domains without automatic verifiers, majority voting and reward models plateau beyond several hundred samples, so the trick converts compute into score only where an oracle exists. Established The complementary metric is tau-bench's pass^k, the probability that a policy succeeds on all of k independent trials: gpt-4o sits under 50% at pass^1 and collapses to under 25% at pass^8 in the retail domain (arXiv:2406.12045). Frontier Reliability, not accuracy, is the quantity a deployment decision actually needs, and pass^k is the only widely available instrument that reports it.

Established Saturation is real, dated, and documented by the successors' own rationales. The best-sourced saturation marker in the literature is a sentence in Humanity's Last Exam: “LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting informed measurement of state-of-the-art LLM capabilities” (arXiv:2501.14249, v1 dated 2025-01-24). Established MMLU-Pro and MMLU-Redux, both June 2024, are the two independent responses; FrontierMath and ARC-AGI-2 are the current floors, at under 2% and 2.4–3.0% respectively at publication. Established The treadmill runs faster than the literature that describes it: Humanity's Last Exam reached v11 by 2026-07-28, eighteen months after v1; FrontierMath reached seven versions in thirteen months; SWE-bench has three; AgentBench's current version is v3, dated 2025-10-04, two years after v1. Established A brief that cites any of these without a version cites an object that no longer exists, and this brief cites versions for that reason.

Frontier Machine g is now an empirical programme rather than a slogan, and its own authors flag the confound. Ilić and Gignac fitted twelve tests across 591 language models and report “strong empirical evidence for a positive manifold and a general factor of ability,” with parameter count positively but diminishingly related to that factor (arXiv:2310.11616). Frontier Their title asks whether this indicates artificial general intelligence or achievement — in psychometric terms, whether they have found an ability factor or an attainment factor, which is exactly the right question to ask of models trained on overlapping web corpora. Frontier The second programme is Hendrycks and a coalition of co-authors who agree about very little else, grounding a definition of AGI in Cattell–Horn–Carroll theory, decomposing general intelligence into ten core cognitive domains, and reporting composite scores of 27% for GPT-4 and 57% for GPT-5, alongside a qualitative finding of a “highly jagged” cognitive profile with critical deficits in long-term memory storage (arXiv:2510.18212). Frontier Neither instrument has yet been applied, unchanged, to a model released after it was published. Speculative And no published test of measurement invariance across model families — the technical requirement for a factor structure to be the same thing in two populations — surfaced in the research behind this brief. Its absence is the decisive missing experiment for the whole machine-g question.

Frontier Item response theory is half-built in this field and nobody is calling it that. tinyBenchmarks demonstrates that 100 curated examples suffice to estimate MMLU performance against MMLU's roughly 14,000, and releases tiny versions of the Open LLM Leaderboard, MMLU, HELM and AlpacaEval 2.0 (arXiv:2402.14992). Frontier That is an item-response result in everything but name: it demonstrates extreme item redundancy and strong cross-model correlation in item difficulty, which is what an IRT difficulty parameter is. Frontier What is missing is the deliverable — published, reusable per-item difficulty and discrimination parameters that another team could administer as an instrument. Speculative Zhuang and colleagues propose the full programme, computerised adaptive testing tailored per model, arguing that “psychometrics, a theory originating in the 20th century for human assessment, could be a powerful solution” to cost, contamination and unreliable items (arXiv:2306.10512, whose own title changed across four versions to the current “Position: AI Evaluation Should Learn from How We Test Humans”).

Frontier Item robustness is being isolated as a quantity separate from capability, and the separation is large. Functional benchmarks re-instantiate the same problem with different surface content: MATH() produces reasoning gaps of 58.35% to 80.31% in models that score well on the static version (arXiv:2402.19450). Frontier A gap that large between a problem and the same problem re-drawn is a measurement of the instrument, not of the model. Frontier It is also the most direct available estimate of how much of a static benchmark score is item memorisation rather than method, and it is far larger than the contamination literature's own headline drops — which suggests memorisation and contamination are not the same quantity and are being conflated.

Established Cost has become a first-class axis and is still routinely omitted. The ARC-AGI-2 report supplies the extreme instance in the o3 preview figures: $200 per task versus $20,000 per task for the same system at two compute settings, twelve percentage points apart on ARC-AGI-1. Established A score reported without its compute setting is not a measurement, and Kapoor et al. make the general form of that argument for agents. Frontier The honest reporting unit for an agentic system is a triple — success rate, reliability across repeated runs, dollars per task — and almost no leaderboard publishes it.

Frontier Independent evaluation bodies exist; their published methodology could not be verified for this brief, and that is itself the finding. The UK AI Security Institute maintains the open-source Inspect evaluation framework, and the International AI Safety Report documents an intergovernmental structure of 30 nations plus the UN, OECD and EU, with 100 experts contributing under a chair, Bengio, holding “full discretion over the report's content” (arXiv:2501.17805). Established Those structural facts are sourced. Speculative Everything downstream of them — specific AISI methodology, the joint UK/US pre-deployment testing reports, any US CAISI evaluation — was behind access gates during the research for this brief and is therefore stated nowhere in it with numbers. Handwave A separate confound, evaluation awareness or deliberate sandbagging by a model that detects it is being tested, is frequently discussed and was not established by any source read here. It is a coherent worry and this brief asserts nothing about it.

Established The workback target is an instrument that survives contact with a new model. Concretely: a measurement which, applied unchanged to a system released after the measurement was designed, yields a number that predicts that system's performance on tasks nobody had thought of when the instrument was built. Established Every instrument surveyed here fails that test. They are saturated, gamed by sampling volume, contaminated, or so new that no subsequent model has been put through them. The chain below is ordered, and the binding link is not the expensive one.

Frontier L1 — item-level psychometric infrastructure. Per-item difficulty and discrimination parameters, fitted across many models and published as a reusable instrument. Partially present: tinyBenchmarks and Anchor Points establish the underlying fact, neither publishes the parameters. Frontier L2 — a generative item model. A procedure emitting new items at a specified difficulty from L1's parameters, so the test can be regenerated after every model release. Exists for mathematics, via the functional-benchmark method; not demonstrated for semantic transformation. Frontier L3 — contamination-proof administration. Items that provably did not exist before a model's training cutoff, administered once. FrontierMath's private-set protocol and ARC's semi-private / private split are the two working instances; both cost expert-specialist hours per item, and the funding and problem-access disclosures for at least one of them could not be obtained for this brief.

Speculative L4 is the binding link: construct validity. The step is a demonstration that a latent ability estimated on one instrument predicts out-of-distribution performance on tasks drawn from a different generator built by a different team. Speculative Nobody has done it. Until that experiment exists, every machine-IQ number is a description of a test rather than a measurement of a system, and the standing objection — that benchmarks claiming generality “operate as stand-ins for a range of anointed common problems” while suffering construct validity problems (arXiv:2111.15366) — stands unanswered. Established The important property of L4 is that it is cheap. It needs no new hardware, no new corpus and no new model: it needs a pre-registration, two teams who do not talk to each other, and the willingness to publish a calibration curve that might be flat.

Frontier L5 — invariance under scaffolding. A demonstration that the latent ability belongs to the model rather than the harness, obtained by running the same models under several independently-built agent scaffolds at fixed compute and decomposing the variance. Kapoor et al. document the confound precisely and do not decompose it. Frontier L6 — cross-population human anchoring. Human comparisons sampled from a specified population with a reported distribution and time budget, as HCAST does for software; nothing equivalent exists for wet-lab work, litigation or clinical practice. Speculative L7 — the survival test. Apply the instrument, unchanged, to three models released after its publication; it passes if rank order and predicted-versus-actual calibration hold. No instrument in this field has passed L7. Most have never been asked to.

Established This subject depends on results it does not produce, and two of them are not scientific results at all. The first is training-data disclosure. Contamination for a closed model is unfalsifiable in practice: for a system whose corpus is undisclosed, no contamination claim can be settled, so every benchmark number is an upper bound of unknown tightness. Established That is a governance dependency, not a methodological one, and it belongs with AI governance. The second is model access: pre-deployment evaluation by a third party happens at the discretion of the developer, which makes the independence of an evaluation a contractual property before it is a scientific one.

Established The scientific dependencies are mostly imports. The statistics of experiments — interval estimation, paired comparison, power analysis — already exist and are simply not applied; Miller's paper is an import, not an invention. Frontier Item response theory and computerised adaptive testing are mature in human assessment and are being reconstructed here without their vocabulary. Frontier Measurement invariance testing, the tool that would settle whether a machine general factor is one thing across model families, is standard in cross-cultural psychometrics and has not been applied to models in any work this brief could find.

Frontier Two dependencies run the other way, and this brief is upstream of them. Any claim in artificial general intelligence about how close a threshold is depends on an instrument that can locate it; any claim in artificial scientists or artificial creativity about a system exceeding human performance depends on a specified human population. Established Where the population is unspecified, the claim is not weakly supported, it is not yet a claim. Frontier The same holds for multi-agent systems, where the compute-matched comparison that would make an orchestration gain interpretable is the missing measurement rather than the missing engineering.

Established M1 — publish per-item parameters. Fit a two-parameter item response model to MMLU-Redux and GPQA across at least a hundred models spanning three orders of magnitude of training compute, and release the difficulty and discrimination parameters as a public instrument. Cost: low, on the order of one graduate-year of compute and labour. Established Nobody has done it, which is why the field regenerates a fresh benchmark instead of calibrating the one it has.

Established M2 — report every score with an interval and a cost. Re-run the top twenty rows of any agentic leaderboard under Miller's estimators with dollars per task recorded, and publish the intervals. Frontier The prediction worth pre-registering: a substantial fraction of adjacent leaderboard ranks will prove not statistically separable. If that prediction holds, most leaderboard movement of the last three years was reporting noise as progress.

Frontier M3 — publish a 95% time horizon. Re-analyse the METR and HCAST data at a 95% success threshold instead of 50%, and report the doubling constant. Frontier This is the highest value-per-dollar experiment in the category: the data already exist, the analysis is an afternoon, and the entire economic argument built on the time-horizon curve implicitly assumes the 50% and 95% curves are parallel. Nothing establishes that. A system that completes month-long tasks half the time is a different economic object from one that completes them nineteen times in twenty, and no published curve distinguishes them.

Speculative M4 — the pre-registered out-of-distribution prediction. Fit latent ability for N models on instrument A. Pre-register point predictions of their scores on instrument B, built by a disjoint team from a disjoint generator. Publish the predictions before instrument B exists. Report calibration. Speculative This is L4, the binding link, and it is the single experiment that would convert benchmark scores from descriptions into measurements. Frontier M5 — the scaffold variance decomposition. Same models, five independently built agent scaffolds, fixed compute budget; report the variance component attributable to scaffold. If scaffold variance dominates model variance, the object being measured is not the model, and a large part of the agentic literature is measuring engineering teams. Speculative M6 — measurement invariance across model families. Test whether the reported general factor has the same structure in models trained on disjoint corpora. A negative result would retire machine g; a positive one would be the first evidence that the scale is a scale.

Established Most of what is needed here is unglamorous instrumentation, and its absence is why results do not reproduce. The harness maintainers name setup sensitivity, cross-method comparability and reproducibility as the three obstacles; all three are engineering problems with known solutions. Established A conforming evaluation run should emit, as a machine-readable record alongside the score: the exact prompt template, the shot count, the answer-extraction rule, the decoding parameters, the number of attempts, the wall-clock and token cost, and the immutable identifier of the benchmark version used.

Frontier Agentic evaluation needs a sandbox with a meter on it. The tasks are long, stateful and side-effecting, so a comparable run requires a containerised environment with fixed tool versions, network policy and time limits, plus per-run telemetry recording dollars spent. Established Kapoor et al.'s finding that agents become needlessly complex and costly is a direct consequence of the meter being absent from the leaderboard. Frontier Where a benchmark has no holdout set, the engineering fix is trivial and the incentive to apply it is negative, which is why it has not happened.

Frontier Private-set custody is a security engineering problem the field has not treated as one. A private evaluation set retains its value only while it has never been transmitted to a system that logs inputs. That implies air-gapped or contractually bound administration, single-use item retirement, and a refresh pipeline whose cost is bounded. Speculative Currently the refresh cost is expert-specialist hours per item, which does not scale; an item generator calibrated on L1 parameters is the only route to a private set that can be regenerated as fast as models are released.

Frontier Benchmarks need version control with the discipline of software releases. Eleven versions of one exam in eighteen months, seven of another in thirteen months, and a title change on a widely cited methods paper between versions are all normal in this field and all invisible in a citation that names only an identifier. Established Immutable, individually citable benchmark releases would cost almost nothing and would make a large class of irreproducible comparisons impossible to publish.

Established The mature adjacent discipline is human psychometrics, and the borrowing has been partial. Item response theory, computerised adaptive testing, construct validation and measurement invariance are a century of accumulated method, and the AI evaluation literature is currently rebuilding the first two under different names while ignoring the third and fourth. Speculative The explicit attempt to import the whole apparatus is the adaptive-testing proposal (arXiv:2306.10512), which argues for estimating the characteristics of each test item and tailoring the assessment per model instead of administering a fixed benchmark to everything.

Established Clinical trial methodology supplies the governance half. Pre-registration, pre-specified endpoints, blinded assessment and independent data monitoring exist because medicine discovered that outcome-switching and selective reporting are the default behaviour of interested parties. Frontier Every structural problem documented in the leaderboard literature — private variant testing, score retraction, asymmetric data access — has a named counterpart in the trials literature and a known remedy.

Speculative Metrology is the deeper analogy and the more demanding one. A measurement in the metrological sense is traceable: it chains back through calibrated references to a defined unit, and it carries an uncertainty budget. Speculative No AI capability measurement is traceable in that sense, because there is no defined unit and no reference standard. The time-horizon metric is the closest existing candidate, because a minute of expert human work is at least a unit with an external referent.

Frontier Two non-behavioural instruments sit alongside. Mechanistic interpretability offers the possibility of measuring a capability by inspecting the mechanism rather than sampling the output, which would be immune to contamination by construction; it is nowhere near able to do so for a frontier model. Speculative Adversarial red-teaming offers the opposite approach, a standing bounty for finding a task where the system fails and a competent human succeeds, with the failure rate of the search as the measured quantity — a measurement series nobody currently funds. Both connect to machine consciousness and cognitive architectures, where the same behaviour-versus-mechanism question recurs.

Established The bodies scoring the models are frequently the bodies that built them, and this is documented rather than alleged. The Leaderboard Illusion analysis of Chatbot Arena reports that “undisclosed private testing practices benefit a handful of providers who are able to test multiple variants before public release and retract scores if desired”; that 27 private variants were tested by one provider in the lead-up to a single model release; that two providers received an estimated 19.2% and 20.4% of all arena data while 83 open-weight models together received an estimated 29.7%; and that additional data access produced performance gains of up to 112% on the arena distribution (arXiv:2504.20879v2). Frontier A leaderboard with private pre-testing, asymmetric sampling and an option to retract a bad score is an incentive system with a measurement-shaped interface.

Established The comparison with human assessment is the argument, and it is unflattering. Human educational and psychological testing is a licensed profession with published test manuals, documented norming samples, standardised reliability and validity reporting, and professional standards that a test publisher can be held to. Established The papers proposing psychometrics for AI are explicitly appealing to that tradition: Zhuang and colleagues describe psychometrics as “a theory originating in the 20th century for human assessment” that could solve the field's problems with cost, contamination and unreliable items. Established Machine evaluation has none of the accompanying apparatus. There is no test manual, no published item parameter set, no norming sample, no requirement to disclose who funded an evaluation or who had prior access to its items, and no accreditation of evaluators. Frontier The consequence is that a benchmark result and a marketing claim are, institutionally, the same kind of object.

Frontier Independent bodies exist and are constrained in a specific way. National institutes and the intergovernmental report structure are real: 30 nations plus the UN, OECD and EU, 100 contributing experts, and an explicit statement that those experts held full discretion over content. Speculative But third-party pre-deployment evaluation depends on model access granted by the developer, on a schedule set by the developer, often under non-disclosure. Speculative Independence of judgement without independence of access is a weaker guarantee than it reads as, and the published methodology that would let an outsider check the guarantee was not obtainable for this brief. Frontier The funding and prior-access disclosures for at least one prominent private-set mathematics benchmark are likewise unresolved, which matters because prior access to a private set is exactly the failure the private set exists to prevent.

Established These numbers allocate capital and shape law, which raises the cost of getting them wrong above the cost of an ordinary scientific error. A capability score is quoted in funding rounds, procurement decisions, safety cases and legislative testimony, and by the time it reaches those places the protocol has been stripped. Established The bar-exam case is the model: OpenAI's own technical report published a top-10% claim, a peer-reviewed re-analysis recomputed it downward by reference population, and the correction did not propagate. Established When the English Wikipedia article on stochastic parrots was checked for the research behind this brief, it still listed a greater-than-90th-percentile bar exam result among the published counterarguments to Bender and colleagues — the refuted figure still doing argumentative work in a live debate, two years after the correction appeared in a peer-reviewed law-and-AI journal.

Established Overclaiming and underclaiming are both harms, and they are not symmetric. An inflated score routed into a deployment decision puts an unreliable system in front of people who were told it was reliable; tau-bench's collapse from under-50% single-attempt to under-25% across eight consecutive attempts is what that failure looks like quantitatively. Frontier A deflated or dismissed measurement, by contrast, delays scrutiny of a system that is in fact capable, which is the failure mode that matters for existential risk governance. Frontier The skeptical literature has its own protocol failures, and a brief that only audits enthusiasts is not auditing.

Frontier There is a distributional question underneath the technical one. Contamination is unfalsifiable for closed models and falsifiable for open ones, so the systems whose scores can be audited are systematically not the systems whose scores matter most. Established The same asymmetry appears in leaderboard data access, where the open-weight majority receives a small minority of the evaluation traffic. Speculative A measurement regime whose rigour is inversely proportional to a system's commercial importance is not a neutral instrument, whatever the intentions of the people running it.

Speculative A civilisation that cannot measure its most consequential technology is making decisions by anecdote at scale. Every projection of labour displacement, every safety threshold written into regulation, every capital allocation premised on capability arriving by a date rests on instruments that have not passed a survival test and mostly have not been asked to. Speculative The gap is not between what is known and what is claimed; it is between what is measured and what is measurable with the methods now in use.

Speculative The historical analogue is metrology, and it is encouraging. Standardised measurement of length, mass, time and electrical quantity was built as public infrastructure, funded by states, maintained by dedicated institutes, and made traceable through chains of calibrated references. It took roughly a century and it made industrial civilisation possible. Speculative There is no principled reason a comparable regime for machine capability could not be built. There is a practical one: the object being measured changes faster than the instrument can be validated, which is a problem metrology never faced because the metre did not improve itself.

Handwave The exotic version is a permanent, adaptive, adversarially-refreshed public instrument. An item generator calibrated to a published parameter set, administering novel items to every released system under a fixed compute and cost budget, reporting an ability estimate with an interval and a calibration record against its own prior predictions — a benchmark that is a running institution rather than a file. Handwave Nothing technical forbids it. What forbids it is that no one currently pays for measurement whose main output is the news that a headline number was smaller than reported.

These horizons track the instrument, not the systems. What is being forecast is when a measurement of machine capability could be trusted the way a well-built human test is trusted.

• 10 yr: Frontier Published per-item parameter sets and interval-reporting become normal for at least the major static benchmarks; cost per task appears alongside accuracy on agentic leaderboards. A 95% time-horizon curve is published and either confirms or breaks the parallel-curves assumption. Contamination remains unfalsifiable for closed models absent a disclosure regime.
• 25 yr: Speculative A construct-validity result exists: at least one pre-registered out-of-distribution prediction from a latent ability estimate has been published, with its calibration, whether or not it succeeded. Whichever way it comes out, the field learns whether it has been measuring systems or describing tests.
• 50 yr: Speculative An instrument passes a survival test — applied unchanged to models released after its design, preserving rank order and calibration. If machine general intelligence admits a scale at all, this is when the evidence would first exist. If jaggedness is fundamental, this is instead when the field settles on a vector-valued profile and stops asking for a scalar.
• 100 / 250+ yr: Handwave Capability measurement is public infrastructure with the standing of metrology: defined units, traceable references, uncertainty budgets, accredited evaluators, and item generators that outpace model releases. Or the question dissolves, because the systems being measured participate in designing the measurement and the distinction between instrument and subject stops doing useful work.

• Depends on Nothing on this map produces a result this brief waits on. Its dependencies are a governance decision (training-data disclosure), a contractual one (third-party model access), and a body of imported method — item response theory, measurement invariance testing, and the statistics of experiment design — all of which already exist in human assessment and clinical research and simply have not been applied here.
• Requires (not on this map) Two institutional facts that do not exist. The first is an evaluator whose access to a model, whose funding, and whose publication schedule are not controlled by the model's developer — independence of judgement without independence of access is a weaker guarantee than it reads as. The second is the apparatus human assessment has and machine evaluation lacks entirely: published test manuals, reusable item parameters, norming samples, mandatory disclosure of funding and prior item access, and accredited evaluators. Without the second, a benchmark result and a marketing claim are institutionally the same kind of object.
• Enables This brief is the instrument for the category. The threshold claims in artificial general intelligence cannot be adjudicated without a measurement that survives contact with a new model, and every capability figure quoted anywhere on this site inherits the protocol questions set out here.
• Adjacent Shares its methods problem with artificial scientists and artificial creativity, where a claim of superhuman performance depends entirely on which humans; with multi-agent intelligence systems, where the compute-matched control is the missing measurement; with AI governance, which is where the disclosure requirement would have to come from; and with machine consciousness, where the behaviour-versus-mechanism question is the same question in a harder form.

Established “o3 solved ARC-AGI.” The December 2024 figures of 76% and 88% belong to a preview configuration on the ARC-AGI-1 semi-private set, at estimated costs of $200 and $20,000 per task. The shipped o3 (Medium) scores 53.0% on that same semi-private set and 3.0% on ARC-AGI-2. The prize target of 85% is on the private set under a compute cap, which the 2024 competition moved from 33% to 55.5%. All of these numbers appear in sources this brief cites; the gap between the first pair and the second is the reason the Institute treats a score without its protocol as not yet a fact. Frontier A secondary error rides along: the frequently quoted pair “75.7% / 87.5%” does not appear in the ARC-AGI-2 paper, which reports 76% and 88%. Anyone quoting the more precise figures owes a separate source.

Speculative “The ARC-AGI public training set was in o3's pre-training” — and its denial. Both are asserted with confidence and neither was established by any source read for this brief. It is open. It is also the most important open factual question in the category, because it decides whether the o3 result measures skill acquisition on novel structure or measures preparation — and that distinction is the entire content of the argument ARC was built to test.

Established “GPT-4 passed the bar exam in the top 10%.” The 90th-percentile figure is OpenAI's own. The published re-analysis computes roughly the 62nd percentile against first-time takers overall, roughly the 48th against licensed or license-pending attorneys, and on the essay components roughly the 15th against those who passed. The correct lesson is not that one number was wrong. It is that a percentile with no reference population is not a claim about ability at all.

Established “Model X scores Y% on MMLU.” Incomplete without the shot count, the prompt template, the answer-extraction method, and whether the set was MMLU or MMLU-Redux. Up to five points of any MMLU score are prompt artifact, and about 6.5% of the items are miskeyed. Established “Benchmark X is saturated, therefore models are superhuman at X.” MMLU's ceiling is partly the ceiling of the ruler: 6.49% of items contain errors and 57% of the analysed Virology subset does. A test cannot distinguish above its own error floor.

Established “Models beat human experts on GPQA.” The experts scored 65%, or 74% discounting mistakes they identified retrospectively; GPT-4 scored 39% at publication. A later model exceeding 65% is exceeding domain PhDs answering questions outside their own subfield under time pressure — a specific and unusual population, and not the one the phrase “human experts” conjures.

Established “Agents now solve most of SWE-bench.” Roughly a third of successful SWE-bench patches had the solution present in the issue text or comments, and roughly another third passed under inadequate tests; removing both dropped one measured agent from 12.47% to 3.97%. Over 94% of the issues predate the models' knowledge cutoffs, and models identify the buggy file from the issue text alone at 76% inside the benchmark against 53% outside it.

Established “The model achieved N%, therefore it can do the task.” Under tau-bench's pass^k metric, gpt-4o sits under 50% at a single attempt and under 25% across eight consecutive attempts in retail. A system that succeeds about half the time, and completes eight consecutive runs less than a quarter of the time, is not a system that can do the task; it is a system that sometimes does it. Frontier Conversely, a score reported at pass@250 is a statement about a sampling budget: repeated sampling took one model from 15.9% to 56% on SWE-bench Lite without changing the model at all.

Established “AI will automate month-long human tasks by 2030.” The underlying sentence is explicitly conditional — “if these results generalize to real-world software tasks” — at a 50% success threshold, extrapolating a seven-month doubling the authors themselves flag as possibly having accelerated, which makes the trend line less stable for extrapolation rather than more. Established And the paper narrowed its own title between versions: v2 reads “Measuring AI Ability to Complete Long Tasks”, v4 reads “Measuring AI Ability to Complete Long Software Tasks”, with an identical abstract. Cite the version. The scope claim lives in the title.

Established “Arena rankings are a neutral crowd measurement.” Private pre-testing of many unreleased variants, the option to retract an unflattering score, and an estimated 19.2% and 20.4% of all arena data going to two providers against 29.7% shared among 83 open-weight models. Frontier None of that makes the ranking worthless; it makes it a measurement of a process that includes strategic behaviour, and it should be quoted as one.

Frontier “Emergent abilities prove that scale produces qualitative jumps.” The claim and its metric-choice rebuttal are adjudicated in artificial general intelligence, and the measurement lesson is the one that belongs here: an apparent discontinuity in a score is a joint property of the capability and the scoring function, so a jump reported under exact-match or multiple-choice accuracy is not the same observation as a jump reported under a continuous metric. Frontier The deflationary reading is routinely overstated in the other direction too.

Established Skeptic-side error: “benchmarks are meaningless.” GSM1k is the counterexample. A purpose-built, matched, uncontaminated benchmark found frontier models generalising, with drops of at most 8%. The defensible position is that benchmark scores are noisy and biased upward by an unknown amount, not that they are noise. Established Related error: “small evaluations are unreliable, so bigger is better.” A hundred curated items reproduce MMLU's fourteen thousand. The problem with a small evaluation is item selection, not item count — and most of a large benchmark is redundant, which is a fact about the benchmark rather than the models.

Speculative “Independent institutes verify frontier models before release.” National evaluation institutes exist, an intergovernmental report structure exists with 30 nations and 100 contributing experts, and an open-source evaluation framework is published. What could not be verified for this brief is any specific methodology, any pre-deployment testing report, or any evaluation number from those bodies — every relevant domain was behind an access gate. This brief therefore states the structures and prints none of the results, and a reader should treat confident public summaries of what those institutes found with the same suspicion this brief applies to a vendor's own benchmark table.

Handwave “A machine IQ score.” The phrase presupposes what the field has not shown: a latent variable with a stable factor structure across model families, corpora and scaffolds, whose estimate predicts behaviour on tasks drawn from a generator the estimator never saw. Handwave There is a coherent version of the exotic claim — that a general factor exists, that jaggedness is an artifact of incomplete scaffolding, and that a single number will eventually summarise a system as usefully as g summarises a person. Speculative What would have to be true is specific and testable: measurement invariance across model families, a positive manifold surviving disjoint corpora and post-cutoff tasks, and one successful pre-registered out-of-distribution prediction. Established None of the three exists yet. Until they do, the honest object is a profile with error bars and a cost column, and the honest sentence about any single capability number is the one this brief has been arguing for throughout: it is a description of a test, administered under a protocol, on a date.