Deep Learning Theory

Speculative If a theory existed that could name a network’s scaling exponent before the network was trained, what it would save is the ladder — the sequence of smaller models a laboratory builds for the sole purpose of extrapolating to the model it actually wants to buy. That is the condition, and it has never been met. Every exponent in use today was fitted to runs that had already happened; no published theory has predicted one in advance, in public, and then been checked against the run. Section 7 says exactly what such a check would look like, and it is cheap by the standards of the industry it would serve.

Established The magnitude is best read from what happens when the fit is wrong, because that has now been measured. The 2022 Chinchilla paper supplied the exponents that allocate frontier training budgets between parameters and tokens. A 2024 reconstruction of its published figures refitted the same three-term form and moved the data exponent from 0.2849 to 0.3658 — a 28% change — the parameter exponent from 0.3392 to 0.3478, and the irreducible-loss term from 1.6934 to 1.8172. Established The original reported a confidence interval on its compute-allocation exponent running from 0.454 to 0.455; the replication’s bootstrap gives a standard error of 0.018, and estimates that an interval that tight would need on the order of 600,000 training runs behind it rather than the roughly 400 the paper ran.

Frontier The only piece of deep learning theory with a costed industrial saving attached is the maximal-update parameterisation, and the size of that saving is the honest ceiling on this subject’s near-term value. Tuning a 6.7-billion-parameter model by transferring hyperparameters from a 40-million-parameter proxy cost 7% of total pretraining compute, against a baseline of tuning at full scale or not tuning at all. Frontier That is a real number and a modest one: it is a saving on hyperparameter search, not on training, and a 2024 study across tens of thousands of models up to 26.8 billion parameters found that other parameterisations achieve the same transfer, so even this result is a piece of engineering practice rather than a theorem anyone has cashed.

Handwave The upside here is narrower than the field’s rhetoric and it accrues to whoever already owns the compute. A theory of generalisation cannot be tested without training ladders, ladders cost what frontier laboratories spend and universities do not have, and so the institutions best placed to check the theory are the ones whose budgets the theory would reallocate. Handwave The condition on everything above: the regularities have to be properties of learning rather than properties of a particular decade’s corpora, optimisers and hardware — and section 5 records that nobody has separated those.

Deep learning theory is the attempt to explain why a function with more free parameters than training examples, fitted to convergence by a first-order method with no explicit capacity control, predicts well on data it has never seen. The practice works; the explanation does not exist; and the gap between those two sentences is the subject.

Established Three bodies of evidence define the problem and they point in different directions. The first is classical statistical learning theory, which bounds test error by training error plus a capacity term, and whose capacity terms for modern networks are numerically larger than one — vacuous, since test error is a probability. The second is a set of empirical regularities that are startlingly clean: cross-entropy loss falling as a power law in model size, dataset size and compute, with trends the original 2020 study reports as spanning more than seven orders of magnitude, and network width and depth having minimal effect within a wide range. Frontier The third is a family of solvable limits — the neural tangent kernel, mean-field and maximal-update parameterisations, random-feature models — which can be analysed exactly and which describe a regime that trained networks demonstrably are not in.

Speculative The distinction that organises the whole field is between a regime in which the network’s internal representation is fixed at initialisation and only the readout moves, and a regime in which the representation itself changes. The first is called the lazy or kernel regime and it is where almost all of the exact theory lives. The second is called feature learning and it is where every result anyone cares about lives. Handwave No theory bridges them at the widths and depths people actually train, and the honest description of the field is that its solvable cases and its interesting cases are disjoint.

Established This brief does not own benchmark validity. Intelligence Measurement owns the question of whether a reported score measures anything, and its answers — protocol disclosure, contamination, confidence intervals, the 6.49% error rate found in a widely used benchmark — are prerequisites here rather than content. Frontier The question this page owns is one level down: given that the loss numbers are real, why is the loss low, and why does it fall on a straight line in log-log coordinates that nobody can derive.

Speculative The framing this page tests is that deep learning is a pre-paradigmatic science with excellent instruments — and it survives, with the qualification that the instruments are owned by the parties whose commercial decisions the readings justify. Handwave That is not an accusation of fraud. It is a structural observation about a field whose single most cited quantitative artefact went two years before anyone outside the originating laboratory refitted it, and whose refit found an optimiser bug.

Established Start with what is actually measured, because the measurements are good and the explanations are not. The 2020 scaling study of transformer language models reports that loss falls as a power law in model size, dataset size and training compute, with some trends spanning more than seven orders of magnitude, and that architectural details such as width and depth have minimal effect within a wide range. Established Those two findings together are the empirical foundation of the entire modern training regime: they say that the quantity which predicts performance is a budget, not a design.

Established The commission for this brief names the Chinchilla exponents as the anchor, and the anchor is real but it is not what it is usually taken to be. The 2022 paper fitted a three-term form — an irreducible loss plus a term falling as a power of parameter count plus a term falling as a power of token count — by three separate approaches, and reported the parametric fit (its Approach 3) with parameters A = 406.4, B = 410.7, E = 1.6934, α = 0.3392 and β = 0.2849. Established A 2024 replication reconstructed the underlying data from the published figures and could not recover that fit. Its own estimates are A = 482.01, B = 2085.43, E = 1.8172, α = 0.3478 and β = 0.3658. The data exponent moves by 28% and the coefficient B by a factor of five.

Established The replication names three specific defects and two of them are arithmetic rather than statistical. First, rounding: using the reported 0.28 in place of the fitted value introduces a positive bias of roughly 13% for a typical dataset size. Second, the optimiser used for the fit stopped before convergence because of a poor choice of loss scale, compounded by averaging Huber-loss values across data points instead of summing them. Third, and most consequential, the reported confidence interval on the compute-allocation exponent ran from 0.454 to 0.455, a width the replication estimates would need on the order of 600,000 observations to support, against roughly 400 training runs actually performed; its own bootstrap standard error is 0.018, around fifty times wider. Frontier The internal inconsistency is the part that matters for practice: the parametric fit’s own implied allocation is about 70 tokens per parameter, while Chinchilla itself was trained at about 20, which is what the paper’s other two approaches recommend. Established The famous number survived because it came from the approaches that were not broken.

Established Now the theory side, and it opens with a negative result that has held for seven years. A 2019 analysis showed that on a five-layer, 1024-wide ReLU network trained on MNIST, distance from initialisation grows as roughly the 0.4 power of training-set size and the product of layer spectral norms grows linearly in it, so that the norm-based bounds then in circulation increase with more data — growing as about the 0.68 power of sample size — while measured test error falls as about the 0.43 power. Established The same paper constructs a linear setting with a small signal subspace and a large noise subspace in which the learned classifier has generalisation error at most ε while any uniform-convergence bound on the algorithm is at least 1 − ε, and a 1000-dimensional two-sphere setting with a two-layer network of width 100,000 where test error falls as the inverse square root of sample size and the bound remains vacuous.

Frontier That result is contested in a way the field rarely reports. A 2019 reply proposes analysing the learned predictor through a coupled surrogate predictor belonging to a class with a structural Glivenko–Cantelli property, and proves for a hypercube construction modelled on the original counterexample that the expected generalisation gap vanishes — a uniform-convergence argument that does explain generalisation in the setting built to defeat uniform-convergence arguments. Speculative The disagreement is about what “explain” means: whether a bound that holds for a derandomised surrogate is an explanation of the original algorithm, or a different theorem about a different object. Handwave Nobody has adjudicated it, and both papers are cited approvingly by people who have not noticed they conflict.

Frontier Bounds are no longer uniformly vacuous, which is the largest change in this corner of the field since 2018. A 2024 result reports the first non-vacuous generalisation bounds for pretrained large language models, obtained by a compression argument valid for the unbounded log-likelihood loss using prediction smoothing, extended to handle subsampling, with a low-dimensional non-linear reparameterisation as the compression device. Frontier Its headline structural claim is the interesting one: larger models have better generalisation bounds and are more compressible than smaller ones, which inverts the classical relationship between capacity and guarantee. Speculative The bounds are still bounds on a compressed surrogate rather than on the shipped model, and this page does not treat them as an explanation of why the shipped model works.

Established Double descent is the second measured regularity and it is more robust than its explanations. The 2019 study that named it reports three forms — model-wise, epoch-wise and sample-wise — with the test-error peak located at the interpolation threshold, the point where the model is just barely large enough to fit the training set. Established The same work reports a regime in which adding data makes performance worse: transformers trained on a language task with 4.5 times more samples did worse at intermediate model sizes, because more data moves the interpolation threshold to the right. Established The effect is most prominent with added label noise, and the authors leave the mechanism in real networks explicitly open.

Frontier Exact theory of double descent exists and it is theory of a different model. A 2021 solution of gradient flow in the random-feature model gives closed-form training and generalisation error as integrals over spectral densities, predicts that the peak develops at the interpolation threshold after a definite time and is preceded by a dip, and predicts triple descent and two distinct epoch-wise structures. Speculative The authors state that the analysis is not entirely mathematically controlled and applies to fixed random features rather than to networks that learn representations. Handwave Between that model and a trained transformer there is no derivation, only an analogy, and the analogy is doing all of the explanatory work in most secondary accounts.

Established Double descent also reproduces outside machine-learning benchmarks, which is a real piece of evidence about its generality. A 2025 workshop study on particle-physics data found both epoch-wise and model-wise double descent across models from 774 to 139 million parameters on a jet transverse-momentum regression, with a billion-parameter model beating classical approaches on a fifteen-way classification task at 150,000 training events. Established It also reports that the overparameterised regime did not beat classical models on the regression task, and that the interpolation threshold was found empirically — by observing where the training loss vanished — rather than predicted. Speculative That last clause is the state of the art: seven years after the phenomenon was named, its location is still measured after the fact.

Frontier The neural tangent kernel is the field’s one genuinely exact result about wide networks and its limitations are now quantified rather than asserted. A 2023 analysis reports that convergence to the kernel regime requires width growing polynomially in depth, orders of magnitude beyond common architectures; that the Jacobian-stability constant needed for the predicted fast convergence of second-order methods remains many orders of magnitude above its required value even at widths up to 10⁷; that a neural bandit algorithm using kernel-theoretic exploration parameters performs no better than random search at depth three; and that the predicted reduction of catastrophic forgetting with width disappears once the networks are trained to convergence. Established These are three separate practical predictions of the theory, tested, and failed.

Frontier The kernel picture is also being attacked from inside. A 2025 analysis argues that the standard characterisation of the infinite-width limit as a frozen-feature regime is wrong in a specific way: individual neuron deviations from initialisation do vanish, but the aggregate norm of those deviations does not, with layer-wise deviation measured at 4.89 and 10.56 at widths of ten and thirty thousand while per-neuron deviation stayed at 0.095 and 0.138. Speculative The same work proves that feature correlations between data pairs vary during training with probability one in the infinite-width limit. Handwave If that holds up, a decade of statements about “lazy training” are describing an artefact of how the limit was taken rather than a property of wide networks.

Frontier And the widths at which feature learning is observed to stop are far below the widths in use. A 2025 study tracking the kernel’s trace and effective rank through training reports maximum feature-learning widths of about 500 on MNIST and Fashion-MNIST and about 1,000 on CIFAR-10, with the relative growth of the kernel trace declining as roughly the −0.7 to −0.8 power of width, against feed-forward widths in deployed language models of 14,336 to 73,728. Speculative Read one way that says deployed models are firmly in the kernel regime on those tasks; read another it says the diagnostic does not transfer across tasks; the study does not settle which, and neither does this page.

Established The theory that has been adopted industrially is the maximal-update parameterisation, and its record is mixed in an instructive way. The 2022 result showed that optimal hyperparameters can be made stable across width, so that a 40-million-parameter proxy can be tuned and the settings transferred to a 6.7-billion-parameter model at 7% of pretraining cost. Established A 2024 study training tens of thousands of models across fourteen sizes up to 26.8 billion parameters, three optimisers, four parameterisations and more than a dozen learning rates found that all the parameterisations tested can achieve hyperparameter transfer, that a per-layer learning rate for standard parameterisation beats the maximal-update one, and that Adam’s epsilon must be scaled correctly to avoid gradient underflow — a numerical detail that had been silently confounding scaling comparisons.

Frontier The failures of transfer are now catalogued and they are not edge cases. A 2025 analysis decomposes transfer into a loss gap, a hyperparameter gap and a transfer suboptimality, shows fast transfer requires the hyperparameter gap to fall faster than the square root of the loss gap, and exhibits a two-layer ReLU network learning a norm-indicator function where it does not. Frontier The same work reports that the Muon optimiser shows a more pronounced learning-rate shift than Adam, with transferred hyperparameters producing diverging scaling curves, and that transfer breaks on evaluation sets concentrated on hard examples. Frontier A 2025 extension to mixture-of-experts models finds learning-rate transfer across expert count but not across expert granularity, and reports two training runs at embedding dimension 128 diverging under the maximal-update parameterisation where a simpler scheme did not. Speculative A theory whose predictions hold along the axis it was derived for and fail along a neighbouring axis is a partial theory, and calling it a solved problem is the commonest overstatement in this area.

Established Finally, the link from loss to capability — the thing anyone actually wants predicted — is weak, and this is now measured rather than suspected. A 2025 re-analysis of 46 downstream tasks found that performance improves smoothly and predictably with pretraining loss in only 39% of cases, with the remainder distributed across inverse, non-monotonic, noisy, trendless and breakthrough behaviours, and that seemingly benign changes to the experimental setting can reverse the qualitative shape of a curve. Established A 2026 system that treats the problem as time-series extrapolation rather than curve-fitting reports 1.99% mean absolute error across 66 downstream tasks against 3.56% for logistic scaling laws, with its 10–90% predicted interval containing 77.6% of outcomes against a nominal 80%, and 76.6% accuracy at ranking which of two configurations will do better. Speculative A learned predictor beating the parametric form by 44% on error is evidence that the parametric form is a convenience rather than a law.

Frontier Four positions are live on why the scaling exponents take the values they do, and none of them has predicted one in advance.

Frontier Position one: the exponent is set by the intrinsic dimension of the data. The 2024 account that carries this view separates four regimes — variance-limited scaling in dataset size and in width, where loss falls as the inverse first power, and resolution-limited scaling in dataset size and in parameters, where it does not — and proposes that the resolution-limited exponent is inversely proportional to the intrinsic dimension of the data manifold, specifically about 4 divided by that dimension. Established In teacher–student experiments where the effective dimension can be dialled, the measured interpolation exponents closely match the prediction. Frontier On CIFAR-10 and CIFAR-100 with a four-layer convolutional network and a wide residual network, the same paper describes the relationship as “less clear”. Speculative That is the whole field in one sentence: the prediction works where the dimension is a parameter of the experiment and degrades where it has to be estimated from real data.

Speculative Position two: the exponent is a property of the data covariance and nothing deeper. On this account, random-feature models trained on data with power-law eigenvalue decay reproduce the correct scaling analytically, so power laws come from adding capacity to data that always has more second-order structure to fit, whether or not the added capacity is doing anything a person would call learning. Frontier The argument against it is that measured phenomena sit outside covariance: sharp capability transitions over narrow ranges of model size while loss curves stay smooth, and data-pruning results that beat power-law scaling by selecting examples on criteria unrelated to the spectrum. Handwave This page reports that debate as reported rather than adjudicating it, because the strongest statements of it are in blog posts and workshop papers rather than in anything that has been replicated.

Frontier Position three: the exponent is set by feature learning, and the kernel limit is the wrong baseline. A 2025 analysis derives, in a two-layer linear network with power-law spectra, that feature learning changes the loss-versus-time exponent from β in the lazy limit to 2β/(1 + β) for tasks outside the reproducing-kernel space of the initial kernel — nearly doubling it for hard tasks — and changes nothing for easy ones. Speculative It gives a matching compute-optimal exponent. Handwave The model is two layers, linear and exactly solvable; the claim is about transformers; the bridge is an assumption that the spectral picture carries over, and that assumption is the step where this position works by assertion.

Speculative Position four: there is no exponent to derive, and the power law is a fitting convenience over a range. The evidence is the 2026 result that a learned extrapolator beats parametric scaling laws by 44% on downstream error and generalises to unseen model families where the parametric form fails, and the 2025 finding that only 39% of downstream tasks scale predictably at all. Frontier The counter-evidence is that pretraining loss itself really does track a power law over many orders of magnitude and does so across laboratories, architectures and corpora. Speculative Both can be true: the loss is lawful and the map from loss to anything anyone wants is not.

Frontier Beneath those four sits a question that would reorganise the subject if answered: what is the correct notion of capacity for a network trained by a first-order method. Parameter count is plainly wrong, since the same count generalises or memorises depending on the labels. Norm-based capacity is wrong in a measured way, since the norms grow with data while error falls. Speculative Compressibility is the current best candidate, on the strength of non-vacuous bounds that get tighter for larger models. Handwave Nobody has shown that a compressibility measure computed before training predicts anything, which is the test that would move it from candidate to quantity.

Speculative And one question this page thinks is under-asked: whether the exponents are properties of learning or properties of a decade. Every exponent in circulation was measured on transformer architectures, trained with Adam or its relatives, on web-scraped English-dominant corpora, at bfloat16 or lower precision, on a narrow family of accelerators. Handwave There is no published exponent measured on a corpus, optimiser and architecture family chosen to be adversarial to the previous fit, which means the claim of universality has never been given a chance to fail.

Frontier The binding constraint on this subject is not mathematics. It is that the experiment which would discriminate between theories costs a training ladder, and the people with ladders are not running discriminating experiments.

Established Bottleneck one: the raw data behind published scaling laws is not released. The 2024 Chinchilla replication had to reconstruct loss values by reading points off the original paper’s figures, because the underlying runs were not published. Established That reconstruction is what surfaced the rounding bias, the unconverged optimiser and the impossible confidence interval — and it took two years. Speculative The cost of releasing the loss table is approximately zero and the cost of not releasing it was a field-wide allocation rule with a factor-of-three internal inconsistency.

Frontier Bottleneck two: numerical confounds are large enough to masquerade as science. The 2024 parameterisation study found that Adam’s epsilon has to be scaled with model size to avoid gradient underflow, and proposed an optimiser variant that removes the hyperparameter entirely. Speculative A measured “scaling exponent” contaminated by underflow at the large end is an artefact with a straight line through it. Handwave No published scaling law states whether it controlled for this, because before 2024 nobody knew to.

Frontier Bottleneck three: the dependent variable is disputed. Pretraining loss is cheap, continuous and lawful; downstream capability is what the budget is for and scales predictably in 39% of measured cases. Frontier Everything Intelligence Measurement says about protocol, contamination and intervals applies before a scaling curve of a benchmark score means anything, and most published downstream scaling curves answer none of its five questions. Speculative A theory that predicted the loss exponent exactly would still not tell you what the model can do.

Speculative Bottleneck four: seed variance is rarely reported and is the right size to matter. The mixture-of-experts parameterisation study reports two runs at the same width diverging under one scheme and not another; the hyperparameter-transfer analysis reports transfer failing when evaluation concentrates on hard examples. Handwave Both are variance findings dressed as method findings, and neither field practice nor this page can currently tell the difference.

Frontier Bottleneck five, and the one that will not yield to money: the interesting limit is not the tractable one. Exact analysis requires width growing polynomially in depth — orders of magnitude beyond deployed architectures — and the diagnostics that track feature learning place its maximum around widths of 500 to 1,000 on small vision tasks while deployed feed-forward widths run from 14,336 to 73,728. Handwave Either the diagnostic does not transfer or the models are in a regime the theory says has no features, and the field has not chosen.

Frontier This subject waits on results from four neighbouring fields, and none of them is being produced for its benefit.

Frontier Statistical physics of disordered systems. Every exactly solvable model in this field — random features, the kernel limit, mean-field dynamics — is imported from the physics of high-dimensional random landscapes, and the transfer has so far been of techniques rather than of results. Speculative Glass Transition Theory is the map’s nearest page on the same mathematics and carries the cautionary case: a subject where the solvable mean-field model has been available for forty years and the finite-dimensional system it was meant to explain still is not explained. Handwave The structural parallel is close enough to be worth stating and not close enough to be an argument.

Established Random matrix theory and free probability supply the spectral densities in which the double-descent solution is expressed, and the closed algebraic equations for their Stieltjes transforms are what make that solution exact. Speculative The dependency is real and it binds in one direction only: more powerful spectral tools would extend the solvable class, but no spectral result will produce a feature-learning network.

Frontier Measurement. This brief cannot test a theory of capability without an instrument for capability, and Intelligence Measurement records that no benchmark in the field has been validated the way a human test would be. Speculative The dependency is asymmetric: loss-based theory can proceed without it, capability-based theory cannot proceed at all.

Frontier Compute and data provenance. Any exponent measured on a corpus of unknown composition is an exponent of unknown generality, which puts Synthetic Data and Training Provenance upstream of the universality claim, and The Physical Stack of AI and Compute Concentration upstream of who can measure anything at all. Handwave The energetic floor under the whole activity belongs to Thermodynamics of Computation, and it is many orders of magnitude below current practice, so it constrains nothing here yet.

Frontier Six experiments, ranked by how much a result would change this brief’s assessment rather than by difficulty. The first is the decisive one and it is the cheapest thing on this list that nobody has done.

Speculative E1, the decisive experiment: a pre-registered prediction of a scaling exponent, or of the interpolation-threshold location, from architecture and data statistics alone, published before the runs. The form is specific. Choose an architecture family and a corpus that no published fit has used. From the corpus statistics and the architecture, predict the parameter exponent, the data exponent, the irreducible loss and the compute-optimal token-to-parameter ratio, with intervals. Deposit the predictions. Then train the ladder and report. Frontier Every ingredient exists: the candidate theories give formulas, the ladders are run routinely, and the cost is a rounding error against a frontier pretraining budget. Handwave It has never been done, and the reason it has never been done is that a public failure would be more informative than a private success is valuable.

Speculative E2, the same experiment in its cheap form: predict where double descent peaks. The 2025 particle-physics study found its interpolation threshold by watching the training loss vanish, across models from 774 to 139 million parameters. Speculative A theory that named the threshold in advance — in parameters, for a stated architecture, dataset size and label-noise level — would be the first quantitative prediction this subject has made about a phenomenon it claims to understand. Frontier The random-feature solution predicts a peak at the threshold preceded by a dip, and the threshold in that model is where parameters equal samples. Handwave Whether that carries to a network that learns features is the entire question, and it is answerable on a single GPU.

Frontier E3: the intrinsic-dimension test, run on real corpora rather than teacher–student data. The claim that the resolution-limited exponent is about 4 divided by the data manifold dimension matches measurement where the dimension is set by the experimenter and is described by its own authors as less clear on CIFAR-10 and CIFAR-100. Speculative The test is to estimate intrinsic dimension on several text and image corpora by independent methods, predict the exponents, and fit them afterwards. Speculative A confirmation would be the first derivation of a scaling exponent from a measurable property of data; a failure would retire the position, which is why it is worth running either way.

Frontier E4: the hyperparameter-transfer falsification suite, which is half-built already. Transfer is reported to hold across width and expert count, to fail across expert granularity, to degrade with the Muon optimiser where transferred settings produce diverging scaling curves, and to break on hard-example subsets. Frontier The experiment is to enumerate the axes — width, depth, batch size, context length, expert count, granularity, optimiser, precision — and report for each whether transfer holds, with seed variance. Established This is the one item on the list that is being done piecemeal by several groups, and the only thing missing is that nobody is doing it as a single controlled sweep.

Speculative E5: the compressibility pre-test. If compressibility is the right notion of capacity, then a compressibility statistic computed on a partially trained model should predict the final generalisation gap better than parameter count or norm. Frontier The ingredients exist, since non-vacuous bounds for pretrained language models are now obtained by exactly such a compression argument, and they get tighter for larger models. Handwave Turning a post-hoc bound into a forecast is the step nobody has taken, and until someone does, “capacity is compressibility” is a slogan.

Speculative E6: the adversarial universality test. Deliberately construct a corpus, optimiser and architecture family chosen to break the fitted exponents — a corpus with a different spectral decay, a second-order optimiser, a non-transformer sequence model — and report whether the three-term form still fits and whether the exponents move. Handwave Universality has been asserted for six years on the strength of samples drawn from one architecture family, one optimiser family and one kind of corpus. Speculative A negative result here would be the most important finding on this page, and it would cost less than a week of a frontier cluster.

Established And the things that are reported as tests of theory and are not. A new leaderboard state of the art is not a test of a generalisation theory. Established A reported emergent capability is not a phase transition unless the metric is shown to be continuous, and a discontinuous metric applied to a smooth loss curve manufactures one. Frontier A scaling curve fitted to four points and extrapolated two orders of magnitude is a projection, not a measurement. Speculative Each of these appears in the literature described as evidence for a theory of deep learning, and none of them discriminates between any two positions in section 4.

Frontier What this subject needs built is not hardware. It is instrumentation and disclosure around training runs that already happen.

Established An instrumented ladder. The minimum useful artefact is a family of runs at a dozen sizes with every loss value, seed, learning-rate schedule, tokenisation, data mixture and precision recorded and published, including the runs that diverged. Speculative The 2024 Chinchilla replication exists because someone read points off a figure; the same work done against a released table would have taken a week rather than two years. Frontier The 2024 parameterisation study is the existence proof that this scale of sweep is affordable: fourteen sizes to 26.8 billion parameters, three optimisers, four parameterisations, tens of thousands of models.

Frontier Numerically controlled optimisers. Scaling experiments need optimisers whose behaviour does not change qualitatively with model size for reasons unrelated to the question — which is what epsilon-driven gradient underflow in Adam was doing. Speculative A scale-invariant variant that eliminates the epsilon hyperparameter is the kind of engineering that makes a measurement possible, and it arrived in 2024 rather than 2019 because nobody was looking.

Frontier Evaluation harnesses that carry protocol. Everything Intelligence Measurement specifies — shot count, prompt template, answer extraction, contamination status, intervals, dollars per task — has to travel with every point on a downstream scaling curve, or the curve is uninterpretable. Speculative That is an engineering requirement, not a methodological nicety: the 2025 finding that benign setting changes reverse the shape of a scaling curve is exactly what an uncontrolled harness produces.

Speculative And compute that is not owned by the party whose fit is being checked. Compute Concentration and The Physical Stack of AI describe the market that decides this; the requirement it imposes here is a standing allocation, on the order of a few thousand accelerator-years, reserved for replication rather than for capability. Handwave No such allocation exists anywhere.

Frontier The nearest page on this map is Intelligence Measurement, and the division is clean. That brief asks whether a reported number means anything — protocol, contamination, intervals, construct validity, the 6.49% item error rate in a benchmark everyone quotes. This one assumes the number is real and asks why it is what it is. Established Read in the other direction, this page is the reason that one matters: if capability were derivable from architecture and data, benchmark validity would be a convenience rather than the binding constraint.

Frontier Artificial General Intelligence inherits this brief’s central weakness directly. Threshold claims on that page rest on extrapolated curves, and the honest statement of what an extrapolated curve is worth is here: the loss curve extrapolates well and the capability curve does so in 39% of measured cases. Speculative Artificial Scientists and Technology Forecasting both build arguments on fitted exponents and both should be read with the Chinchilla refit in hand.

Speculative Glass Transition Theory is the structural sibling nobody in machine learning cites. Same mathematics, same gap between an exactly solvable mean-field limit and an unexplained finite-dimensional system, and four decades of prior experience with what happens to a field in that position. Handwave The analogy is not evidence and this page does not use it as any.

Frontier Downstream sit the pages that spend money on this brief’s exponents: Compute Concentration for who can run the experiments, The Physical Stack of AI for what the runs cost in silicon, Synthetic Data and Training Provenance for whether the corpus underneath a fitted exponent is what it claims to be, and Thermodynamics of Computation for the floor beneath the energy bill. Speculative Philosophy of Science owns the general version of this page’s awkward question: what a field is doing when its predictive instrument works and its explanatory account does not exist.

Frontier This field’s institutional problem is specific and it has a documented instance: the most cited quantitative result in modern machine learning carried an optimiser bug and an impossible confidence interval for two years, and was corrected by an outside group working from screenshots.

Speculative Requirement one: publication of the fitting data with any published scaling law. Loss values, model sizes, token counts, seeds, and the fitting code. Established The cost is a table; the benefit is that the 13% rounding bias, the unconverged optimiser and the factor-of-three inconsistency between a paper’s own approaches are visible on the day of publication. Handwave No venue in this field requires it.

Speculative Requirement two: pre-registration for predictive claims. A theory that predicts an exponent should deposit the prediction before the ladder runs. Speculative This is standard in clinical research and unheard of here, and its absence is why the field cannot distinguish a theory that predicts from a functional form that fits.

Frontier Requirement three: an evaluator with compute and without a product. The organisations that can afford training ladders are the organisations whose allocation decisions the exponents justify. Speculative Intelligence Measurement makes the same argument for benchmarks and reaches the same dead end: independence of judgement without independence of access is a weaker guarantee than it reads as, and independence of access costs accelerator-years. Handwave The cheapest institutional fix on this page is also the only one with no constituency: an agreement, written before the next result, on what would count as a scaling law being wrong.

Speculative Requirement four: negative results have to be publishable. The 2025 catalogue of hyperparameter-transfer failures and the 2025 finding that only 39% of downstream tasks scale predictably are both, in effect, reports that a widely believed thing is narrower than believed. Handwave Both appeared in workshop or findings tracks rather than in main proceedings, which is the field pricing its own corrections.

Frontier The ethical content here is unusual because the object is an equation rather than a device: a fitted exponent has become a policy input, and it was fitted badly at least once.

Speculative Compute thresholds in regulation are scaling-law artefacts. The reasoning that picks a training-compute number as a regulatory trigger assumes a stable, monotone map from compute to capability. Established The measured map from pretraining loss to downstream capability is predictable in 39% of cases and includes inverse and non-monotonic regimes. Handwave That does not make thresholds wrong; it makes their stated justification weaker than their advocates believe, and nobody has published the sensitivity analysis.

Speculative Forecasts inherit fit errors silently. Downstream work — capability timelines, investment cases, national compute strategies — consumed the 2022 exponents for two years. Frontier The refit moved the data exponent 28% and the allocation exponent from 0.454 to 0.5126. Speculative Nobody has published a reconciliation showing which downstream conclusions survive, and this page treats any forecast citing the 2022 parametric fit without noting the correction as provisionally unsound.

Frontier The distributive fact is that understanding is being privatised along with capability. A theory here is tested with ladders; ladders cost compute; compute is concentrated. Speculative The result is not secrecy so much as asymmetry: a laboratory can know from internal runs that a published exponent is wrong and have no obligation, and often no incentive, to say so. Handwave A science in which the negative results are commercially valuable to withhold is a different epistemic object from one in which they are not, and this is the clearest case of it on the map.

Speculative And a narrower point about honesty of register. “Scaling law” is a term of art borrowed from physics, where it denotes a relation derived from a theory and confirmed by measurement. Handwave Here it denotes a curve fitted to measurements with no derivation behind it, and the borrowed word does argumentative work the underlying object cannot support.

Speculative If the power law is real and general, then machine capability is a purchasable quantity, and the sentence that follows is about capital rather than about intelligence. Frontier That is the reading the industry has already adopted, and the whole structure of Compute Concentration follows from it.

Established The exponents, taken at face value, are unkind to that reading. Loss falls as a small power of compute, so each successive equal fraction of loss reduction costs a multiplicative increase in spend. Frontier A subject in which progress is a power law with a small exponent is a subject with a rising price per unit of progress, and the observed pattern of capital expenditure is consistent with that. Speculative Which of those two readings is correct is not a philosophical question; it is the value of one number, and section 7 asks for it.

Speculative If instead the exponents are properties of a decade — of transformers, Adam, web text and bfloat16 — then the strategic picture inverts. A discontinuity would come from a change of architecture family or data regime rather than from spend, the advantage of incumbency would be smaller than it appears, and the correct national investment would be in breadth rather than in scale. Handwave No evidence currently distinguishes these worlds, because no one has run the adversarial universality test.

Frontier There is also a slower civilizational consequence and it is about science rather than about machines. Deep learning is the most economically consequential technology ever deployed without a theory of why it works, and it is being used to do science — see Artificial Scientists. Speculative A field that accepts an unexplained instrument at the centre of its method has done so before, with steam engines before thermodynamics and with anaesthesia before pharmacology, and in both cases the theory eventually arrived and changed the practice. Handwave Whether the analogy holds is unknowable from here, and a page that claimed otherwise would be doing what this page exists to flag.

These horizons track what would have to be published for the assessment on this page to change, because there is no deployment programme to track:

• 10 yr: Frontier Expect the empirical side to keep improving: released ladders, controlled parameterisation sweeps, and learned extrapolators that beat the parametric form by wider margins than the 44% error reduction reported in 2026. Speculative Expect the first pre-registered exponent prediction, and expect it to be wrong, which is the useful outcome. Speculative Expect hyperparameter transfer to be mapped axis by axis, with the current failures — granularity, second-order optimisers, hard-example subsets — either explained or generalised into a rule. Frontier Do not expect a derivation of a scaling exponent from first principles on this horizon; nothing in the current literature is close.
• 25 yr: Speculative This is the window in which the field either acquires a capacity measure that predicts before training — most plausibly a compressibility statistic, given that the only non-vacuous bounds get tighter for larger models — or concludes that no such measure exists for gradient-trained networks. Speculative A positive result would make architecture selection a calculation rather than a search, and would be the first time this subject predicted anything. Handwave A negative result is equally informative and considerably more likely on the current evidence.
• 50 yr: Handwave The plausible shape is consolidation into a statistical-physics account in which power-law loss is a generic consequence of power-law data spectra, with exponents predicted from measurable data statistics and architecture entering only through a small number of effective parameters. Speculative The alternative, equally consistent with the record so far, is that the whole regularity turns out to be an artefact of one architecture family and one training corpus, and is retired the way the pre-1990 neural-network literature was.
• 100 / 250+ yr: Handwave Beyond useful forecasting, and the defensible statement is about shape rather than date. Handwave Subjects with excellent instruments and no theory resolve in one of two ways — the theory arrives and the instrument is understood, or the instrument is superseded and the question stops being asked — and nothing on this page indicates which. Handwave Any specific claim about what a completed theory of generalisation would permit is an assertion, and this page does not make one.

• Depends on One typed dependency on this map and it is a measurement dependency rather than a physical one: Intelligence Measurement has to supply an instrument for capability before any theory here can be tested against capability rather than against loss, and that brief records that no such instrument has been validated. Glass Transition Theory shares the mathematics of high-dimensional random landscapes but blocks nothing, and Compute Concentration and The Physical Stack of AI determine who can run the experiments without producing a result this page waits on.
• Requires (not on this map) Six conditions, in the order of the tokens above. A predicted exponent. Every exponent in use was fitted after the runs; a theory that named one in advance, deposited the prediction, and survived the ladder would convert this subject from phenomenology into science, and no candidate position in section 4 has attempted it. A capacity measure computable before training. Parameter count is measurably wrong, norm-based capacity grows with data while error falls, and compressibility — the only candidate that gets tighter for larger models — has never been shown to forecast anything. The ladder, published. Loss values, seeds, mixtures, precision and the diverged runs, released as a table; the 2024 Chinchilla correction exists because someone read points off a figure, and took two years. Pre-registration. A deposited prediction before the runs, which is routine in clinical research and unknown here, and without which a fitting form and a theory are indistinguishable. Independent compute. On the order of a few thousand accelerator-years reserved for replication rather than capability, held by a body with no model to sell. Seed variance. Reported as standard in scaling and transfer studies, because the field currently cannot tell a method finding from a variance finding, and at least two published transfer failures are candidates for either.
• Enables A predicted exponent would make the compute-allocation arguments in Compute Concentration and the threshold arguments in Artificial General Intelligence calculable rather than extrapolated, and would give Technology Forecasting its first machine-learning forecast with a derivation behind it. No typed enabling edge is claimed, because a theory that has never predicted anything enables nothing on a schedule, and the honest downstream consequence of the current state is that every page quoting a fitted exponent inherits an unquantified error bar.
• Adjacent Intelligence Measurement for whether the dependent variable means anything; Glass Transition Theory for the same mathematics and a forty-year cautionary case; Artificial Scientists and Artificial General Intelligence for the arguments built on this page’s curves; Synthetic Data and Training Provenance for what is actually in the corpora the exponents were measured on; Thermodynamics of Computation for the energetic floor under the training run; and Philosophy of Science for what a field is doing when its instrument works and its explanation does not exist.

Established “Scaling laws are laws.” They are curves fitted to measurements. A law in the sense borrowed from physics is derived from a theory and confirmed by measurement; nothing in this field has been derived. Established The underlying regularity is real — loss falling as a power of model size, data and compute, over trends the 2020 study reports as spanning more than seven orders of magnitude — and that is a strong empirical finding. Speculative It is not a law, and the word is doing argumentative work the object cannot support.

Established “Chinchilla proved 20 tokens per parameter.” Two of the paper’s three approaches support roughly that ratio and Chinchilla itself was trained at it. Established Its third approach, the parametric fit that everyone quotes and that supplies the exponents, implies about 70 tokens per parameter, and that inconsistency was in the paper from the start. Frontier The 2024 replication’s refit restores consistency with about 20 — so the practice was right and the equation everyone cited was not.

Frontier “The Chinchilla result was overturned.” It was not. The compute-optimal finding — that models of the era were undertrained relative to their size — survives, because it rests on the two approaches that were not broken. Established What was overturned is the fitted parametric form: a rounding bias of roughly 13% for a typical dataset size, an optimiser that stopped short because of a poor loss scale, Huber losses averaged where they should have been summed, and a reported interval of 0.454 to 0.455 that would need on the order of 600,000 runs to justify against about 400. Speculative Both statements are true at once, and most secondary accounts pick one.

Frontier “The neural tangent kernel explains deep learning.” It explains infinitely wide networks with a static representation, which is a different object. Established Convergence to that regime requires width polynomial in depth, orders of magnitude past deployed architectures; the stability constant needed for its prediction about second-order methods stays many orders of magnitude above the required value at widths up to 10⁷; kernel-theoretic exploration parameters in a neural bandit perform no better than random at depth three; and the predicted reduction in catastrophic forgetting with width vanishes once networks are trained to convergence. Speculative Three tested predictions, three failures — and the kernel remains the most useful mathematical object in the field, because being wrong precisely is more useful than being vague.

Speculative “Infinite-width networks do not learn features, and that is settled.” A 2025 analysis reports that per-neuron deviations from initialisation vanish while the aggregate norm of those deviations does not — 4.89 at width ten thousand and 10.56 at thirty thousand, against per-neuron figures of 0.095 and 0.138 — and proves that feature correlations between data pairs vary during training with probability one in the limit. Handwave If that stands, the “lazy regime” is partly an artefact of which quantity the limit was taken on, and a decade of intuition needs rewriting. Speculative It is one paper and this page flags it as such.

Established “Double descent explains why bigger models work.” Double descent is a description, not an explanation. Established Its measured content is that test error peaks at the interpolation threshold in model size, in epochs and in sample count, and that the effect is most prominent under added label noise. Frontier Its exact theory exists for the random-feature model, whose authors state it is not fully mathematically controlled and applies to fixed rather than learned features. Speculative Seven years after the phenomenon was named, the 2025 study that reproduced it on particle-physics data located its interpolation threshold by watching the training loss vanish, because no theory would say where it was.

Frontier “More data always helps.” The paper that named double descent reports transformers trained with 4.5 times more samples doing worse at intermediate model sizes, because more data moves the interpolation threshold to the right. Speculative The regime is narrow and it is real, and it is the cleanest existing demonstration that the relationship between data and performance is not monotone in the way the scaling-law summary implies.

Established “Generalisation bounds are all vacuous, so theory has nothing to offer.” That stopped being true in 2024, when compression arguments with prediction smoothing produced the first non-vacuous bounds for pretrained large language models, with larger models getting better bounds than smaller ones. Frontier The bounds are on a compressed reparameterisation rather than on the deployed model, so they establish that non-vacuity is achievable rather than that the shipped system is explained. Speculative The interesting part is the inversion: on this measure capacity and guarantee move together rather than against each other, which no classical account predicts.

Frontier “Hyperparameters transfer, so scaling is solved.” Transfer holds across width, and a 2024 sweep of tens of thousands of models found it holds for every parameterisation tested rather than only the one it is named after, with a per-layer learning rate for standard parameterisation beating it. Frontier Transfer fails across mixture-of-experts granularity, degrades under the Muon optimiser to the point of diverging scaling curves, and breaks on hard-example subsets. Speculative The honest summary is that a useful heuristic has been mapped along one axis and is unmapped along six others.

Established “Loss predicts capability.” Across 46 downstream tasks, performance improves smoothly and predictably with pretraining loss in 39% of cases; the rest are inverse, non-monotonic, noisy, trendless or abrupt, and benign changes to the setting can reverse the shape. Established A 2026 learned extrapolator cuts downstream prediction error from 3.56% to 1.99% precisely by refusing to assume a parametric form. Speculative Anyone quoting a smooth capability projection off a loss curve is asserting the 39% case and should say so.

Frontier And the correction this page makes to its own commission. The anchor named for this brief is neural scaling laws — loss falling as a clean power law over many orders of magnitude, with Chinchilla exponents allocating real training budgets. Established The first half is right and the second half is right only by accident. The power law over more than seven orders of magnitude is a solid 2020 measurement. The Chinchilla exponents that allocate budgets are the ones from the two approaches that were not the parametric fit; the parametric fit itself, which is what the phrase “Chinchilla exponents” normally denotes, was mis-estimated, implies a token ratio three and a half times the one actually used, and reported an interval fifty times too narrow. Speculative The field allocated billions of dollars of compute using a rule of thumb that was correct, derived from an equation that was not. Handwave That is the strongest single argument on this page for why a theory would be worth having.