Forecasting Replication

Speculative If a crowd of researchers can price the truth of a published claim before anyone replicates it, and price it better than simply betting on the base rate — which the DARPA SCORE programme has begun to test in public and section 7 says is still open on domains nobody has forecast — the prize is a triage signal for reliability bought for the cost of a survey rather than the cost of a replication. That conditional is the whole brief: the accuracy is real and repeatedly measured, but so is the base rate it has to beat.

Established The magnitude of the saving is set by the gap between forecasting and replicating. A direct replication of a single social-science experiment in the large projects cost thousands of dollars and months to years of a team’s time; the Reproducibility Project ran for years across 270 authors to resolve 100 psychology studies, and the SCORE programme mobilised 865 researchers to reproduce, reanalyse and replicate a subset of 3,900 papers. Against that, an aggregated forecast — a survey or a prediction market — resolves a claim in weeks for the cost of recruiting a few dozen to a few hundred forecasters. Across the pooled projects those forecasts land at roughly 70 percent classification accuracy: 73 percent for markets and 66 percent for surveys over 103 findings, 71 percent out of sample on the social-science project, and 76 to 78 percent for the two human methods scored by SCORE.

Established The gain accrues to whoever must decide what to build on before the replication exists, which is almost everyone. A funder choosing which result to extend, a policy unit citing an effect, a journal deciding what to flag, a meta-analyst weighting a literature: today reliability is established only by replicating, and well under one percent of published claims is ever replicated. A calibrated 70-percent triage signal does not settle any single claim — it is a probability, not a verdict — but it lets scarce replication budgets be aimed at the claims where the forecast and the stakes disagree.

Speculative The condition that caps the whole case is that the base rate is roughly a coin flip, so 70 percent accuracy is a smaller edge than it sounds. When about half of tested claims fail, a rule that predicts failure every time is already correct most of the time, and one reanalysis found a prediction market beat that null rule by a margin that was not statistically significant. Whether aggregated forecasting adds decision-relevant information beyond the base rate, and does so in fields outside psychology and economics, is exactly what section 7 says the forward scoring is now measuring.

Established This brief is about a single, unusually clean idea: elicit a probability that a published finding will replicate, before the replication is run, and later score that probability against the outcome. The finding is the unit; the forecast is a number between zero and one; the replication is the ground truth. Because the ground truth arrives, this is one of the few places in the science of science where a prediction can be graded rather than admired. It sits beside Research Front Detection, which forecasts which topics will grow, and Technology Forecasting, which forecasts rates and dates; this brief owns the narrower question of whether a specific claim is true.

Established Three elicitation methods compete, and they are variations on aggregating judgement. A survey asks each forecaster for a probability and averages them. A prediction market lets forecasters trade a contract that pays out if the claim replicates, so the price is a continuously updated collective estimate. Structured expert elicitation — the repliCATS process built on the IDEA protocol — has small groups Investigate, Discuss, Estimate and Aggregate, deliberately mixing private judgement with argument. Speculative Whether deliberation improves on a plain average or merely spreads one confident voice through a group is itself contested. A fourth family, machine-learning models trained on the text or statistics of papers, tries to remove the human forecaster entirely.

Established The claim this brief lands is that all the human methods work, at about the same modest level, and that the level is anchored to a base rate nobody escapes. Prediction markets, surveys and structured elicitation all cluster around 65 to 78 percent classification accuracy across every large project that has scored them. Frontier The differences between the methods are usually smaller than the gap between the accuracy and the base rate. The differences between the methods are usually smaller than the difference between the accuracy and the base rate, and the machine-learning models that looked competitive in early studies did not hold up when SCORE scored three of them blind. The honest summary is a real, replicated, unspectacular signal. Established On the current evidence humans carry that signal better than machines do.

Frontier The most important recent development is that the scoring finally ran forward rather than backward. Every early study forecast studies that were about to be replicated inside the same project, then scored the forecast against that replication. The DARPA SCORE programme (2019 to 2024) instead assembled forecasts on thousands of claims and had independent teams replicate a subset afterwards, and the Center for Open Science’s Predicting Replicability Challenge now scores fresh teams on domains they have not seen. That shift from hindsight to prospective scoring is what makes the numbers in section 3 evidence rather than demonstration.

Established A note on sourcing. This brief was commissioned in September 2026. Several primary papers sit behind publisher paywalls this research base could not fetch — the original 2015 prediction-market paper in PNAS and the 2016 economics replication in Science among them; those are cited by name, and every quantitative claim carried here comes from a source that was fetched and read, dated no later than August 2026.

Established The base rates the forecasts must beat are now measured across a dozen fields, and they sit near one half. The Open Science Collaboration’s 2015 Reproducibility Project replicated 36 percent of 100 psychology studies by significance; the Experimental Economics Replication Project replicated 61 percent of 18 laboratory economics experiments; the Social Sciences Replication Project replicated 13 of 21 studies from Nature and Science, or 62 percent. The SCORE programme’s own replication arm found 49 percent of 164 papers replicated on the significance criterion. Whatever a forecaster is doing, it is doing it against a coin that is close to fair. Frontier That base rate is why an accuracy figure means little until it is set beside the accuracy of simply betting that half of all claims fail.

Established The founding result is that a prediction market on 41 psychology replications called 71 percent of them correctly. Dreber, Pfeiffer, Almenberg, Isaksson, Wilson, Chen, Nosek and Johannesson ran markets over two rounds on studies from the Reproducibility Project and reported the markets correctly predicted the outcome of 29 of 41 studies, outperforming a survey of individual forecasts taken before trading. That paper, in PNAS in 2015, is the anchor the commission named, and it holds: markets beat individual judgement on this task. Established The markets also returned an aggregate replication estimate close to the rate the project later observed.

Frontier The same result also carries the first warning, and it survives reanalysis. A later reanalysis split the two markets and found the first predicted 91 percent (20 of 22) and the second only 53 percent (10 of 19), a difference significant at p = 0.016, and noted that a rule betting on failure every time would have scored 63 percent given the base rate — so the market’s edge over that null was not statistically significant (p = 0.177). The signal is real; its margin over the base rate in that first study was fragile. The whole field has been narrowing that uncertainty ever since.

Established Scaling up and pooling four projects returns the same central number more stably. Gordon, Viganola, Bishop, Chen, Dreber, Goldfedder, Holzmeister, Johannesson, Liu, Twardy, Wang and Pfeiffer pooled markets and surveys across the psychology, economics, social-science and Many Labs 2 projects: over 103 findings, the prediction markets had an accuracy of 73 percent (75 out of 103 studies), while the surveys had an accuracy of 66 percent (68 out of 103 studies). Markets carried a significantly lower absolute error than surveys (0.384 against 0.423, p < 0.001), but the difference in classification accuracy was not significant (chi-squared 1.12, p = 0.29). Established The market and survey price series correlated with the outcomes at 0.58 and 0.56 respectively, both highly significant.

Established On a genuine out-of-sample test the number held. Altmejd, Dreber, Forsell, Huber, Imai, Johannesson, Kirchler, Nave and Camerer trained a model on 131 study-replication pairs from four projects and, applied to the pre-registered Social Sciences Replication Project, predicted 71 percent of replications correctly with an area under the curve of 0.73 across 21 Nature and Science studies. Camerer and colleagues had already reported, in that project, that peer beliefs of replicability were strongly related to actual replicability, and that the market-derived overall replication rate was nearly identical to the one observed. Frontier That the estimate held on pre-registered studies nobody had forecast before is the strongest single reason to think the signal is not an artefact of hindsight.

Frontier Structured expert elicitation matches or beats the markets, which complicates the tidy story that markets are best. The repliCATS process assessed 3,000 research claims over an 18-month period in Phase 1 of SCORE, plus 900 more in Phase 2, using the IDEA protocol. Its authors report a classification accuracy of 84 percent and an area under the curve of 0.94 — figures at or above every other method — though these are the programme’s own validation numbers (the method’s developers on their own instrument) and the earlier literature they compare against ranged between 61 and 86 percent. Frontier Structured deliberation may add most exactly where a lone forecaster would anchor on the p-value and stop. On the Many Labs 2 subset the same paper records prediction markets at 75 percent and pre-market surveys at 67 percent.

Established The decisive shift is that SCORE scored the methods forward, at scale, and published who won. The programme examined 3,900 papers published 2009 to 2018 across 62 journals and 12 disciplines, mobilising 865 researchers. When human forecasts were scored against the completed replications, repliCATS reached 76 percent and Replication Markets 78 percent success by their best-performing metric. The three automated machine-learning systems tested — Synthetic Markets, MACROSCORE and A+ — were, in the programme’s words, not consistently effective at predicting which claims would replicate, with the authors cautioning against earlier optimism about machine methods. Established That the human crowds beat the machines on a blind, prospective test is the field’s most consequential recent result.

Frontier A 2024 decision-market study closed the loop from forecasting to selection. Holzmeister, Johannesson, Camerer and 19 co-authors had 162 social scientists trade on 41 MTurk studies from PNAS, then replicated the 12 highest-priced and 12 lowest-priced. The top 12 replicated at 83 percent and the bottom 12 at 33 percent, with a point-biserial correlation between market price and outcome of r = 0.505. That is the mechanism working as intended: the market did not just describe reliability, it sorted studies by it well enough to be used as a filter. Speculative Whether the same ordering survives outside online experiments is untested.

Established Machine-learning models were the field’s brightest hope and are its clearest cautionary tale. Yang, Youyou and Uzzi reported a text-based model, trained on 96 Reproducibility Project studies and tested on 413 manually replicated studies from 80 journals, reaching 0.65 to 0.78 accuracy out of sample and comparable to prediction markets. Herzenstein, Rosario, Oblander and Netzer found the language of a paper predicts replication with an area under the curve near 0.72 over 299 replications. Both are real signals; both are small samples; and SCORE’s blind test of three deployed systems is the reason this brief flags the machine approach frontier rather than established.

Established The community also forecasts that replication rates differ by field, and it has been proved right about the ordering. The SCORE community-forecast survey found participants expected economics to replicate at 58 percent, psychology at 42 percent and education at 42 percent, and expected rates to rise for more recent work. The subsequent replications broadly bore out that fields differ, which matters because a forecasting instrument calibrated on psychology cannot be assumed to transfer to criminology or finance without being rescored there.

Frontier Does aggregated forecasting beat the base rate by enough to change a decision? Seventy percent accuracy against a fifty-percent base rate is a real edge; seventy against a sixty-three-percent null is barely one. The quantity that matters is not raw accuracy but calibration and discrimination — whether the probabilities are trustworthy and whether they separate replications from failures — and the Predicting Replicability Challenge found that between rounds it was calibration, not accuracy alone, that moved. Nobody has published a decision-theoretic value of the signal for a named funder. Established The Challenge’s own analysis separated calibration from resolution and found calibration did most of the moving between rounds.

Frontier Why do markets and surveys land so close together? The theory says markets should aggregate dispersed private information and incentivise effort, yet across projects the market edge over a simple survey average is small and often not significant. Either the surveys already capture most of the recoverable signal, or the markets are thin enough that they mostly track the same crowd belief. Distinguishing the two requires varying market depth and incentive size while holding the forecaster pool fixed, which no study has done cleanly.

Frontier What are the forecasters actually reading? Reported cues include statistical power, effect size, p-value proximity to 0.05, and surprisingness, and the language-model work suggests the writing itself carries signal. If the recoverable information is mostly the original statistics, an aggregated human forecast is an expensive way to compute something a formula could, and the machine models should win; SCORE says they did not, which means either the humans use something beyond the statistics or the models are not yet exploiting the statistics well. Speculative If the recoverable signal is mostly the reported statistics, a formula should in principle match the crowd.

Frontier Does the signal generalise beyond the two fields it was built on? Almost every scored forecast concerns experimental psychology or economics. SCORE deliberately spanned 12 disciplines, and its headline conclusion — that credibility is multidimensional with no broadly applicable shortcut — is a warning that a detector tuned on psychology may not carry to fields with different designs, base rates and publication norms. Frontier The one genuinely new domain scored so far, health behaviour, regressed rather than confirmed.

Speculative Can a forecast be made to price something other than statistical significance? Replication is only one dimension; SCORE separated reproduction (rerunning the analysis on the same data), robustness (independent reanalysis) and replication (new data). A forecast that predicted robustness rather than significance would answer a different and arguably more useful question, and nobody has run large markets on that target.

Established The binding constraint is ground truth: forecasts can only be scored where a replication was actually run. Across every project combined, the number of claims with both a forecast and a completed replication is in the low hundreds, and SCORE’s replication arm resolved 164 papers. A model trained and tested on samples this small cannot support fine distinctions between methods, and it is why a difference of seven accuracy points between markets and surveys fails to reach significance. Frontier Larger scored corpora are the single most valuable input the field lacks.

Established The base rate is a denominator that makes precision hard to interpret. Because roughly half of tested claims fail, accuracy figures are inflated by the ease of the majority class, and a method must be judged on its lift over betting the base rate, not on accuracy alone. The field has largely moved to Brier scores and area-under-the-curve for this reason, but a great deal of the public discussion still quotes a bare percentage that the base rate half-explains. Established An accuracy figure quoted without its base rate is close to uninformative.

Frontier Selection of which claims to forecast is unaudited. The claims that reach a replication project are chosen — for feasibility, for citation prominence, for being surprising — and a forecaster scored only on that curated set has an easier or harder task than one facing the literature at random. No project has drawn its forecasting targets as a random sample of a defined population of claims, so the reported accuracies do not have a clean population to generalise to.

Established Forecaster recruitment is a standing cost and a standing bias. Markets and elicitations need dozens to hundreds of domain-literate participants, drawn overwhelmingly from the same open-science-adjacent community whose priors about which results are fragile are correlated. That correlation is exactly what a market is supposed to break down, and it is unmeasured here. Speculative A market recruited from a different community might return a different price for the same claim.

Frontier Machine methods are bottlenecked on labelled data and on leakage. A text model needs thousands of papers labelled by replication outcome to learn robustly, and only hundreds exist; worse, if training and test papers share authors, journals or eras, the model can learn to predict the label from provenance rather than from content. SCORE’s blind, prospective test is the setting most likely to expose that, and it is where the machine systems underperformed.

Established The enterprise depends on replications being run and published, which is the expensive input. A forecast has no value without a resolution, and resolutions come from replication programmes that are episodic, grant-funded and concentrated in a few fields. The rate-limiting resource is not forecasting capacity but replication capacity, and the two are funded by different people on different timescales. Frontier Any expansion of forecasting is therefore rate-limited by the funding of replication, not by the cost of the forecast.

Established It depends on a fixed, published resolution criterion agreed before the forecast. Replication can be scored by significance, by effect-size overlap, by a Bayesian true-positive rate, or by relative effect size, and the Social Sciences Replication Project reported its rate varying from 57 to 67 percent depending on the indicator chosen. A forecast is only gradeable against a criterion fixed in advance; a criterion chosen afterward lets the grader move the goalposts. Established The Social Sciences Replication Project’s own rate moved by ten points with the indicator chosen.

Frontier It depends on the statistics of calibration, which the field has adopted only recently. Proper scoring rules, reliability diagrams and discrimination metrics are standard in weather and in Collective Intelligence’s forecasting literature, and their adoption here is what let the Predicting Replicability Challenge distinguish improvements in calibration from improvements in resolution between rounds. The methods are borrowed, not native, and are still unevenly applied.

Established It depends on machine-readable claims. A forecast is made about a specific claim, and extracting one testable claim per paper from prose is itself a labour-intensive step that SCORE performed by hand across thousands of papers. Automating that extraction is a precondition for scaling forecasting beyond hand-curated sets, and it is not solved. Speculative Language models might automate the extraction, but their reliability on this task is unmeasured.

Speculative It may come to depend on the outputs of automated research systems. If a growing share of papers is machine-generated, the base rate, the writing cues and the very meaning of a claim shift, and a forecasting instrument calibrated on human-authored literature would need recalibration on a moving target.

Frontier The decisive test is the one the SCORE programme has begun to run in public: dated forecasts, made before the replication, scored against the outcome by teams with no forecasting product to sell. That design is what converts a demonstration into evidence, and it has already returned its first verdict — human aggregation at 76 to 78 percent, three machine systems not consistently effective, over replications the forecasters could not have seen. The result that would most change this brief’s assessment is the continuation of that scoring across fields beyond psychology and economics, because the single largest open question is whether the signal generalises.

Established The prospective scoring is not hypothetical; it is running now on unseen domains. The Center for Open Science’s Predicting Replicability Challenge scores fresh teams against replications completed after they forecast: Round 1 had 10 teams forecast 132 claims of which 52 percent replicated, and no team beat a baseline Brier score of 0.25; Round 2 had 15 teams forecast 130 claims of which 55 percent replicated, and all teams beat the baseline. This is the natural experiment the field most needed, and it is already producing a scored public record. Established That record is dated and public, which is what makes it evidence rather than testimony.

Frontier The most recent round is the sternest test so far, and the field partly failed it. Round 3, with results published on 7 August 2026, scored 15 teams on 41 completed replications from the Replicability Project: Health Behavior — a domain none had trained on — of which 46 percent replicated; only 8 of 15 teams beat the coin-flip baseline, and weaker calibration explained much of the decline. That regression on a new domain is the clearest existing evidence that the 70-percent figure does not automatically travel.

Established A cheaper confirmatory experiment already has its data. The 2024 decision-market study replicated the 12 highest-priced and 12 lowest-priced of 41 studies and found 83 percent against 33 percent, so the sorting works on held-out studies. Repeating that selection design in a non-experimental field — picking the top and bottom of a market’s ranking and replicating both — costs only the replications and would show directly whether the price ordering carries reliability information outside the laboratory.

Frontier A head-to-head that has never been run cleanly would settle the method question. Hold the forecaster pool and the claim set fixed, and vary only the elicitation — individual survey, deliberating group under IDEA, and market of varying depth — then score all three against the same replications. Because the observed differences between methods are smaller than the noise from differing samples, only a within-sample comparison can say whether markets genuinely beat surveys or merely tend to be run on easier sets. Frontier No project has yet held the forecaster pool and the claim set fixed while varying only the elicitation.

Speculative An adversarial test nobody has run would price the instrument’s robustness. Once a forecast is wired to a funding or credibility decision it becomes a target; offering a modest reward for moving a claim’s market price without new evidence would measure how cheaply the signal can be manipulated, and therefore how much weight it can bear in an allocation rule.

Established The reproduction-versus-replication split is a measurement the data already support. SCORE found 74 percent of 143 papers reproduced at least approximately and 54 percent precisely, while only 49 percent replicated on new data, and reproduction rose to 91 percent when data and code were shared and fell to 11 percent precise when data had to be reconstructed. Forecasting each of these three targets separately, and scoring them separately, is a clean experiment that would show which dimension of credibility the crowd is actually reading.

Frontier What is missing is not a technique but a custodian. Every experiment above is cheap relative to the replications it rides on, and none requires new science; what none of them has is a standing, independent body that holds dated forecasts and scores them on a fixed criterion across fields, year after year. Section 10 argues that this custodian, not any algorithm, is the load-bearing missing piece. Speculative A single funded decade of such scoring would settle most of this brief’s open questions.

Established A working forecasting system is a pipeline with five stages, and the human ones are the hard ones. Extract one testable claim per paper; recruit and qualify a forecaster pool; run the elicitation (survey, IDEA group, or market with a scoring rule and a liquidity parameter); aggregate to a probability; and, much later, resolve against a replication scored on a pre-agreed criterion. The elicitation software is the easy stage; claim extraction and resolution are where the cost and the judgement sit. Frontier No off-the-shelf system performs claim extraction and blind resolution end to end.

Established The market design is a solved engineering problem borrowed wholesale. The SCORE community markets used a logarithmic market scoring rule with base 2 and a liquidity parameter b = 100, standard machinery from the prediction-market literature that Collective Intelligence covers. The engineering novelty here is not the market but the resolution: building the durable record that ties a dated price to a replication run years later.

Frontier Versioning and pre-registration of the forecast are the requirements most systems skip. To be scoreable, a forecast must be stamped with a date, a claim identifier and a resolution criterion before the replication is known, and stored immutably. The Predicting Replicability Challenge is scoreable precisely because it does this; an internal credibility dashboard that updates its scores after outcomes arrive is not evidence of anything. Established A forecast that is not date-stamped before the outcome cannot be scored at all.

Established Automated claim extraction is the stage that limits scale. SCORE extracted claims from 3,900 papers largely by hand, which is why the forecasting corpus is thousands of claims rather than millions. This is the same identifier-and-extraction infrastructure that Research Front Detection depends on, and it is unglamorous, uneven and unbuilt at corpus scale.

Frontier The resolution step is where a system silently breaks. A replication can differ from the original in sample, setting or analysis, and someone must decide whether it counts — a judgement that, if made by a party with a stake in the forecast, quietly corrupts the score. The engineering answer is separation of duties: the forecaster, the replicator and the grader must be different people, which is an institutional constraint expressed in software. Frontier Most operational credibility tools do not enforce that separation, which is why their scores are hard to trust.

Established The nearest neighbour on the map is the detection of emerging fields, and the boundary is sharp. Research Front Detection forecasts which topics will grow; this brief forecasts whether a specific published claim is true. Both are validated against outcomes, both fight a base-rate problem, and both need the same claim-and-identifier infrastructure, but they answer different questions and should not be conflated.

Established It sits inside the science of science and beside forecasting. Science of Science owns the broad quantitative study of research; Technology Forecasting owns dated predictions of capability and the scored record of how those fared. This brief is the case where the science of science can grade its own predictions cleanly, which is rare enough to be worth isolating. Established Grading a claim’s truth is cleaner than grading a field’s growth because the ground truth is unambiguous.

Established It draws its core method from collective intelligence. Collective Intelligence owns the theory and evidence on prediction markets, survey aggregation and the wisdom of crowds; the contribution here is applying that machinery to a target with clean ground truth and finding the edge over the base rate is modest.

Frontier It is adjacent to the pathologies it is meant to catch. Pathological Science and Research Productivity Decline describe the failure modes — results that do not hold, literatures that stall — that a working replication forecast would help triage. Whether triage changes the trajectory of a field, rather than merely labelling it, is untested. Speculative Triage might redirect a field, or it might merely relabel work already destined to fail.

Established Outside the map it borrows from three disciplines. Weather forecasting supplies the calibration and proper-scoring discipline; signal-detection theory supplies the base-rate and precision-recall framing; and experimental economics supplies the market designs and much of the personnel — the same names recur across the founding papers.

Established The programme that produced the forward evidence was a defence programme, and that shaped the instrument. DARPA’s SCORE programme — Systematizing Confidence in Open Research and Evidence, now complete — sought automated confidence scores so a defence consumer of social and behavioural research could judge how far a claim was likely to replicate. A tool built for a consumer who needs to screen a literature optimises differently from one built to allocate research funding, and the same score does not serve both without saying which error it minimises. Frontier A screening tool can tolerate false positives; an allocation tool cannot.

Established The load-bearing missing institution is an independent custodian of dated forecasts. The Center for Open Science plays that role for the Predicting Replicability Challenge, holding forecasts and scoring them on a fixed criterion after the fact. Nothing equivalent exists as a standing service across fields; without one, most credibility scores are produced and graded by the same party, which is exactly the arrangement Research Front Detection identifies as the reason miss rates go unpublished.

Frontier The suppliers of credibility scores are interested parties and should be labelled. A method’s developers reporting 84 percent accuracy and an area under the curve of 0.94 for their own process (the developers on their own instrument) is not evidence of bad faith, but it is the reason the SCORE design — independent replication teams scoring everyone’s forecasts blind — is the standard the field should hold to rather than self-reported validation. Established Independent, blind scoring is the only report the field should treat as decisive.

Established No incumbent has an incentive to publish its own miss rate. A vendor selling a credibility score loses a selling point by revealing its false-positive rate; a journal that adopted forecasting would invite an audit of its back catalogue; a funder would expose its portfolio. This is the same governance problem the detection brief names, and it is why the forward scoring had to be paid for by a research programme rather than emerging from the market. Speculative A public custodian would have to be endowed rather than sold, because no customer pays to learn a tool is wrong.

Frontier This Institute’s own longlist is an instance of the object under study. Assigning any claim a flag — established, frontier, speculative, handwave — is a credibility forecast, and treating it as one imposes the same obligations SCORE accepted: date the judgement, fix what would count as being wrong, and keep the calls that failed. The cost is embarrassment; the benefit is that the flags become evidence about forecasting rather than another undated inventory of opinion.

Frontier A credibility score attached to a person’s paper is a reputational instrument with an unquantified error rate. A forecast that a specific finding will fail to replicate, published before any replication, lands on the authors, who had no part in the scoring and no route to appeal it, and at roughly 70 percent accuracy the instrument is wrong about three claims in ten. The asymmetry matters most for early-career researchers whose single flagged result carries more weight than a senior figure’s portfolio. Frontier The instrument pushes its errors onto the least powerful party in the system.

Established The signal is a probability and will be read as a verdict. The founding papers are careful that a market price is an estimate; the users of a credibility dashboard will treat a low score as a finding of falsity. That gap between what the instrument says and what it will be taken to say is the central ethical hazard, and it is worse where the base rate is near one half and the score adds only a modest edge. Established The gap between a probability and a verdict is where the reputational damage happens.

Frontier Forecasting can become self-fulfilling. A claim publicly forecast to fail attracts fewer extensions, less funding and more sceptical replication, all of which make failure more likely; a claim forecast to hold attracts the opposite. Because the forecasters are drawn from a small, correlated community, their collective prior can shape the outcome it purports to predict, and no study has measured that feedback.

Speculative Cheap forecasting could displace expensive replication. If a 70-percent signal is available for the cost of a survey, the temptation is to forecast instead of replicate, which would starve the enterprise of the ground truth the forecasts depend on. The forecast is a complement to replication, not a substitute, and an institution that forgot that would erode its own instrument. Speculative The cheaper the forecast, the stronger the pull to stop replicating altogether.

Frontier The strongest civilisational case is a standing, cheap early-warning layer over the whole research literature. If aggregated forecasting holds at 70 percent across fields, a society could triage which of its published claims are safe to build policy and further research on, aiming its limited replication budget at the claims where forecast and stakes diverge. That is a change in how a civilisation manages the reliability of its own knowledge base, and it is conditional on the generalisation section 7 is testing. Frontier A reliability layer over the literature is only as trustworthy as its worst-calibrated discipline.

Established The counter-case is that the edge over the base rate is small and may not scale. A signal that is barely better than knowing half of everything fails is a weak lever on a literature of over ten million papers a year, and SCORE’s conclusion that credibility is multidimensional with no broadly applicable shortcut is a direct warning against expecting a single number to do civilisational work. Established Credibility is multidimensional, and no one score captures reproduction, robustness and replication at once.

Frontier The capability is concentrated where the replications and the forecasters are. The scored projects sit in a handful of wealthy countries and two disciplines; a research system elsewhere inherits credibility scores calibrated on a literature not its own, under loss functions set elsewhere — the same dependency the detection brief describes for emerging-technology lists.

Speculative If machine-generated research floods the literature, forecasting reliability becomes the binding constraint rather than a convenience. A literature no human can read is one where whatever credibility filter is in use effectively sets the research agenda, and the unmeasured error rate of that filter stops being a footnote. On present evidence the human forecasters outperform the machines, which is an uncomfortable position for a future where humans cannot keep up with the volume. Speculative If machines write most papers, a filter humans cannot audit becomes the research agenda by default.

These horizons track the evidence base for forecasting replication, not the pace of the sciences being forecast.

• 10 yr: Frontier Prospective scoring becomes routine: the Predicting Replicability Challenge and successors publish scored records across several new fields each year, settling whether the 70-percent figure travels beyond psychology and economics; structured elicitation and markets remain within a few points of each other; machine models are held to the same blind forward test that Research Front Detection demands and are judged on it.
• 25 yr: Speculative Either a standing independent custodian scores dated forecasts across fields as a public utility and reliability triage is a normal step before major funding, or the practice stays inside defence and vendor screening where miss rates are not published by design. These futures are distinguishable now by whether any funder endows the custodian.
• 50 yr: Speculative If a large share of research is machine-generated, forecasting shifts from grading a human literature to allocating scarce human replication effort across a machine one, and the instruments in this brief are recalibrated or replaced rather than improved.
• 100 / 250+ yr: Handwave Any claim at this range presupposes that science remains organised around discrete published claims that can be replicated one at a time. That format is a few centuries old and is exactly what the automation of research might dissolve.

• Depends on nothing on this map in the strict sense — the elicitation methods are published and the mathematics is settled — but it consumes the crowd-aggregation evidence owned by Collective Intelligence, the claim-and-identifier infrastructure described in Research Front Detection, and the scored-forecast discipline developed in Technology Forecasting.
• Requires (not on this map) completed replications in fields beyond psychology and economics, because the 70-percent figure is measured almost entirely on two disciplines and one round on a third already regressed; a measured base rate of replicability for each discipline, without which an accuracy figure cannot be separated from the ease of the majority class; an independent custodian that holds dated forecasts and scores them on a fixed criterion before the outcome is known; a buyer willing to pay for a calibrated miss rate rather than for a reassuring credibility badge; and automated claim extraction from full text at corpus scale, since forecasting is stuck at thousands of hand-extracted claims rather than millions.
• Enables reliability triage of published claims before replication; defensible aiming of scarce replication budgets; credibility weighting in meta-analysis and evidence synthesis; and the maintenance of any curated research map, including this one, as something scoreable rather than merely asserted.
• Adjacent to Science of Science for the broader quantitative programme, Scientific Funding Models for what happens when a forecast is wired to allocation, Pathological Science for the failure modes it triages, and Philosophy of Science for what replication does and does not establish.

Handwave “Prediction markets can tell you whether a study is true.” They return a probability, not a verdict, and at about 70 percent accuracy against a base rate near one half they are wrong about three claims in ten. The correct reading of a low market price is that the claim is a good candidate for replication, not that it is false.

Frontier “Markets clearly beat surveys.” Pooled across 103 findings, markets scored 73 percent and surveys 66 percent, but that difference was not statistically significant (chi-squared 1.12, p = 0.29). Markets did carry a significantly lower absolute error, so they are better calibrated; but the tidy claim that markets dominate is not supported, and structured expert elicitation matched or exceeded both.

Established “Seventy percent accuracy is impressive.” It is modest once the base rate is included. When roughly half of tested claims fail, a rule that predicts failure every time scores in the low sixties, and one reanalysis of the founding study found the market’s edge over that null rule was not statistically significant. Accuracy must be read as lift over the base rate, which is why the field now prefers Brier scores and area-under-the-curve.

Frontier “Machine-learning models have automated this.” Early text models reported accuracies comparable to markets, but when SCORE scored three deployed systems blind and prospectively, none was consistently effective, and the programme cautioned against the earlier optimism. The language of a paper does carry some signal, but the machine approach is frontier, not settled, and its small labelled datasets invite provenance leakage. Speculative A model that learns to predict from journal or author identity is reading provenance, not content.

Established “A forecast can replace a replication.” The forecast is scored against the replication and has no meaning without it; treating the cheap signal as a substitute would starve the enterprise of the ground truth it depends on. Forecasting is triage, and triage that eliminates the diagnostic it is triaging for defeats itself.

Frontier “The 70-percent figure generalises across science.” It is measured almost entirely on experimental psychology and economics. On the one genuinely new domain scored prospectively — a health-behaviour set in the Challenge’s third round — only 8 of 15 teams beat the coin-flip baseline. Generalisation is the open question, not a settled property. Established Every accuracy figure in this brief carries the discipline it was measured on.

Established “Reproducing, reanalysing and replicating are the same thing.” SCORE separated them and got different numbers: 74 percent of papers reproduced at least approximately on the same data, 34 to 57 percent survived independent reanalysis within tolerance, and 49 percent replicated on new data. A forecast is only meaningful against a stated one of these three, and conflating them makes any accuracy figure uninterpretable.

Speculative “A credibility score is a neutral measurement.” It is a forecast made by a correlated community and, once published, it can change the outcome it predicts by steering funding, extensions and scrutiny. Neutral measurement does not feed back on its object; a credibility score does, and that feedback is unmeasured. Frontier A neutral measurement does not move its object; a published credibility score can.