Retrofit Performance Gap

Established The prize here is a number rather than a device: the ratio of realised to modelled energy savings for the retrofit of an existing building, measured on the same house by both methods. Two programmes have produced it at scale and they disagree by nearly a factor of two. In Michigan, a randomised encouragement design covering 34,110 households found model-projected savings running at between 2.5 and more than three times realised savings. In Medicine Hat, Alberta, thirteen years of metered gas and electricity for roughly 20,000 single-family houses put realised savings at about 60 per cent of the EnerGuide audit model’s prediction. Between a third and three-fifths of the business case for retrofitting the existing stock is arriving at the meter, and which end of that range is right is the open question of section 4.

Established The money at stake per house is measurable and it is the same order as the subsidy. The Michigan trial valued the improvements delivered at about $5,150 per household against a central estimate of realised savings, in present value, of roughly $2,400. The Department of Energy’s own retrospective evaluation of the same programme, which spent $4,695 per dwelling unit in programme year 2008, reports a savings-to-investment ratio of 1.47 on energy alone. Both cannot be true, and which is true decides whether a national retrofit programme is an investment or a transfer.

Frontier The gain from resolving it is a reallocation rule, and one published decomposition already says where the money should go. Across more than 9,800 Illinois homes, engineering-model bias accounts for up to 41 per cent of the wedge between projected and realised returns, workmanship for about 43 per cent, and rebound for 6 per cent. If that split survives replication, the largest correction available is an inspection regime rather than a better simulation engine. The cost of knowing is itself known: one Californian pay-for-performance programme budgeted $900,000 of a $6.0 million two-year programme, about 15 per cent, for measurement and verification.

Frontier Who collects it depends entirely on the contract. Under a deemed-savings rebate the subsidy is paid on the model output, so a 40 per cent realisation rate is absorbed invisibly by the treasury and the household; under a metered-savings contract the same shortfall lands on the contractor. Identical physics, different payer.

Speculative The condition on all of it: the sharpest number in the field comes from one trial, in one cold state, on low-income households, at about 6 per cent take-up. Nothing in that design forbids the result generalising and nothing establishes that it does. Section 5 is meter access, randomisation and inspection rather than physics, and section 7 names the trial that would settle it.

The retrofit performance gap is the difference between the energy a building model says an upgrade will save and the energy the meter says it saved. It is a narrow question with an enormous denominator: the existing stock is the largest physical asset most countries own, and essentially every national decarbonisation plan assumes a large fraction of it is upgraded rather than replaced.

Established Three different quantities travel under the name “performance gap” and conflating them has produced most of the confusion in the policy literature. The design-versus-as-built gap in new construction: a building certified to a standard that consumes more than the standard implies. The savings gap in retrofit, which is this brief’s subject: modelled against metered savings on an existing dwelling. And the prebound effect, the finding that poorly rated dwellings consume substantially less than their rating predicts before anyone touches them — a gap in the baseline rather than the upgrade, which mechanically shrinks the savings any upgrade can deliver.

Established The causal chain has five links and each can break independently. An auditor characterises the house; a model converts that characterisation into projected annual savings; a subsidy or loan is sized against the projection; a contractor installs measures; an occupant lives in the result. The published evidence can distinguish some of these breaks and cannot distinguish others, and saying which is which is most of the work of section 3.

Frontier This brief separates three named causes and adds a fourth the field usually forgets. Engineering-model error: the model is wrong about the house, the weather, or the occupant’s baseline. Rebound: the occupant converts part of the efficiency gain into comfort rather than a lower bill. Installation quality: the measure specified is not the measure installed. The fourth is evaluation-method error — the counterfactual savings are measured against, which moves the headline number by more than the first three combined when the comparison group is chosen badly.

Established The anchor result is a randomised trial, which is rare here and is why it dominates the brief. Almost everything else in the retrofit-evaluation literature is a billing analysis with a constructed comparison group. One programme was subjected to a genuine randomisation and returned a number far worse than the observational studies of the same programme. That disagreement, rather than the gap itself, is the most interesting thing in the subject.

Established The anchor is Fowlie, Greenstone and Wolfram’s randomised encouragement design on the United States Weatherization Assistance Program, and the registered trial record makes its scale exact. Registry entry AEARCTR-0000416 gives 34,110 Michigan households randomised individually by computer, 8,645 encouraged and 25,465 control, with the encouragement — house visits, telephone calls and in-person follow-up appointments — delivered between January 2011 and March 2012 inside a study window running from January 2010 to November 2014.

Established The headline finding is that the engineering model was wrong by a large multiple and the programme’s upfront cost exceeded the energy it bought. The July 2015 working paper states that “the upfront investment costs are about twice the actual energy savings” and “the model-projected savings are roughly 2.5 times the actual savings”, at an average rate of return of approximately −9.5 per cent annually. Recipients “received approximately $5,150 worth of home improvements on average, at zero out-of-pocket costs”, against a “central estimate of the realized average savings per household” of “roughly $2,400” in present value at a 3 per cent discount rate.

Frontier The widely quoted figure of 39 per cent is close but the two deposited versions of this paper do not agree with each other. The 2015 working paper’s 2.5 multiple implies a realisation rate near 40 per cent, which the trade press rendered as 39 per cent. The author-deposited version dated 22 January 2018, matching the Quarterly Journal of Economics article at 133(3), 1597–1644, instead states that “the model-projected savings are more than three times the actual savings” and gives the return as approximately −7.8 per cent annually. A realisation rate under a third is a materially different claim from 40 per cent. The publisher’s page served metadata only on retrieval, so this brief cannot say which figure the journal of record carries; it reports both and treats the realisation rate as lying between about 30 and 40 per cent.

Established The trial tested rebound directly and did not find it. Indoor temperature was measured rather than assumed, and the paper “fails to find evidence of significantly higher indoor temperatures at weatherized homes”; the point estimate is an increase of 0.65 °F with a confidence interval wide enough to include zero. This is the single most load-bearing negative result in the subject, because rebound is the explanation the field reaches for first and this design could see it.

Established The carbon arithmetic follows directly and is brutal. Valuing emissions avoided by realised rather than projected savings, the paper reports that “the most plausible estimates are $329/ton, which is about an order of magnitude larger than the U.S. government’s estimate of the social cost of carbon of roughly $38”, at a 3 per cent discount rate and a sixteen-year measure life. Evaluated this way, a retrofit programme is an expensive abatement instrument.

Frontier The design’s limitation is take-up, and it is not a small one. Participation was under 1 per cent in the control group and about 6 per cent among the encouraged. The causal estimate is a local average treatment effect on households who took up weatherization only because someone knocked repeatedly — a group that by construction was not seeking the programme out. Whether their houses, baselines or behaviour differ systematically from volunteers’ is not settled by this trial, and it is the most defensible line of attack on generalising the result.

Established The same programme, evaluated observationally by its own sponsor, returns a number with the opposite sign. The Department of Energy’s retrospective national evaluation of programme year 2008, led by Oak Ridge National Laboratory, drew on about 30,000 dwelling units and more than 8,000 utility-bill analyses across over 1,000 utilities (programme’s own figures). It reports average annual energy cost reductions of $283 for single-family homes, 12 per cent overall with gas down 18 per cent and electricity down 7 per cent, $4,695 spent per unit, and a single-family savings-to-investment ratio of 1.47 on energy alone — rising to $13,550 of total benefits per household and a programme-wide benefit-cost ratio of 4.1 once health and safety are added.

Frontier A savings-to-investment ratio of 1.47 and a randomised finding that costs are twice savings are not reconcilable, and the difference is the counterfactual. The national evaluation’s comparison group was drawn from households weatherized in an adjacent period rather than randomised, and savings were estimated by billing analysis. When an observational and a randomised design on the same programme in overlapping years disagree by a factor of three, the prior that the randomised estimate is closer to the causal truth is strong but not absolute — the trial estimates a different population, and both statements can be locally true.

Established The result replicates outside the United States, at a less extreme magnitude, in a cold Canadian climate. Papineau, Rivers and Yassin used thirteen years of monthly gas and electricity billing data, 2007 to 2019, for 1,475 participating and 18,460 non-participating single-family houses in Medicine Hat, Alberta, matched to EnerGuide for Houses audit records carrying the model’s own predicted savings for the same dwellings. Retrofits under the federal ecoENERGY programme of 2008–2011 cut natural gas consumption in the average participating home by about 21 per cent, which the authors put at about 60 per cent of predicted gas savings; the total-energy realisation rate is 55 to 56 per cent.

Established The Canadian data resolve the gap by measure, and the variation is larger than the average. Attic insulation realised 77 per cent of predicted savings. Air sealing realised 145 per cent — the model understated it — on a modest absolute base. Basement insulation realised 13 per cent, windows and doors 33 per cent, central air conditioners no statistically significant saving at all. Whole-envelope retrofits, predicted to cut gas consumption by about 67 to 70 per cent, delivered roughly half of that. A single programme-level realisation rate is an average over measures whose individual rates span an order of magnitude, so a programme that changed its measure mix would change its realisation rate without changing anything about models, contractors or occupants.

Established The Canadian savings persist, which removes one popular explanation. The event study finds them persisting for at least a decade. Decay — insulation settling, equipment drifting out of tune, occupants reverting — is a real mechanism and it is not what produces this gap.

Frontier A third large evaluation finds the gap changing sign between fuels, which is hard to explain by behaviour and easy to explain by model error. Allcott and Greenstone studied the Wisconsin programmes of the federal Better Buildings Neighborhood Program across 101,881 owner-occupied single-family homes, 1,394 of which completed audits. Households achieved 68 per cent of predicted savings in aggregate — but the gas realisation rate was 35 per cent and the electricity rate 317 per cent. An occupant taking back comfort does not produce a threefold electricity surprise in the same houses where gas savings fall two-thirds short. A model that allocates end uses wrongly does.

Established The one study that decomposes the gap into causes is Christensen, Francisco, Myers and Souza, and it is the most important result here after the trial itself. Using the Illinois Home Weatherization Assistance Program — more than 9,800 homes served by 34 local agencies between programme years 2009 and 2016 — and a machine-learning event-study design, they find that “bias in engineering models can account for up to 41% of the wedge, primarily from overestimated savings in wall insulation”, that “heterogeneity in workmanship can also account for a large fraction (43%) of the wedge”, and that “the rebound effect can explain only 6%”. It appeared in the Review of Economics and Statistics, 105(4), 798–817, in 2023.

Frontier Two independent designs now agree that rebound is a minor term, which contradicts the field’s standing intuition. The Michigan trial measured indoor temperature and found no significant increase; the Illinois decomposition assigns rebound 6 per cent of the wedge. Against that, the building-physics literature has for two decades put direct rebound and comfort-taking for space heating “within the range of 10–35%”. These are not straightforwardly the same quantity — a 10 to 35 per cent take-back of the technical saving is compatible with rebound explaining a small share of a 60 per cent total shortfall — and the brief treats the reconciliation as open.

Established The prebound effect is the part of the gap that exists before anybody installs anything, and it is large in the European data. Sunikka-Blank and Galvin’s analysis of about 3,400 German dwellings found measured heating consumption averaging around 30 per cent below the energy performance rating’s prediction, the shortfall rising with how bad the rating was: roughly 17 per cent at 150 kWh/m²/yr and about 60 per cent at 500. The same authors later put the German average at about 35 per cent below calculated consumption and concluded that retrofit payback estimates are over-optimistic “often by 50% or greater” in consequence.

Established Prebound is a mechanical explanation for model error and it requires nobody to behave irrationally. A standard audit model computes savings as the difference between two modelled states, both assuming a normative heating regime. A household that has been under-heating — because heat is expensive and the house is bad — never occupied the modelled baseline, so the saving available to them is smaller than the model’s difference by roughly the size of their prebound. This is the clearest reason to expect engineering models to overstate savings systematically in exactly the low-income stock public programmes target.

Established The audit model at the centre of the American evidence is a nameable piece of software with a documented validation history. The programme’s projections come from the National Energy Audit Tool, NEAT, maintained at Oak Ridge; its realisation rates against metered data were studied inside the programme as early as 1998. The performance gap is not a discovery of the 2010s economics literature. It is a known property of a tool that kept sizing subsidies.

Frontier Non-energy benefits are the programme side’s strongest answer and they are both real and unstable. The Department of Energy’s health and household benefits study, by Tonn, Rose, Hawkins and Conlon, surveyed 665 pre-weatherization and 803 comparison households in 2011 and re-surveyed 398 and 430 of them in 2013 at a 70 per cent response rate, monetising eleven benefits over ten years to $14,148 per unit (programme’s own figures). The largest line is cold-related thermal stress at $3,911, then asthma symptom reduction at $2,009 and prescription adherence at $1,929.

Established The instability is in the statistical-life term, and the programme’s own later review says so plainly. The Oak Ridge synthesis of non-energy impacts reports the thermal stress benefit as “$172 per household (excluding life value); $3,911 (including life value)” — a factor of 23 turning on a valuation convention rather than a measurement — and notes that only about a third of the 74 articles it examined were peer-reviewed and that “few define the methods used to quantify the benefits in a transparent way”. A benefit-cost ratio of 4.1 that depends on which side of that convention the analyst lands is not a refutation of the trial; it is a different question with a different answer.

Established The practitioner objection is about what was counted as cost, and it is fair without overturning the result. The Journal of Light Construction quoted Merrian Borgeson of the Natural Resources Defense Council arguing that the researchers counted health and safety work and full equipment costs rather than the efficiency increment — about a third of participants received a new furnace — and cautioning that the conclusions “are not widely applicable to energy efficiency, or even necessarily to other weatherization programs”. That is a genuine dispute about the denominator. It does not touch the numerator, which is the realisation rate.

Established Where the evidence stands: the gap is real, large, not primarily rebound, and programme-specific in size. Four independent large-sample estimates put realisation between roughly 30 and 68 per cent, on four programmes, in four places, using three identification strategies, and no credible study finds it near 100 per cent when savings are projected by an engineering model and measured at a meter. That is the established core; everything the policy literature does with it afterwards is contested.

Frontier The first open question is whether the gap widens or narrows with the depth of the retrofit. In Medicine Hat, whole-envelope retrofits predicted to cut gas by 67 to 70 per cent delivered about half of that, a worse relative performance than the programme average of 60 per cent. Prebound theory predicts exactly this: the deeper the modelled intervention in a poor dwelling, the more of the modelled saving consists of heating the occupant was never buying. If that is right, deep retrofit is systematically the worst case for realisation rather than the best, which inverts the standing policy preference.

Frontier The rebound magnitude is genuinely contested and the disagreement is partly about which denominator is in use. The Michigan trial found no significant temperature increase; the Illinois decomposition assigns rebound 6 per cent of the wedge; the building-physics literature puts direct rebound and comfort-taking for space heating at 10 to 35 per cent of the technical saving; the Wisconsin study applied an elasticity adjustment cutting predicted savings by roughly 20 per cent. These can be reconciled only by stating whether the quantity is a share of the technical saving, a share of the shortfall, or a temperature change — and most citations do not.

Speculative Rebound may be small in low-income cold-climate samples for a reason that does not generalise. A household rationing heat has large latent demand for comfort, so the naive expectation is a large take-back in exactly this population. Two studies found the opposite. One coherent reading is that the retrofits did not make comfort affordable even after the upgrade, so the household stayed at its budget constraint rather than its comfort optimum. If so, rebound would be larger in a better-off sample or after a deeper retrofit, and the reassuring null is an artefact of the programme studied.

Frontier The workmanship term is the least examined and the most actionable, and it rests on a single study. That workmanship heterogeneity accounts for about 43 per cent of the Illinois wedge says the same measure, specified identically, installed by different crews, produces savings varying enough to dominate the model’s own error. Nothing else reviewed here replicates it. If it holds, quality assurance has a return comparable to the measures themselves; if not, a large inspection programme is money spent on the wrong term.

Frontier Nobody has separated model error from prebound, and it is not obvious the data permit it. Model bias in the Illinois decomposition is the difference between prediction and realised saving after the workmanship and rebound terms are removed. A model internally correct about heat transfer but wrong about the household’s actual pre-retrofit indoor temperature lands entirely inside that term. Distinguishing a wrong physics module from a wrong occupancy assumption requires pre-retrofit indoor temperature data, which almost no programme collects.

Frontier The measure-level results are the strongest lead in the subject and nobody has chased them. Basement insulation at a 13 per cent realisation rate and windows and doors at 33 per cent are not marginal failures; they are measures whose modelled savings are largely fictional in that climate. Air sealing at 145 per cent is a measure the model undersells. A programme that dropped the first two and expanded the third would raise its realisation rate substantially with no change in model, contractor or occupant. Whether those rates transfer to other climates is unknown, and cheap to find out wherever audit records and meter data coexist.

Frontier Electrification changes the question and none of the anchor evidence covers it. Every large study cited here evaluates envelope measures and combustion heating in a gas-heated or mixed-fuel stock. A heat pump retrofit substitutes fuels as well as reducing demand, so its savings depend on a coefficient of performance varying with outdoor temperature, on the tariff, and on whether the emitters were sized for a lower flow temperature. The performance-gap literature has nothing measured to say about it at comparable scale, and the policy pipeline has already moved there.

Speculative A retrofit with a measured performance target and a commissioning step may be a different product from a measure-by-measure subsidy, and there is no large-sample test of it. The Welsh metered-savings work assembled only 42 UK properties from three research portfolios because the data did not exist at scale, which is a fair summary of how much evidence bears on that proposition.

Frontier The realisation rate under pay-for-performance is unknown, despite claims otherwise, because the measurement changes when the contract changes. Advocacy cites a 2016 Pacific Gas and Electric residential pilot improving realisation rates “from 27% to 100%” (coalition’s own figures), but under a metered contract the rate is 100 per cent by construction. Whether metered contracting causes more energy to be saved per dollar, rather than more accurately counted, requires a randomised comparison of contract forms that nobody has run.

Established The workback plan from “we do not know the realisation rate” to “we know it and know why” has seven links and none is a physics problem. The target state is a realisation rate with a confidence interval, decomposed into model error, rebound and workmanship, for a named measure package in a named climate zone against the specific model used to set the subsidy.

Established B1 — meter data at household resolution, linked to audit records. This is the gate. The Canadian result exists because Medicine Hat is a municipally owned utility whose billing data could be matched to EnerGuide audit files containing the model’s own predictions for the same houses. Without that join, the modelled and measured numbers describe different objects.

Frontier B2 — metering coverage, which is not universal even in rich countries. DNV reports that roughly 27 per cent of American residential customers lack advanced metering infrastructure (vendor), leaving paper billing as the fallback. Monthly data supports a portfolio estimate; it does not support the hourly analysis that would let an evaluator see a comfort take-back directly.

Established B3 — a counterfactual that survives scrutiny, which is where the observational literature breaks. The disagreement between a savings-to-investment ratio of 1.47 and a randomised finding that costs are twice savings is about comparison groups, not meters. Randomisation, an oversubscription lottery, or a defensible regression discontinuity in the eligibility rule are the three instruments available, and only the first has been used at scale on a retrofit programme.

Frontier B4 — indoor conditions, measured rather than modelled, before and after. Rebound and prebound are both statements about indoor temperature, and both are usually inferred from consumption rather than observed. The Michigan trial’s temperature measurement is why its rebound null is credible and why the same null is unavailable in every study that lacked one.

Frontier B5 — an installation-quality instrument independent of the crew being assessed. If workmanship carries about 43 per cent of the wedge, measuring it requires blower-door and infrared verification by an inspector with no stake in the payment, on a random sample rather than on jobs a crew nominates. This is the link whose cost is recurring rather than one-off.

Established B6 — statistical power, which fails at the individual house and holds at the portfolio. The Californian market-access programme targets a fractional savings uncertainty of plus or minus 25 per cent at 90 per cent confidence and states this applies to portfolio results rather than individual projects. A single house’s saving is not reliably measurable against its own weather-normalised baseline, which constrains every contract form that tries to pay per house.

Frontier B7 — method sensitivity, which is larger than practitioners admit. DNV reports that the same project, processed at different data granularities, produced metered savings ranging from 870.23 to 1,225.94 kWh (vendor) — a roughly 40 per cent spread from the choice of input resolution alone, on the quantity a pay-for-performance contract pays against. The Welsh assessment concluded compatibly that portfolio savings “can be evaluated today using CalTRACK or closely aligned methods” while for individual dwellings “further research and development is needed”.

Speculative The binding constraint is none of the above; it is that nobody is obliged to know. Every link here is buildable now, at a cost the Californian example prices at about 15 per cent of programme budget. The reason no second randomised trial has happened in the eight years since the first is institutional, and section 10 argues that is the only bottleneck that matters.

Established This brief waits on results nobody is producing, and every one is cheap by the standards of energy research: data joins, trial designs and measurement conventions, not discoveries.

Frontier First: a realisation rate computed against the specific model that set the subsidy, in more than one climate zone. The Canadian result has this property and the American ones partly do. Nothing comparable exists for the European calculation methods that size the continent’s renovation subsidies, or for any cooling-dominated climate. Until it does, the performance gap is a North American and German finding being applied globally.

Frontier Second: pre-retrofit indoor temperature at scale, the only thing that separates prebound from model error. Without it the two are algebraically indistinguishable, and the policy implications are opposite: prebound means the saving was never available and the household gains comfort instead; model error means the saving was available and the tool mis-stated it.

Frontier Third: a workmanship replication outside Illinois. The 43 per cent finding is one study, one state, one identification strategy — the most policy-relevant number in the brief and the least replicated.

Established Fourth: non-energy benefits measured from administrative health records rather than surveys. The New Zealand design using hospitalisation and pharmaceutical records is the existence proof; the American programme’s own reviewers rate survey-based valuation the least rigorous method in use. A retrofit programme’s cost-benefit case turns on the weakest evidence in its own file.

Frontier Fifth: heat pump performance measured in situ across a heating season. This brief cannot say what the realisation rate is for electrified heating, because the field measurement does not exist at the scale the envelope literature has. Industrial Heat Electrification carries the same structural problem one sector across — a modelled coefficient of performance standing in for a measured one.

Speculative Sixth, and outside this brief’s competence: a pass-through estimate for retrofit subsidy into property value. Future Housing Systems establishes that in constrained markets a construction-cost saving can be capitalised into land rather than reaching the household. The same question applies to a subsidy that improves a dwelling’s rating, and this brief does not know the answer.

Frontier The decisive experiment is a second large randomised trial of a retrofit programme in a different climate zone, with metered consumption before and after, evaluated against the specific engineering model used to set the subsidy, and with a randomly assigned inspection arm. One trial is an anecdote about Michigan. Two trials in different climates, against different models, on different housing stocks, would establish whether a realisation rate near a third is a property of retrofit programmes or of that programme.

Established The design is not exotic and its expensive component has a worked precedent. Randomised encouragement — offering help enrolling to a random subset of eligible households — withholds nothing from anyone, and the Michigan trial showed it works, moving take-up from under 1 per cent to about 6 per cent. Oversubscription lotteries are equally defensible wherever demand exceeds budget, which is the normal condition of these programmes.

Frontier The trial must carry four instruments the first one carried only partly. Metered consumption for at least twelve months either side, which the measured pathway of the United States rebate programmes already requires with final verification at least nine months after completion. Indoor temperature logging in a random subsample, which makes the rebound estimate causal rather than residual. The audit model’s own per-dwelling prediction, retained rather than discarded. And a random half of jobs assigned to independent post-installation inspection.

Established The inspection arm is what makes this a decomposition rather than another realisation rate. With inspection randomised, the difference in realised savings between inspected and uninspected jobs is a direct estimate of the workmanship term; with temperature logged, rebound is measured rather than inferred; and model error is then what remains rather than what everything else is attributed to. The Illinois study obtained that decomposition observationally and deserves a randomised confirmation.

Frontier The climate zone matters because the measure-level results say it should. Basement insulation realising 13 per cent of prediction in a cold dry Alberta climate says nothing about the same measure in a cold humid one, and nothing at all about cooling-dominated stock. The strongest version of the experiment runs one protocol in two climate zones simultaneously.

Speculative A cheaper partial substitute would answer the second-most-important question. Wherever a jurisdiction holds both audit-model predictions and meter data for the same dwellings — the ordinary condition of any programme using a home energy rating — measure-level realisation rates can be computed observationally for a few hundred thousand dollars. That does not settle the causal question, because the comparison group is not randomised, but it settles which measures are worth arguing about.

Frontier A natural experiment is already running and nobody has designed it as one. The United States Home Efficiency Rebates programme, created under section 50121 of the Inflation Reduction Act, pays through two pathways: a modelled one using software calibrated to historical usage, and a measured one using weather-normalised metered energy before and after, computed with Department of Energy-approved open-source software. Households and contractors are sorting between two payment rules for physically identical work. Randomising pathway assignment in even one state would answer the contract-form question directly.

Frontier A second natural experiment is running in California and its parameters are published. The market-access programme pays aggregators quarterly on portfolio savings computed with CalTRACK 2.0 methods through the OpenEEmeter, with a 365-day baseline, comparison-group adjustment, a 10 per cent quarterly holdback released at twelve months, and outlier flagging beyond plus or minus 50 per cent. It demonstrates that paying on measured savings is operable at a $6.0 million scale. It does not demonstrate that doing so saves more energy per dollar, because no such comparison was built into it.

Speculative The experiment that would settle the non-energy benefits dispute is a records linkage, not a survey. Link a randomised retrofit cohort to administrative health records for five years either side, as the New Zealand work did observationally. That converts the most contested term in the cost-benefit case from a valuation convention worth a factor of 23 into a measurement.

Frontier Nobody has scheduled or funded the decisive trial. Eight years after the Michigan result was published, no second randomised retrofit trial at comparable scale has been announced in any jurisdiction this brief could find. The horizon is therefore set by a funding decision rather than by any technical readiness date, and the instruments required are all commercially available now.

Established The engineering requirement is a measurement stack specified well enough to cost. Nothing in it is a research instrument: metering, a data pipeline, a statistical method with published acceptance criteria, an inspection workflow.

Established The methods layer exists as open source and its scope is explicitly bounded. CalTRACK specifies “a set of methods for calculating site-based, weather-normalized, metered energy savings from an existing conditions baseline” for single-family retrofits from utility meter data, built by a working group including Pacific Gas and Electric, Southern California Edison and Sempra, and tested on 4,777 properties retrofitted in 2014 and 2015. The repository states plainly that the methods do not attribute savings to specific measures — so the measured-savings stack cannot, by design, produce the measure-level realisation rates that section 4 calls the most valuable output available.

Frontier The acceptance criteria are the engineering specification and they are portfolio criteria. A baseline model fit below 1.0 coefficient of variation of root mean square error; fractional savings uncertainty within plus or minus 25 per cent at 90 per cent confidence; non-routine event flagging when post-retrofit fit degrades; outlier review beyond plus or minus 50 per cent annual savings. These are the numbers that decide whether a measured-savings contract can be written, and they are why such contracts are written over portfolios rather than houses.

Established The measurement stack costs about a seventh of programme spend at small scale. The Californian programme’s two-year budget allocates $900,000 to measurement and verification out of $6.0 million, against $4.5 million of incentives. Whether that ratio falls with scale is the obvious question, and the answer is not in the plan.

Frontier The data-access layer is what fails in practice, and it is institutional wearing an engineering costume. The United States rebate guidance offers states three routes to consumption data — the Green Button standard or the Environmental Protection Agency’s Portfolio Manager, direct collection from utilities, or collection from residents — and vendors report some states negotiating with more than twenty separate providers. Smart Cities and Manufacturing Digital Twins reach the same wall from different directions: the sensor exists, the model exists, the right to join the two records does not.

Speculative The audit tool is the component most obviously due for replacement and the least likely to be replaced. A model calibrated on each dwelling’s own metered pre-retrofit consumption rather than on normative occupancy would absorb the prebound effect automatically, which is precisely what the Canadian authors recommend. The obstacle is that a calibrated model projects smaller savings, which shrinks the subsidy each household qualifies for under every existing programme rule.

Frontier Inspection is a workforce problem rather than an instrument problem. Blower doors, duct blasters and thermal cameras are commodity equipment. An independent inspectorate large enough to sample a national programme randomly, with no commercial relationship to the crews it assesses, is a standing institution with a payroll, and Standards as Governance gives the general form: the conformity assessment layer, not the standard, is where these systems fail.

Established The nearest neighbour is Future Housing Systems, and the relationship is complementary rather than overlapping. That brief owns delivery for new dwellings and the five-term decomposition of delivered price, establishing that a construction-cost saving in a supply-constrained market can be capitalised into land rather than reaching the household. This brief owns the existing stock and a different failure: not that the saving is captured, but that it was never as large as the model said. Both end where the household is the last party to find out.

Frontier The structurally closest brief is Carbon Removal Verification, and the shared problem is the counterfactual. A tonne of avoided emissions from a retrofit and a tonne removed by a soil or forestry project are both differences against a baseline nobody observed. Both fields found the baseline is where the money leaks, both converged on measured rather than modelled crediting, and both find measurement too noisy at the individual project and workable over a portfolio. Retrofit is a decade ahead in having a randomised trial and a decade behind in having a market that prices the uncertainty.

Frontier Insurance as Regulator describes the mechanism that would price this gap without anyone legislating it. A savings guarantee, a retrofit warranty or an energy-performance contract is an insurance product whose loss ratio depends on the realisation rate. An underwriter holding reserves against modelled savings would demand the measurement this brief asks for, because it is their money.

Established Three further briefs carry adjacent pieces. Design Codes Under Nonstationarity owns the weather file every one of these models normalises against, the input quietly drifting underneath all of them. Automated Construction Systems carries the industrialisation case whose retrofit analogue is the only plausible route to whole-house upgrades at controlled quality. Climate Health Adaptation holds the non-energy benefits question in its own right.

Frontier The institutional requirement dominating all others is that whoever can run the trial must not be the party evaluated. Retrofit programmes are evaluated by their administrators, by contractors those administrators select, under guidelines the funding agency writes. That arrangement produced a national evaluation reporting a savings-to-investment ratio of 1.47 for a programme an independent randomised trial found returning less than half its cost. No bad faith need be alleged to notice the incentive gradient runs one way.

Established The deemed-savings convention is the institutional heart of the problem and it is load-bearing for the whole subsidy architecture. A deemed saving is a number in a table that a measure is assumed to deliver, used to size rebates and credit utilities against efficiency obligations. It is administratively cheap, auditable and contestable in a rate case — real virtues. It is also the mechanism by which a model error becomes a payment, and it survives because replacing it with measurement would reopen every settled number in a regulatory proceeding.

Frontier Measured-savings regulation now exists in at least two jurisdictions and is young enough that its failure modes are unknown. California’s population-level normalised metered energy consumption framework supports programmes where “all aggregator payments are tied to the delivery of savings”, with a single implementation partner and more than ten aggregators competing inside it; the United States rebate programme carries a measured pathway alongside a modelled one. Both shift realisation risk onto the party doing the work, which is the right direction and creates a new selection problem: a contractor paid on measured savings has every incentive to seek the worst houses and avoid the marginal ones.

Speculative The clean institutional fix is that any programme sizing a subsidy from a model must publish the realisation rate of that model, on its own portfolio, annually. It costs a data join, it requires no new science, and it would make the model’s error a public number attached to the tool rather than a finding an academic occasionally produces eight years late. No jurisdiction this brief examined does it.

Frontier The evaluation community’s instruments are sound; the problem is where they sit in the decision. Fractional savings uncertainty thresholds, model fit criteria and non-routine event flags are competent statistical practice applied after a programme is designed, to report on it, rather than before, to choose it. An evaluator whose finding arrives after the money is spent is a historian.

Established The most honest institutional signal in the record is that the programme knew about the gap decades before the economists did. Oak Ridge published realisation rates for the National Energy Audit and a validation of NEAT against a gas utility’s metered low-income programme in 1998. The knowledge sat in the institution’s own publication list and the subsidy formula did not change. Institutions do not fail to measure because measurement is hard; they fail to act on measurements that would shrink their programme.

Frontier The coalition arguing for measured savings is an interested party and its argument is still mostly right. More than sixty demand-flexibility and efficiency firms argue for aligning incentives “around delivered measured results” (coalition’s own figures), and they sell measured-savings services. The structural argument — paying on an output moves the risk to the party who can act on it — does not depend on who makes it. The specific realisation-rate claims do.

Established The central ethical asymmetry is that the sharpest negative evidence in this field was generated on low-income households and is cited in arguments about programmes that serve them. The Michigan trial, the Illinois decomposition and the American national evaluation all study low-income weatherization. A result saying the energy savings do not justify the cost is one step from a conclusion that the programme should shrink, and the population that would bear that has no part in the argument.

Frontier The prebound effect is a measurement of deprivation dressed as a modelling artefact, and reading it correctly changes the policy conclusion. A household consuming 30 to 60 per cent less heat than its dwelling’s rating assumes is not gaming a model; it is cold. Galvin and Sunikka-Blank say it directly: a high prebound effect with low income indicates fuel poverty, where occupants are not achieving adequate thermal comfort at all. The saving a retrofit fails to deliver to such a household is one they were already making, involuntarily, by suffering.

Established It follows that rebound in this population is a welfare gain and should not be counted as a loss. A retrofitted household that raises its indoor temperature instead of lowering its bill has been delivered health and comfort rather than kilowatt-hours. The Michigan trial found essentially no such take-back, which is the uncomfortable reading: those households got neither the saving nor the warmth in measurable quantity.

Frontier Non-energy benefits are where the ethics and the accounting collide, and the collision is about the value of a statistical life. The same thermal-stress benefit is worth $172 per household excluding life value and $3,911 including it. A cost-benefit case that turns on that choice is not a measurement of whether the programme works; it is a statement about how the analyst prices mortality risk among poor households in cold weather. Both numbers are defensible and the brief declines to pick one, because picking one is the ethical act being disguised as arithmetic.

Speculative The distributional question specific to this brief is who bears the realisation risk, and it is the party least able to price it. Under a deemed-savings rebate a household borrows against a projected bill reduction. If the projection is 2.5 to 3 times the realised saving, the repayment schedule was built on a number wrong in a predictable direction. Loan-based retrofit finance transfers that error onto the borrower; a measured-savings contract does not. Climate Adaptation Finance carries the general form.

Frontier Extending the analysis to the global south raises a harder version of the same question. Sunikka-Blank and Galvin’s 2025 work on informal settlements in India and Kenya finds under-consumption across cooking, ventilation, cooling, refrigeration and lighting, driven by lack of money and by cultural practice, with women bearing most of the health and discomfort burden and a prebound effect of 30.0 to 61.7 per cent in one rural settlement. There, an intervention modelled against a comfortable baseline shows a very low realisation rate and a very large welfare gain at once, and a programme evaluated on kilowatt-hours would be judged a failure for succeeding.

Frontier Every national net-zero pathway contains a retrofit term, and if the realisation rate is between 0.3 and 0.6 rather than 1.0, that term is wrong by the same factor. The consequence is not that the buildings sector fails; it is that the residual demand other parts of the system must supply is larger than planned by an amount nobody has recalculated. Generation, storage and network build-out are all sized against a forecast embedding modelled rather than realised efficiency.

Speculative The error is one-directional and repeats annually. A gap of the measured size, applied to a stock upgraded over thirty years, is a persistent shortfall against plan that is invisible in programme reporting, because programme reporting counts measures installed rather than energy saved. A system that measures inputs and forecasts outputs discovers the discrepancy at the point where the lights have to stay on.

Frontier The wider lesson is about engineering models substituting for measurement in climate policy, and buildings is simply where the counter-evidence exists. The same structure — a physical model producing a number, a subsidy sized to it, no meter anywhere in the loop — appears in agricultural soil carbon, industrial efficiency obligations and avoided-deforestation crediting. Buildings got a randomised trial. Most of the others never will, and the honest inference is not that they are fine.

Speculative If the workmanship finding generalises, the implication is about labour rather than technology. A 43 per cent share of the wedge attributable to who installed the measure makes the binding constraint a trained, inspected, well-paid installation workforce at a scale no country has. That is slower to build than a factory and does not benefit from learning curves the way manufactured products do.

These horizons track measurement and contracting practice rather than building technology, because the building technology is not the limiting term.

• 10 yr: Frontier Measure-level realisation rates get computed observationally in several more jurisdictions, because the audit-record and meter-data join is cheap wherever a home energy rating scheme already exists; the Canadian and Illinois results are replicated or contradicted; and at least one large measured-savings programme accumulates enough history to report what happens to realisation rates when contractors are paid on outcomes. Speculative A second randomised trial is possible in this window and requires only a funder; nothing technical prevents it starting next year.
• 25 yr: Speculative Deemed-savings tables are displaced for major measures in the jurisdictions with universal advanced metering, surviving where metering coverage is thin; audit models are routinely calibrated to each dwelling’s own pre-retrofit consumption, which lowers projected savings and forces a redesign of subsidy formulas built on the old numbers. Frontier The heat-pump realisation rate becomes the live question and the envelope literature becomes historical, because the stock being retrofitted has changed fuel.
• 50 yr: Speculative The question dissolves for new-ish stock as metering, submetering and in-use performance disclosure become ordinary, leaving a residual performance-gap problem concentrated in heritage and informal housing where neither modelling nor metering is straightforward. Handwave Whether building energy modelling as a discipline survives the transition to measurement-led practice, or is relegated to design-stage sizing, is a guess.
• 100 / 250+ yr: Handwave On this horizon the existing stock in question has largely been replaced or rebuilt, and the durable legacy of this subject is methodological rather than physical: whether the discipline of paying for measured rather than modelled outcomes propagated from buildings into the rest of climate policy, or stayed a curiosity of the efficiency sector. The claim that it propagates is an assertion, and this brief has no evidence for it.

• Depends on nothing on this map blocks it, and that is the finding rather than an omission: the decisive experiment needs meters, a randomisation, a temperature logger and an inspector, all of which existed in 2011. The nearest genuine dependency is inherited from Future Housing Systems, whose pass-through result — that a cost saving in a supply-constrained market can be capitalised rather than delivered — applies unchanged to a retrofit subsidy that raises a dwelling’s rating, and which this brief cannot resolve on its own evidence. Design Codes Under Nonstationarity supplies the weather normalisation that every measured-savings method assumes is stable.
• Requires (not on this map) a legal route for an evaluator to join household meter records to audit records, which is the gate on every realisation rate ever computed; a second randomised trial in a climate zone unlike Michigan, which is what turns one result into a finding; an audit model calibrated on the dwelling’s own metered consumption rather than a normative occupancy schedule, which is the only structural fix for the prebound term; a standing inspectorate with no commercial relationship to the crews it samples, without which workmanship stays a residual; a contracting form that pays installers on metered savings, which moves the realisation risk to the party who can act on it; and a finance product that underwrites realised rather than projected savings, so that the household is not the counterparty holding the model error.
• Enables an honest denominator for building-sector decarbonisation plans, which currently size residual demand from modelled savings; a measure-level procurement rule dropping the measures that realise a fraction of prediction; a savings-guarantee market underwritable because the loss distribution is known; and a template Carbon Removal Verification can borrow, its baseline problem being the same one with worse data.
• Adjacent Insurance as Regulator for the mechanism that prices this without legislation; Standards as Governance for the conformity-assessment layer where the inspection requirement actually lives; Manufacturing Digital Twins and Smart Cities for the same model-versus-instrument tension in other asset classes; Industrial Heat Electrification for the electrified version of the question this brief cannot answer.

Handwave “A randomised trial proved that energy efficiency does not work.” It proved that one programme’s realised savings were a third to two-fifths of its model’s projection and that costs exceeded those savings for the households an encouragement campaign induced to participate. Consumption in treated homes fell by 10 to 20 per cent. The measures worked; the projection did not. Generalising from one programme’s cost-benefit ratio to “efficiency does not pay” is the commonest misuse of this result, and its authors did not make it.

Established “The gap is the rebound effect.” Two designs that could see rebound did not find much of it. The Michigan trial measured indoor temperature and found a point estimate of 0.65 °F with a confidence interval including zero. The Illinois decomposition assigns rebound 6 per cent of the wedge. This is the most confidently refutable claim in the subject, and it remains the field’s first explanation.

Frontier “Therefore rebound is negligible everywhere.” Also wrong, in the other direction. The building-physics literature puts direct rebound and comfort-taking for space heating at 10 to 35 per cent of the technical saving, and the Wisconsin study’s elasticity adjustment cut predicted savings by about 20 per cent. The reconciliation is that a moderate share of a technical saving can be a small share of a large shortfall. Anyone citing either number without saying which denominator they mean is not making a claim that can be checked.

Established “Insulation does not save energy.” Attic insulation realised 77 per cent of predicted savings in the Canadian data and air sealing realised 145 per cent. What failed there was basement insulation at 13 per cent, windows and doors at 33 per cent, and central air conditioning at no detectable saving. The measure-level heterogeneity is the actual finding, and collapsing it to a single verdict on “insulation” discards the most useful thing in the dataset.

Frontier “Better models will close the gap.” On the one decomposition available, model bias is up to 41 per cent of the wedge and workmanship is about 43 per cent. A perfect model would leave more than half the gap in place. The corollary is the uncomfortable one for the modelling community: the largest available correction is a clipboard and a blower door, not a better simulation engine.

Frontier “Non-energy benefits make the programme cost-effective anyway.” They might, and the claim is not assessable as usually stated. A reported benefit-cost ratio of 4.1 rests on a $14,148 per-unit health valuation whose largest line moves by a factor of 23 with the statistical-life convention, supported by a literature the programme’s own reviewers call one-third peer-reviewed with methods rarely stated transparently. The energy case and the health case should be argued separately, and the health case currently rests on the weaker evidence.

Speculative “Deep retrofit avoids the problem.” The Canadian data say the opposite: whole-envelope retrofits predicted to cut gas use by about two-thirds delivered roughly half of that, a worse relative result than the programme average. Prebound theory explains why, and it is the finding most likely to be unwelcome, because deep retrofit is where the policy momentum is.

Frontier “Pay-for-performance programmes have realisation rates near 100 per cent.” Under a metered-savings contract the realisation rate is 100 per cent by definition, because the quantity paid for is the quantity measured. The cited improvement “from 27% to 100%” in a 2016 Californian pilot is partly a statement about accounting rather than physics (coalition’s own figures). Whether metered contracting saves more energy per subsidy dollar is the question worth asking, and the comparison has not been run.

Established “Measured savings can be verified house by house.” The operating frameworks say otherwise: uncertainty targets of plus or minus 25 per cent at 90 per cent confidence are set at portfolio level, the Welsh assessment concluded individual dwellings need further development, and one project processed at different data granularities produced savings between 870.23 and 1,225.94 kWh. Per-house measured contracting is not a solved problem.

Frontier “This is an American finding about an unusual programme.” Partly true, and the strongest remaining critique. The Michigan estimate is a local average treatment effect on households induced to participate by door-knocking, in one cold state, in a low-income programme, at about 6 per cent take-up. What survives is the pattern across four programmes in three jurisdictions with three identification strategies, none finding realisation above 68 per cent. The Michigan number may not generalise; the gap’s existence and rough magnitude does.

Handwave “Nobody knew about this until the economists showed up.” Oak Ridge published realisation rates for the National Energy Audit in 1998 and the tool kept sizing subsidies, which says more about how institutions use their own evaluations than about how hard the measurement is.