Replace L1 stability anchoring with exactly calibrated PMP market-response curves (supply + demand) - #48
Open
koen-vg wants to merge 31 commits into
Open
Replace L1 stability anchoring with exactly calibrated PMP market-response curves (supply + demand)#48koen-vg wants to merge 31 commits into
koen-vg wants to merge 31 commits into
Conversation
Adds a positive-mathematical-programming supply curve alongside the existing deviation penalty, off by default. Each production group -- (crop, country), grassland by country, (animal product, country) -- gets a convex marginal-cost curve through its observed activity whose slope reproduces a configured supply elasticity, so response to a shock is graduated instead of the dead band and jump an L1 kink produces, and the curvature comes from an exogenous elasticity rather than a fitted deviation target. The quadratic deviation cost is approximated by tranches priced at the slope times their midpoint deviation. Prices increase with distance from the baseline, so a cost-minimising solution fills the near tranches first and convexity needs no ordering constraints. The outermost tranche in each direction is unbounded, which extrapolates the curve linearly instead of imposing a bound that could conflict with the growth caps. Curves are deliberately per group, not per link: they price how much of a commodity a country produces, not where within the country it sits. The per-link deviation penalty keeps carrying that spatial inertia, so the two mechanisms are complementary rather than alternatives.
Drives a one-group LP through a real solve at a range of output prices and checks the arc elasticity comes out as configured, which is the property the slope formula exists to deliver. Also covers symmetry about the baseline, the elasticity_factor scan dial, monotone response, feasibility beyond the expansion range via the unbounded outermost tranche, and the guards for zero-baseline groups and non-positive marginal cost.
…akdown The curves add their deviation cost on auxiliary tranche variables, which PyPSA statistics cannot see, so without this the objective-breakdown identity check fails on any run with curves enabled -- the same trap the multi-cropping bounded subsidy hit. The evaluation recovers the term from the solved tranche variables and the prices recorded when the curves were built, and reports it as its own category rather than folding it into production stability: the two anchor production by different mechanisms and a run may carry either, both, or neither. For the same reason it sits outside the deviation-penalty gate, which a curves-only run disables.
Activity moves in whole tranches, so with equal widths the curve cannot express a move smaller than expansion_range / n_blocks of baseline. At the previous default that was 12.5%, against typical group moves of a few percent: measured on a full-resolution solve, 97% of moving groups in the curves-plus-penalty regime had their entire move inside the first tranche, so the curve was acting as a flat-rate group penalty and not as a curve at all. width_growth narrows the near tranches so the resolution sits where groups actually are, while the outer tranches still resolve large moves. Six tranches growing by a factor of two span the same range with a finest tranche of 0.8% of baseline, where equal widths would need dozens. Shrinking expansion_range is not equivalent: it would resolve small moves and degenerate for large ones, and large moves are the interesting regime under carbon pricing. Prices remain the slope times each tranche's midpoint deviation, now measured from the accumulated width, so the piecewise cost still traces the quadratic and stays monotone in the tranche index -- convexity needs no ordering rows. At width_growth 1.0 the construction reduces to the previous equal widths exactly, which a test pins.
`granularity` chooses whether a curve prices a country's total output of a commodity, its output per optimisation region, or each production link separately. Finer resolution prices reallocation that a coarser setting leaves to the per-link deviation penalty, so the two mechanisms can be traded off against each other rather than assumed. The elasticity is invariant to the choice: each group's slope is `cost / (elasticity * baseline)`, so a link priced on its own carries the configured elasticity with respect to its own margin, and the aggregate carries the same one wherever costs within a group are similar. Cost is 2 * n_blocks variables per group, so link granularity is the expensive end.
Standard two-phase PMP: supply_response.pin_baseline holds every group at its observed activity behind an elastic pin (pin_slack_cost) and the solve writes the pinning duals -- the per-group price wedges -- next to the solved network; supply_response.intercepts feeds them back so each curve's marginal cost at the baseline equals the marginal value there, making the observed allocation the exact optimum of the unpinned model. The slope becomes (cost + intercept) / (elasticity * baseline), stating the elasticity at the calibrated marginal cost (Merel and Bucaram 2010). The pin is elastic because the reference data is never perfectly consistent with every hard constraint: validation-mode pinning only stays feasible through land and water slack channels enabled by use_actual_production, and a hard equality pin is infeasible without them. Groups that use pin slack get their intercept censored at the slack price, are flagged in the log, and carry a slack column in the intercepts file for diagnosis.
Under a pinned diet the fitted wedges trace the Ricardian rent gradient, so sub-marginal groups at the extensive margin have c + lambda <= 0, where a calibrated-marginal-cost slope (Merel and Bucaram 2010) is undefined -- about a third of link-level crop groups in practice. Exactness only needs the intercept, so the slope reverts to Howitt's accounting-cost form c / (eta * b); a group with wedge lambda then realises elasticity eta * (c + lambda) / c with respect to its market price.
… multi-cropping The hand-rolled tranche formulation (paired bounded variables per direction, midpoint prices, a balance row per group, and a module-level price stash for the post-solve cost evaluation) is replaced by linopy's piecewise machinery: the deviation cost is sampled at the breakpoint grid and one free variable per group is bounded below by the chords from linopy.piecewise.tangent_lines. Minimisation presses the variable onto the chords' upper envelope -- the same piecewise-linear function the tranches encoded -- and its solution is the objective term, so the cost evaluation reads it directly. Beyond the outermost breakpoints the envelope extrapolates at the end chords' slopes, preserving the no-hidden-bounds property. Multi-cropping links now carry curves as their own component (multi_crops, grouped per (combination, country) bundle): at any granularity the bundle is priced as a unit, which sidesteps the attribution question that had left them uncovered.
The curves replace the L1 deviation penalty as the default anchoring mechanism, at link granularity (country and region remain available via supply_response.granularity). A new supply_response step in tools/calibrate -- replacing the legacy stability step in the default chain -- runs one baseline-pinned solve and writes each group's price wedge to data/curated/calibration/<source>/supply_response.csv, which the new intercepts: "calibrated" sentinel resolves against at solve time; both tracked artefact sets are fit and committed. The pinned solve runs under the base config's own operating regime with the policy dials at neutral: no GHG price, demand left to whatever drives it in ordinary solves. Wedges are regime-specific -- a first fit under an enforced baseline diet made a plain default solve walk 20-50% off baseline, because the free-demand regime values production differently. Fit under its own regime, the unpriced default solve reproduces observed 2020 production to 0.3% on crops and under 0.02% on grassland and animals, with the residual confined to groups whose intercepts censor at pin_slack_cost (flagged in the artefact's slack column); a priced solve then shows the pure elasticity-governed response. Configs whose builds differ structurally from the default (tests, tutorials) or that pin production outright (validation) keep the L1 penalty and disable the curves explicitly.
…ection The correction generator was sized on the intersection of harvested-area and land_use-supply bus indexes, silently dropping buses with baseline crop area but no land-cover cropland supply at all. Those 16 buses (0.016 Mha, all in irrigation-heavy or data-poor regions) were the only physically infeasible spots in the supply-response pin solve. They now receive a correction generator sized to their full baseline demand, like every other deficit bus.
At pin_slack_cost 10, 28 Mha of crop baseline (8372 links) had its wedge censored: horticulture production costs are derived as a share of producer revenue (13-30+ bnUSD/Mha for fruit and vegetables), and the neutral fit regime values output only through nutrition demand, so the genuine wedge -- the consumer price premium -- routinely exceeds 10. The censored links produced nothing in the fit and only ~85% of their baseline in deployed solves, accounting for essentially the whole 0.31% crop residual. At 100, censoring collapses to 0.037 Mha (153 links, the greenhouse- cost tail above 100 bnUSD/Mha), and together with the land-correction fix the unpriced default solve reproduces observed production at 0.002% crops / 0.000% multi-crops / 0.000% grassland / 0.000% animals. Both artefact sets refit; interior wedges are unchanged. Fingerprints restamped: the remaining staleness causes are solve-path modules the earlier steps disable, plus the 1e-5-relative land-correction change.
The extractor writes value_bnusd_per_mt = +mu_p_set and keeps negative duals; the flooring described here was reverted in edd3383.
…curves A new demand component in the supply-response machinery gives every (food, country) consumption link a concave marginal-utility curve through its observed intake -- the same convex deviation-cost form as the production curves, whose intercept is minus the fitted willingness to pay and whose slope is stated at a fitted reference price (slope_basis column) over the own-price food demand elasticity. Ordinary solves previously had no demand-side valuation at all; the component supersedes the consumer-values / piecewise-utility pattern and is mutually exclusive with it. The fit is sequential rather than joint: pinning both sides at once closes every commodity chain and leaves the producer/consumer split of each chain's wedge undetermined (the duals park at the pin slack bound). Instead the demand pin runs against the calibrated, elastic production curves (unique food-bus prices, measured against exactly the deployed supply side), the slope basis comes from the same pin against zero-intercept curves (accounting-cost-chain prices), and a configurable number of refinement sweeps re-pins each side against the other's fitted curves; the cross-side drift contracts roughly 4x per sweep. pin_baseline accordingly accepts a component list. Verified on the default config (unpriced): demand reproduces observed intake at 0.004%; production at 0.41% crops / 0.75% multi-crops / 0.01% grassland / 0.19% animals after two sweeps, converging further with the default three. gsa, gsa_fixed_diet and doc_figures keep their existing demand mechanisms via supply_response.components.demand: false. Fitted reference prices are economically sensible (beef ~9.6, chicken ~2.9, dairy ~0.8, tomato ~0.6 USD/kg); elasticity defaults are grounded in the empirical literature in the configuration reference.
The mechanism now calibrates both sides of each market, so the config section, module, functions, rule, calibration step, artefact and group key (mr_group) drop the supply-only name. The per-side cost categories keep their names: supply_response and demand_response remain the two sides in the objective breakdown and network metadata.
data/curated/calibration/default/market_response.csv now carries the sequentially fitted demand rows (intercept, slack, slope_basis) next to the production wedges; the gbd-anchored set stays production-only since its consumers keep the consumer-values mechanism. Verified on the default config (unpriced): demand 0.004%, crops 0.200%, multi-crops 0.404%, grassland 0.008%, animals 0.096% deviation from the observed baseline with the default three calibration sweeps.
…demand by food group market_response.elasticities.demand becomes a per-food-group mapping. The verified meta-regression evidence (Green et al., BMJ 2013) puts own-price food demand magnitudes at 0.43-0.78 depending on food group and income tier -- a flat 0.4 understates everything except rich-country staples -- so the defaults now carry the middle-income column as global central values (staple-like groups 0.55, meat and dairy 0.72, eggs 0.54, fruit and vegetables 0.65, fats and oils 0.5, discretionary 0.74), with documented analogs for groups the meta-regression does not cover. Elasticities enter only at solve time, so the fitted artefacts are unchanged; the deployed verification is unaffected (demand 0.010%, crops 0.201%). Supply-side values survive review and keep their citations honest: crops 0.5 sits mid-band of the long-run Nerlovian range 0.3-1.2 (Rao 1989); Roberts & Schlenker 2013's ~0.1 is explicitly a short-run, weather-identified response; livestock 0.4 is conservative against long-run cattle estimates of 0.3-2.9 and dairy 0.47-0.65 (with Jarvis 1974's negative short-run herd dynamics noted); grassland 0.3 is documented as an assumption by analogy to cropland area elasticities (Iqbal & Babcock 2018), since no dedicated estimate exists.
Member
Author
|
Two additions since opening:
|
The epigraph-over-chords formulation put 2 * n_blocks inequality rows per curve group into the model -- at link granularity about 1.5M rows, half of all rows -- and the near-ties PMP creates at the calibrated optimum made those rows expensive for simplex and barrier alike. The deviation from baseline now splits into bounded segment variables whose objective coefficients are the chord slopes; convexity fills them in merit order, so the formulation is LP-identical (A/B objectives agree to 1e-7 relative, baseline reproduction unchanged) while the chords become variable bounds instead of rows. Full-model solves speed up 7-9x under HiGHS (922 s -> 133 s at the default GHG price, 1646 s -> 187 s unpriced) and 2.4x under Gurobi (96 s -> 40 s single-threaded).
Member
Author
|
Performance follow-up, benchmarked on the full default config: Formulation (5597235): the curves now enter the LP as bounded deviation-segment variables instead of an epigraph over chord rows. LP-identical by construction (A/B objectives agree to ~1e-7 relative; baseline reproduction unchanged at crops 0.201% / animals 0.096% / demand 0.003%), but the old formulation's ~1.5M chord rows (half of all model rows) become variable bounds. Deployed solve times:
This removes the practical need for Gurobi on demand-component configs. Solver tuning (measured, mostly negative results worth recording): |
koen-vg
changed the title
…al base config Calibrating against config/gsa.yaml coupled the shared artefact set to one consumer config: any structural override added there would silently redefine the set, and the fingerprints recorded the whole GSA config as a calibration input. The new config/gbd_anchored.yaml is the default config with the GBD-anchored baseline diet as its only structural change, which is verifiably (via the structural snapshot) what the set was fit against all along.
Full-chain refit after the market-response hardening: the diet preparation and zero-baseline handling changed, and the gbd-anchored set now carries demand curves like the default set (fit against its new minimal base, so the two sets differ only in the baseline diet). Demand censoring is minimal in both (12 groups, mostly sugar in small countries, total slack 0.09 Mt).
Member
Author
|
Recalibrated both artefact sets after d9f3a13 (4839b18), and gave the |
…satiation The equality pin of a phase-1 market-response fit is redundant with the capacity bound wherever baseline_area_mha equals p_nom_max (the suitable-area clip makes this the case for a quarter of crop links), so the pin dual is degenerate and the solver reports the slack-bound extreme: 26% of crop and 15% of grassland intercepts sat at exactly -pin_slack_cost, which at deployment priced contraction of 229 Mha of cropland at 100 bnUSD/Mha and effectively froze it. Capacity bounds of pinned links are now lifted (before the frozen linopy model is created) so the pin is the unique binding constraint and its dual is the true wedge, scarcity rent included. Demand groups whose marginal utility is still positive at the outer end of the sampled range no longer extrapolate that utility forever on the unbounded tail: the tail's cost slope is floored at zero marginal utility, so consumption cannot run away when supply becomes very cheap while nutrition constraints can still push intake up at zero utility. Also: production links now fail loudly on a missing baseline like demand links do (instead of silently becoming a zero-fixed group), grouping columns reject missing values, the intercept extractor and the calibration driver report duals censored at the pin slack bound, the zero-baseline log states the fixed share per component, and solves with no production anchoring at all warn (new use_actual_production solve param, mirrored in both rule harnesses and the cluster manifest).
…ion chain The default chain run by tools/calibrate (and watched by --check and --record) is now feed, food_waste, food_demand, market_response: with cost_calibration disabled by default and mutually exclusive with market_response, the cost artefacts feed nothing on the default path, so cost joins stability as an explicitly-run legacy step (in that order; stability consumes the cost corrections). --check/--record still cover a legacy step whenever its artefacts are present in the set. Seeding of fresh --base artefact sets is removed: each step consumes only its predecessors' artefacts, so the chain regenerates a new set from scratch in order, single steps require their predecessors on disk, and legacy artefacts appear only when their steps are run. This also stops a fresh set from silently carrying the default set's deviation_penalty.yaml under a provenance stamp claiming it was fit against the new base.
With market_response enabled, schema validation now requires components.demand and validation.enforce_baseline_diet to be exact opposites: the demand side is either elastic or held at the observed baseline. Both regimes reproduce the fitted point of the sequential calibration (the final production pin runs with the demand curves active), whereas pairing the production intercepts with any other demand valuation silently breaks baseline reproduction -- which gsa.yaml and doc_figures.yaml did by substituting piecewise food utility. Both are migrated onto the demand component; gsa also drops its consumer-values baseline scenario (nothing consumes it with piecewise utility off) and doc_figures keeps its baseline scenario, demand-off, for the consumer-values figure. Also: the test config restores the legacy cost corrections alongside its L1 anchoring (the cost_calibration default flip had silently left it solving against uncalibrated costs), gsa.yaml points its regenerate hint at config/gbd_anchored.yaml instead of overwriting the shared set with a gsa-specific fit, and tests cover the new schema rule, the supply/demand response breakdown categories, and the GDD unit-map completeness guard.
…nism Fixes the drift a full review of the branch surfaced: the last surviving description of the removed per-component suppression gate (costs.rst), stale claims that the GSA configs anchor with the calibrated L1 penalty (sensitivity_analysis.rst, consumer_values.rst), the objective-breakdown column table missing the supply_response and demand_response categories along with several older ones (analysis.rst), the market-response calibration section not listing market_response.csv among the diet-sensitive artefacts nor among the consumed calibration outputs, missing config-reference entries for pin_slack_cost and the calibration subsection, and the calibration rule's docstring describing a single pinned solve. Also states the contracts the review pinned down: the configured elasticities act per relative change of the reference price (accounting cost / slope basis), not as arc elasticities at the curve's equilibrium wedge; the zero-baseline closure magnitudes; the capacity lift during phase-1 pins; and the demand satiation floor.
Recording only ever updated its own step entry, so the pre-rename supply_response block survived in both committed fingerprint files as a dead, uncheckable entry pointing at an artefact that no longer exists.
…ibration Both artefact sets are refit with the pinned-solve capacity lift: the mass of intercepts censored at the pin slack bound collapses from 26% to 0.6% of crop groups (15% to 4% of grassland), so the wedges of capacity-saturated links are now the true local rent gradient instead of a degeneracy artifact that froze 229 Mha of cropland against contraction. The legacy deviation_penalty.yaml in both sets is re-calibrated against the current cost corrections (default: cropland 1.38, grassland 0.23, feed 0.063; gbd-anchored: 1.52/0.24/0.056), the fingerprints cover the legacy cost and stability steps and drop the dead pre-rename supply_response blocks, and both sets check green. Verification (unpriced default reference solve, nothing pinned): crop area 0.14%, pasture 0.01%, animal feed 0.31%, food consumption 0.003% off the observed 2020 baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Folds in #46 (formerly stacked on it; that PR is closed and its full diff is included here).
What
Production and consumption anchoring is now done by convex piecewise-linear marginal-cost / marginal-utility curves in the positive-mathematical-programming tradition (Howitt 1995), calibrated exactly by the standard two-phase procedure, replacing the L1
deviation_penaltyand its Broydenstabilitycalibration as the default mechanism.Supply side
market_responsestep intools/calibrate, replacingstabilityin the default chain): one solve with every curve group held at its observed activity behind an elastic pin. The pinning duals -- per-group wedges between marginal value and accounting cost -- land indata/curated/calibration/<source>/market_response.csv(both tracked sets fit and committed).market_response.intercepts: "calibrated"feeds the wedges back as curve intercepts, so each group's marginal cost at the baseline equals the marginal value there and the observed allocation is the exact optimum -- no hard constraints, no tuned deviation target. The configured elasticity (crops 0.5, grassland 0.3, animals 0.4; literature-grounded in the configuration reference) governs only the response away from the calibrated point.link(default -- prices spatial reallocation itself, so no deviation penalty is needed),region, orcountry. Multi-cropping links carry curves as their own component, per(combination, country)bundle.Demand side
food_utility_piecewisepattern and is mutually exclusive with it.market_response.calibration.sweeps(default 3) Gauss-Seidel passes contract cross-side drift ~4x per sweep.Formulation and performance
The curves enter the LP as bounded deviation-segment variables (chord slopes as objective coefficients, merit-order fill by convexity) rather than epigraph rows; at link granularity the row-based form put ~1.5M rows (half the model) into the LP. Full-model deployed solves: HiGHS 133 s / Gurobi 40 s at the default GHG price (187 s HiGHS unpriced), making the open-source solver fully viable for demand-enabled configs.
Verification (default config, unpriced reference solve, nothing pinned)
Fitted reference prices are economically sensible: beef ~9.6, sheep ~8.5, chicken/pork ~2.9, dairy ~0.8, eggs ~1.4, tomato ~0.6, rice ~0.3 USD/kg. The priced default solve (GHG 200 USD/t) then shows the pure elasticity-governed response.
Key design points
pin_slack_cost100): reference data is never perfectly consistent with land/water/feed constraints, and a hard pin is infeasible. Censored duals surface inconsistencies instead of hiding them (the extractor and the calibration log now report duals parked at the bound explicitly).baseline == p_nom_max(the suitable-area clip), where the pin is redundant with the capacity bound and its dual degenerate -- the solver parked 26% of crop intercepts at the slack bound, which at deployment froze 229 Mha of cropland against contraction. Phase-1 pins now lift the capacity bounds of pinned links (before the frozen linopy model is created), so those wedges are the true local rent gradient; censoring collapsed to 0.6% of crop groups.docs/calibration.rst).c + lambda <= 0, where a calibrated-cost slope is undefined. A group with wedgelambdarealises elasticityeta * (c + lambda) / c.Demand-side regime is validated, not documented
With
market_responseenabled, schema validation requirescomponents.demandandvalidation.enforce_baseline_dietto be exact opposites: the demand side is either elastic or held at the observed baseline. Both reproduce the calibration's fitted point (the final production pin runs with the demand curves active); pairing the production intercepts with any other demand valuation silently breaks baseline reproduction, and the component switches are solve-time keys that provenance checking cannot see. Consequentlygsa.yamlanddoc_figures.yamlare migrated onto the demand component here (piecewise utility off; gsa's consumer-valuesbaselinescenario is gone), andgsa_fixed_diet.yamlkeeps demand off with the diet enforced.Calibration chain
The default chain (
tools/calibrate, watched by--check/--record) is now feed, food_waste, food_demand, market_response. The legacy anchoring steps --costandstability-- sit outside it and are run explicitly (in that order) for sets consumed by configs that anchor with them;--check/--recordstill cover them whenever their artefacts are present. Fresh--basesets are no longer seeded: each step consumes only its predecessors' artefacts, so the chain regenerates a new set from scratch and legacy artefacts appear only when their steps are run. Both shipped sets carry freshly refit curves, current cost corrections, and a re-calibrateddeviation_penalty.yaml, all checking green.Migration
deviation_penaltymachinery remains available; the tests and tutorials keep the L1 explicitly (withcost_calibration: {enabled: true}, since PMP and the cost corrections are mutually exclusive).tools/calibrate --base <config> market_response.market_response: {enabled: false}+cost_calibration: {enabled: true}+deviation_penalty: {enabled: true}.