Households stock heat migration

[aaccensi]

Context

The sliders/inputs for both the Buildings and the Households stock and heat demand are changing, so both need a migration to go along. This PR now carries both, so they can land together in a single merge to master.

The buildings migration was reviewed and approved in #1796; that PR is being closed in favour of this one rather than merged separately.

Implemented changes

• db/migrate/20260804120000_simplified_buildings_heat_inputs.rb — the buildings migration
• db/migrate/20260813120000_simplified_households_heat_inputs.rb — the households migration
• db/migrate/20260813120000_simplified_households_heat_inputs/dataset_values.json — the start-year heat demand of every housing type and construction period, per dataset. The households insulation sliders are averaged by the demand each housing type carries, and that demand is solved by the graph rather than stored in ETSource, so it cannot be recalculated in the migration. Generated with the refinery_dump task in ETSource.
• db/schema.rb — the schema version (and Rails 8 alphabetical order)
• db/cache_schema.rb — Rails 8 alphabetical order only

Related

• ETSource: Simplified heat sliders - built environment etsource#3470 (households) and the buildings inputs — both must be deployed before this migration runs
• ETModel: Simplified heat sliders - built environment etmodel#4775
• This PR replaces entirely Buildings stock and heat demand migration #1796

Checklist

• I have tested these changes
• I have updated documentation as needed
• I have tagged the relevant people for review

[aaccensi]

@kndehaan, I realized that there is one of those cases where things can happen with the data even if it does not make much sense. See the "important note" block in the PR description.

[kndehaan]

How is this handled in the migration? I assume that the new value is capped to the maximum value of the new slider, right?

[aaccensi]

Currently is unhandled. The new slider currently is just populated with the sum of the previous ones, but the frontend cannot display that properly. A decision on this needs to be made and then I will adapt the code accordingly.

[kndehaan]

Can you cap it to the max slider value? We do not want to allow set user values to be outside the min and max bounds.

[aaccensi]

Sure, we can do that. For scenario owners it means their scenario may lose the homes above the new limit, which is 1/4th of what was possible before for the entire housing stock.

On our side, in the migration code, besides capping the total we need to scale down the four housing-type numbers along with it, so the shares per housing type still add up to 100%.

[kndehaan]

For scenario owners it means their scenario may lose the homes above the new limit, which is 1/4th of what was possible before for the entire housing stock.

I checked the featured scenarios: it's never the case that total nr of new residences exceeds the total nr of existing residences in the start year. So this approach seems fine.

[kndehaan]

It came to me that for converting the household_insulation_level* inputs, a different approach should be taken for the future ones and the existing construction periods.

For the future residences, the current approach of weighing the input with number of units, floor area and kWh/m2 is correct since that corresponds with how we determine heat demand for new residences in the future year.

For the existing residences (all existing construction periods), we determine the demand in the future year based on scaling of the demand in the start year. Calculation example:

heat demand start year = 1000 MJ
typical heat demand start year = 60 kWh/m2
typical heat demand future year = 50 kWh/m2
heat demand future year = 50/60 * 1000 MJ = 833 MJ

Doing a similar calculation for the migration requires a data dump of the demand of the relevant nodes in the start year, see below. The data dump needs to contain these values for all datasets. Notifying @louispt1 as well reg the data dump requirement.

Relevant nodes:

households_useful_demand_for_space_heating_apartments_before_1945
households_useful_demand_for_space_heating_detached_houses_before_1945
households_useful_demand_for_space_heating_semi_detached_houses_before_1945
households_useful_demand_for_space_heating_terraced_houses_before_1945
households_useful_demand_for_space_heating_apartments_1945_1964
households_useful_demand_for_space_heating_detached_houses_1945_1964
households_useful_demand_for_space_heating_semi_detached_houses_1945_1964
households_useful_demand_for_space_heating_terraced_houses_1945_1964
households_useful_demand_for_space_heating_apartments_1965_1984
households_useful_demand_for_space_heating_detached_houses_1965_1984
households_useful_demand_for_space_heating_semi_detached_houses_1965_1984
households_useful_demand_for_space_heating_terraced_houses_1965_1984
households_useful_demand_for_space_heating_apartments_1985_2004
households_useful_demand_for_space_heating_detached_houses_1985_2004
households_useful_demand_for_space_heating_semi_detached_houses_1985_2004
households_useful_demand_for_space_heating_terraced_houses_1985_2004
households_useful_demand_for_space_heating_apartments_2005_present
households_useful_demand_for_space_heating_detached_houses_2005_present
households_useful_demand_for_space_heating_semi_detached_houses_2005_present
households_useful_demand_for_space_heating_terraced_houses_2005_present

[aaccensi]

Ok @kndehaan, so for the migration of the insulation the future residences work as before but the existing residences now weigh in based in their start-year heat demand rather then the floor area. Could you check if this gives you the expected results? Once you give the thumbs up, we would only be needing a code check by @louispt1.

[kndehaan]

@louispt1 note that it's not needed to review 20260804120000_simplified_buildings_heat_inputs.rb, which has already been reviewed in a separate PR by Nora.

[kndehaan]

Migration approved in terms of working principles.

[louispt1]

Why is demolished_key passed in here instead of derived like all the other keys? It seems to follow the same pattern and if you set it up the same way you don't need to pass demolished_key: nil in NEW_PERIOD

Is there a case where you pass a non-derived key?

[aaccensi]

Since the new period never asks for I we can do it.

[louispt1]

I'm concerned that for scaled datasets with round we could end up above the maximum (present_number_of_residences can be fractional)? This effects two places - also the demolished total (scenario.user_values[period.demolished_key] = demolished.round(RESIDENCES_DECIMALS).to_f).

Clamping to the max after rounding could be a solution, although you also end up with fractions there which would be rejected by the step value. Alternatively you can floor and then you're within bounds, in which case you might end up 1 residence short of the maximum.

What do you think?

[kndehaan]

Good point! The most important thing here is to not go beyond the min/max values of the input.

I think floor is the best option here since this is both in line with the max value and step value. @aaccensi what do you think?

[aaccensi]

Lets do something about it, even though this would only happen when demolishing everything or building beyond the cap (Kyra confirmed already the latest was not in our data). Simplest is flooring which is also the closest to what would happen in the frontend to a scaled value defined with 0 decimal places.