Skip to content

feat(FRBC): correct a usage forecast no plan can absorb, instead of clipping it - #286

Merged
Flix6x merged 119 commits into
mainfrom
feat/s2-usage-correction
Nov 10, 2026
Merged

Flix6x merged 119 commits into
mainfrom
feat/s2-usage-correction

Conversation

@Flix6x

@Flix6x Flix6x commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Base main. The integration branch reached main through #279; this branch has it and main merged in, so the diff shows only #286 and #289. The unit changes of #289 reached main separately, through #290.

This branch now also contains #289 (leakage planned as usage, and rates of fill level and site sensors in W), which was merged into it.

Why

S2 lets a resource manager expect any usage, negative included, and says nothing about what happens when the storage cannot take it. FlexMeasures plans a forecast usage as forced. So a gain at a full storage, or a draw at an empty one that the actuator cannot keep up with, leaves no feasible plan, and the storage gets no instruction. Until now the handler clipped negative usage to zero, which hid the first case and not the second.

What

With a usage_correction_sensor_id and FlexMeasures v1.1.0 or newer, FRBCSimple asks for a schedule with a correction of the usage forecast:

  • A second device on the storage's stock (same state-of-charge sensor), on a commodity of its own, so it never touches the site's electricity. No instruction is ever sent for it.

  • Per step, it may only move the expected usage towards zero: give back up to the expected usage, or take away up to the expected gain. It works the same on both sides, since S2's fill levels do not say which bound is an overflow.

  • It is priced as a last resort: at twice an upper bound on what one MWh of stock can be worth to the scheduler. The request carries a coherent set of prices to make that bound finite and numerically sound:

    • band breach prices at 10× the highest absolute market price over the window (soc_breach_price_factor);
    • the asset's own stored per-MW prices (peaks, commitments) added to the bound;
    • a breach of the site's capacity left out of the bound (see below), with the site's breach price kept from the asset, or else stated at FlexMeasures' default of 10000 per kW.

    Device capacities are kept hard, so the correction cannot move a usage past zero.

  • A negative expected usage is then planned as stated. It is also recorded as stated without a correction (since 905254e, from a review comment on docs(changelog): say each S2 change plainly #283): there, the schedule request carries the forecast without its negative values. Only a FlexMeasures server before v0.23.0, which takes a usage only as a sensor, still gets it recorded as zero. The usage is sent inline, from the same forecast as the correction's limits and the translation back into instructions.

  • Where a correction cannot be planned (prices unreadable, a state-of-charge sensor with a resolution, a stored price that cannot be bounded, a stored flex context with commodities), the forecast is planned without its negative values, as before, with a warning saying why.

It is opt-in: FRBCSiteConfigurator(correct_usage=True), or S2_CORRECT_USAGE=1 for the example server. That is because it replaces FlexMeasures' default band breach price (1000 EUR/kWh) with a market-relative one, which is the deployment's call. Handlers without the new sensor behave exactly as before.

How the price was chosen

Probes against FlexMeasures v1.1.0 measured, per scenario, the correction price above which the scheduler stops using a correction it does not need. That threshold is what a MWh of stock is worth to the scheduler. Each matched one term of the bound:

What makes stock worth something Worth per MWh of stock
Buying or disposing of energy |price| / charging efficiency, divided again by the storage efficiency per step held
Flow commitment, peak, or any-breach price per MW price / step length
Site capacity breach P / step length + P (measured, but left out of the bound)
Fill-level band breach P / step length + P per step breached

With FlexMeasures' default breach prices, a dominant price was about 1e8 EUR/MWh, where HiGHS already returned over-corrected plans. With the coherent set, over a 24-hour window with prices from −120 to 450 EUR/MWh, the correction price is about 1.8e6 EUR/MWh:

  • It was used only where physics forced it.
  • It was never used instead of a band breach, or to profit from negative prices.
  • Solves took about 0.7 s and were identical on repeat.

Priced at the band's breach price instead, the correction replaced band breaches (9.24 vs 3.12 kWh) and corrected where nothing needed correcting.

Why the site's capacity breach is left out. Pricing the correction above a site breach needs the request to set a site breach price the solver can live with. In the co-simulation (below), a site breach price of 100× the market price kept the correction solvable but let the houses ignore their steering capacity, and 1000× pushed the correction price to 1.3e7 EUR/MWh, where HiGHS declared every house schedule infeasible (replays put the cliff between 0.7e7 and 1.3e7 for this house). So a correction can now be cheaper than a site breach, while it stays dearer than a band breach, which the plan still uses first.

The asset's stored inflexible devices are carried into the electricity entry, and named empty at the top level, because FlexMeasures merges the stored context in there too and would count them twice (FlexMeasures/flexmeasures#2848).

FlexMeasures fills in its default site breach price only for a flex context without commodities; in a commodity entry, a site capacity without one stays hard. The request therefore states that default itself (FlexMeasures/flexmeasures#2846).

Evidence

  • Tests: 809 pass.
  • Mutation gate: 42 of 42 mutants killed.
  • FlexMeasures v1.1.0 end to end: the exact request this handler builds, for a storage at 118 kWh under a 120 kWh hard top with a forced 14.4 kWh gain, is accepted and solved. The correction removes exactly 12.4 kWh, and the stock never exceeds 120. Without the correction device, the same request has no feasible plan.
  • Adversarial review: found that the stock read every forecast on the sensor while the limits used only the latest; that the asset's stored electricity flex context was dropped under a commodity list; that a stored relax-capacity-constraints would soften the limits; and that efficiencies above one lowered the bound. All are fixed in the second commit, which also makes the correction opt-in.

Co-simulation

One steered day, two houses, the community capacity policy, deterministic solver settings. Every run below had 0 degree-hours outside the comfort band.

Run Client Breach (kWh) Peak (kW) Notes
C1 #279, no correction 0.058 2.21 reference
D3 site breach 100× market 0.348 2.267 correction engaged, steering ineffective
D4 site breach 1000× market every house schedule infeasible (correction price 1.3e7), run aborted
D5 site breach left out, none stated site capacity hard in the commodity entry, schedules infeasible, stopped
D6 4b7b023 0.052 2.21 inflexible devices counted twice: the houses planned against a doubled base load
C2 a31f1cf, correction off 0.052 2.21 matches C1
D7 a31f1cf (this head) 0.745 2.55 inflexible devices counted once, 3.5 h over the limit
D7b a31f1cf, repeat of D7 0.778 2.55 reproduces D7
D7c a31f1cf, community controller fixed 0.052 2.21 matches C2: 0.25 h over the limit
  • With the correction off, this PR behaves like fix(S2): the remaining findings (2 of 2) #279 (C2 vs C1).
  • D6's match with C1 was an artefact. Its planned residual of 2.419 kW came from double-counted inflexible devices: at its peak, a house's planned aggregate power was exactly twice its inflexible load, with the heat pump planned at 0 kW. a31f1cf counts each device once, and D7's planned residual is back at C1's 0.21 kW.
  • With the correction on, the community limit held worse in D7 and D7b, but the cause was in the co-simulation's community controller, not in this PR. With the correction on, each request differs from the last (the correction's capacity and price change per job), so FlexMeasures records each house's plan under a new data source per job. The controller combined those sources oldest first, so it steered against each house's first-round plan rather than its latest. Fixed there, the same run (D7c) holds the limit exactly as C2 does, with comfort kept. Details on the diagnosis are in this comment and the next one.
  • For other deployments: anything that reads a site's schedule should take the latest belief across scheduler sources, not rank sources, since a request that changes per job yields a new source per job.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW

Flix6x and others added 30 commits November 5, 2026 13:10
FillRateBasedControlTUNES.send_leakage_behaviour read its fill-level scale from
self._fill_level_scalefill_level_scale, an attribute name pasted in twice, which
has been there since #158. The AttributeError that raised was caught by the
handler's broad except as if the post had failed, so every FRBC.LeakageBehaviour
was answered with PERMANENT_ERROR and none was ever recorded.

mypy could not see it: the type-check task only looks at files that contain
`from typing import`, and the FRBC base class is not one of them, so the
attribute lookup went to Any. The next commit widens that.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
The type-check task fed mypy only the files that contain `from typing import`,
which were seven of eighteen, and left out frbc_simple.py, both control-type
__init__.py files and the example scripts. With --follow-imports skip, anything
those files defined then looked like Any to the files that were checked, which
is how the previous commit's misspelt attribute went unnoticed since #158.

It now checks all of src, and with --check-untyped-defs, so the bodies of
functions without annotations are checked too. What that surfaced, besides the
attribute fixed in the previous commit:

- get_access_token stored a token whose type could be None and then used it as a
  header value; it now returns the token, so the caller holds a str.
- get_access_token and get_versions indexed a response typed as a dict or a list;
  each now says so plainly when the server answers with something other than an
  object, instead of failing on the lookup.
- The S2 translations annotated their resolution as a string while frbc_simple
  passes a timedelta, which pandas accepts as readily; the annotation now says so.
- FillRateBasedControlTUNES overrode trigger_schedule without the arguments the
  base class declares. It calls its own override without them and ignores them,
  so it now accepts and ignores them.
- The example server indexed an account that can be None and spread a dict of
  mixed values into add_asset; it now says so when the demo user has no
  organisation, and types the dict.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
None of the S2 integration had a coverage number, for two reasons that each hid
the other. The S2 tests run in a CI job of their own, since they need the s2
extra, and that job never asked for coverage. And .coveragerc omitted
src/flexmeasures_client/s2/* outright, since #183, so asking for it would have
reported nothing about S2 while looking as though it did.

The omission is gone, and the S2 job now reports and uploads its own partial
report, which Coveralls merges with the other job's in the finalize step, as it
already does across the test matrix. The poe task that runs the S2 tests locally
collects it as well. The job that skips tests/s2 will now list the S2 files as
unexecuted in its own report; the merged figure is the one that counts.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
A CEM's handlers run for days with nobody watching standard output, so anything
they print is lost to a deployment, which keeps the log instead. FRBCSimple gave
up on a schedule that way when no storage status had arrived, which is the very
failure, a storage left without an instruction, that the handler otherwise goes
to some length to say out loud. It now logs a warning.

A pygrep hook keeps it that way for s2/control_types and s2/cem.py. The example
scripts print on purpose, as do the client's interactive delete confirmations,
which answer an input() prompt on the console, so the hook leaves those alone.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
- The print ban now matches a print anywhere on a line and with a space before
  its parenthesis, plus pprint and writes to stdout or stderr, and covers all of
  s2/ but the example scripts. It only caught an indented `print(` at the start
  of a line before, in two directories, and missed s2/__init__.py, utils.py and
  wrapper.py, which run inside the CEM as much as the handlers do.
- The example scripts get an __init__.py. Without one, coverage never listed
  websockets_server.py at all, since no test imports it, so the S2 report left
  it out instead of showing it at 0%. Running them by path is unaffected.
- The TUNES leakage test now pins what is posted, its value and its unit,
  rather than only that something reached the sensor.
- A client without a token now has a test that it authorises with the one it
  fetches.
- FillRateBasedControlTUNES.trigger_schedule is back to taking no arguments,
  with the override mismatch marked for mypy instead. Accepting the base
  class's arguments and ignoring them would have turned a caller's mistake,
  such as expecting its superseded-request protection, into a silent one.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…s them in

S2 lets the elements of an operation mode, and of a leakage behaviour, cover the
fill-level range in pieces, and s2python checks that they are contiguous on a
sorted copy, leaving the list in whatever order it was sent. Four places took an
element by its position instead, and answered three different ways:

- the measured power of an actuator status took elements[0], so a heat pump that
  derates as its buffer warms was recorded at its cold power when warm: 2000 W
  where it draws at most 1200 W;
- the charging efficiency took elements[-1] as though it were the top of the
  range, so a mode listed high element first planned against the wrong ratio;
- the storage efficiency took the leakage rate of elements[-1], on the same
  assumption, though the leakiest element need not be the top one;
- the schedule translation selected by fill level, but settled a shared boundary
  by list order.

element_for_fill_level now answers wherever a fill level is known: the element
whose range holds it, the upper of two at a shared boundary, and past either end
of the range the element that applies at that end, so the answer does not jump
there. The measured power and the schedule translation use it. Before any storage
status there is no fill level, so the measured power takes the bottom of the range
and says in the log that it guessed.

Where FlexMeasures takes one number for a whole schedule, it does not know which
fill levels the schedule will pass through, so the most cautious element answers,
which is the rule #253 already applies across modes: the lowest conversion
efficiency, and the leakiest element. That is also independent of order.

The efficiency now also skips an element that draws power while the storage does
not fill, such as a standby or circulation mode. Before, such a mode gave a ratio
of zero or below, which then won the minimum across modes, went onto the
charging-efficiency sensor, and told the scheduler that no power could fill the
storage at all; with the fill-level band priced rather than hard, never running
the heat pump was then a feasible plan.

Two guards were dropped rather than tested, because S2 makes them unreachable:
every operation mode and leakage behaviour has at least one element, and every
element at least one power range.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…owest

An adversarial review showed that the lowest efficiency over a mode's elements
is not the cautious choice the previous commit called it. In FlexMeasures' linear
program a charging efficiency scales what the actuator adds to the stock, so
understating it protects the lower band only: the plan overshoots the upper
band, buys more than it uses, and against a usage forecast the plan cannot
relax, it can turn a problem the actuator can meet into one the scheduler
finds infeasible. A heat pump at a COP of 4 cold and 2 warm, planned at 2,
empties its planned buffer in a few hours while the real one keeps up.

So within a mode, the element that applies at the fill level the schedule
starts from answers. That is also the part of a schedule that runs before the
next storage status asks for a new one, and it is what the actuator really does
there. The storage efficiency likewise takes the leakage element at the latest
fill level, and the leakiest only where no fill level is known yet.
Across modes, the lowest still goes out, as #253 decided; the same argument
applies to it, and that is left as a question rather than changed here.

Also from the review:
- A fill level that is not a number, which s2python lets through, counts as no
  fill level, and the selector refuses one, where it used to pick the bottom
  element silently.
- The selector's docstring now says what is true: s2python checks contiguity for
  an operation mode only, never for a leakage behaviour; a single-point range,
  which the standard rules out, applies only at the top; and of two elements
  with the same range, the first listed answers.
- Tests now cover the latest of several storage statuses, a fill-level scale
  other than 1 where scaling would pick another element, no warning for a mode
  with one element, and a schedule starting in the upper element. Each was a
  production mutation that passed the suite before.

Formatted with black as well, which the previous commit was not: the
pre-commit hooks were never installed in this repository, so they did not run
on commit, and CI's check caught it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
S2 makes an operation mode's diagnostic label optional, and
FillRateBasedControlTUNES tells its modes apart by looking for "THP" or "NES"
inside it. On a mode without one that raised a TypeError, so a single
unlabelled mode stopped both planning and the recording of every mode's
conversion efficiency. A missing label now simply matches neither.

Found by type checking against s2python's own types, which also asked for two
variables that held a range and then a number, or a message and then a series,
to be given a name for each. New tests cover the planning path, the efficiency
path, and posting a usage forecast, none of which any test reached before.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…inted

request() ran check_content_type after its retry loop. It could never raise
there: request_once parses every answer with aiohttp's json(), which refuses any
content type but JSON, and the loop already reports that refusal as a
ConnectionError, like every other failure to reach the API.

Had it fired, it would have done harm. It built aiohttp's ContentTypeError from
a message and a dict, where aiohttp expects the request's info and history, so
anything that formatted the error, a log line included, raised an AttributeError
of its own; and it read the body with response.text() without awaiting it.
Its only test passed a mock whose text() was not a coroutine, and never
formatted the error.

The check is gone. The test that replaces it states where a non-JSON answer,
such as a proxy's error page during a restart, is actually handled.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…ncluded

mypy ran with --follow-imports skip, so s2python, pydantic, pandas and aiohttp
were Any to it, and a misspelt attribute on an S2 message passed. It now follows
imports, with pydantic's plugin, since S2 messages are pydantic models, and the
pre-commit hook installs the s2 extra so those types are there to check. The
plugin is also the guard against sliding back: without the extra it fails to
load, rather than leaving every S2 type silently Any. A misspelt message_id on a
leakage behaviour is now an error naming it.

What it found, besides the two fixes in the previous commits:

- get_validation_error_summary iterated a pydantic ValidationError, which is not
  iterable, so it raised a TypeError on any error it was given. Nothing called
  it, so it is gone rather than repaired.
- Revocations were recorded by the RevokeObject's raw id, while every other id
  a handler keeps is text, so a lookup by the text of the id would have missed.
  They are recorded as text now. Nothing calls is_revoked yet, which means a
  revocation is acknowledged and then never consulted; that is recorded as a
  finding for later rather than changed here.
- The rest is annotation: a message that arrives as a model, a dict or text is
  normalised to a dict once and checked as one; get_message returns the dict it
  always returned rather than claiming a str; the message type of a registered
  schema is read through model_fields rather than the deprecated __fields__; an
  operation mode's id is text where a label is wanted; and the actuator's power
  ranges are pairs rather than tuples of any length.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…ested

An adversarial review ran every claim on this branch, and three were false:

- The removed content-type check could fire. aiohttp's json() accepts JSON
  variants such as application/problem+json and application/vnd.api+json, and
  the check refused those, with an error that could not be printed. Removing it
  is right, but it changes behaviour: such an answer is now read like any other.
  The comment at the removed call, the test's docstring and the changelog entry
  now say so, instead of calling it dead code.
- "A non-JSON answer is reported as a ConnectionError, as before" was only true
  of a wrong content type. A body that is not JSON under a JSON content type is
  refused by json() itself, as a JSONDecodeError, as it was before.
- One unlabelled TUNES mode stopped planning, and the recording of the
  efficiencies of the modes listed after it, not of every mode.

Two more overstatements live only in pushed commit messages and the PR body:
following imports leaves pandas Any, since it ships no type information and no
stubs are installed, and the efficiency path was already reached by a test.

Tests now cover what five mutations showed was unpinned: a revocation taken by
the CEM itself, with no control type active; a model of a type a handler does
not handle; a usage forecast starting off the quarter hour; and one averaged
within a quarter hour rather than taking its highest value.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…ions

The schedule translation keeps a running fill level, so that each step is
answered with the operation mode for the fill level the storage is then at. It
advanced that level by a fill rate per second times a step in hours, so it moved
3600 times too little, and every step of a twelve-hour schedule was answered at
the fill level the schedule started from (finding C1). Fixing that alone would
have been worse: the level would then rise at the real rate and never fall, and
pin every step at the top. So the integrator gets one written contract, the one
S2 states for all three rates: a change in fill level per second, in the
resource manager's own units, with usage and leakage lowering it where positive.
Its step is a duration rather than a number of some unit.

- Leakage now counts (C2). It read leakage_rate_per_second, which S2 does not
  have, and an actuator's leakage_behaviour, which cannot exist, both through
  getattr with a default, so the branch never ran. The handler now passes the
  latest FRBC.LeakageBehaviour, and the element that applies at the fill level.
- Usage now counts. The handler passes the latest FRBC.UsageForecast, read per
  step by usage_rate_at; a frame may still carry a usage_forecast column instead,
  per second, which is what TUNES does, now dividing by the quarter hour it
  posts in.
- The step is the schedule's own (B5): its duration over its number of values,
  which is the resolution of the sensor FlexMeasures answered at. It was taken
  to be a quarter hour whatever it was, so an hourly sensor's twelve values were
  laid on forty-eight quarter hours, pandas refused, and no instruction went out.
- An instruction goes out wherever the mode or its factor changes, not only
  where the power does: at a constant power the fill level can pass from one
  element into another, which needs another factor.
- The factor is the one that draws the power asked for (C4), taken from the power
  range, rather than worked back out of the fill rate, which failed wherever the
  fill rate was one value and the power a range: asked for 1200 W from a mode
  running 800 to 1200 W, that asked for 800 W. Where the power is one value and
  the fill rate a range, every factor draws that power, so the one that fills
  most is asked for. op_mode_compute_factor is gone.
- The storage efficiency compounds a leakage rate per second (B10), where it
  compounded it per hour, declaring a buffer that loses about 7% over twelve
  hours as holding its heat to within 0.002%.
- The handler's power and fill-rate measurements cover its own resolution
  (D9), rather than a quarter hour whatever it planned on.
- translations.py is linted again: it opened with `# flake8: noqa`, which hid
  nothing, as it turns out.

TUNES shares the translation, so its instructions change with it, for the
better: it carried the same per-hour integration.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…showed it did not

An adversarial review built its own integrator from S2's definitions, a second at
a time, and found the previous commit agreed with it only while nothing changed
within a step. Where something did, it did not:

- A step that crossed from one element into another was integrated as if it had
  not: from 46, filling at 0.02 per second below 50 and 0.01 above, a quarter hour
  at factor 1 reaches 57, where holding the first element said 64, and at an hour's
  step the client thought a buffer full that was at 84. The same held for leakage.
- Usage was sampled at the instant a step began, where FlexMeasures planned with
  the mean over the step, so a five-minute draw in each quarter hour counted fully
  or not at all, and a forecast starting off the step counted as nothing.

follow_operation_mode now follows a step a minute at a time, as a resource manager
follows an instruction: the mode and the factor are what the instruction fixes,
while the element of the mode and of the leakage behaviour, and the usage a
forecast expects, are each taken where and when they apply. The fill level is held
between its bounds throughout, rather than only at the end of a step.

Also from the review:
- An instruction no longer goes out for a solver's own rounding: near factor 0,
  math.isclose without an absolute tolerance is exact equality, so 800 W give or
  take a ten-millionth from a mode running 800 to 1200 W sent 30 instructions in
  twelve hours instead of one.
- A gap in a usage column, as FlexMeasures returns for a moment it holds nothing
  for, is no usage known, rather than a NaN that pinned the running fill level at
  the top of its bounds for the rest of the schedule.
- FillRateBasedControlTUNES reads its usage forecast back divided by the
  fill-level scale it posted it with, which left it ten times too small at TUNES'
  default scale, and plans with its latest leakage behaviour. The previous commit
  message called TUNES' change "for the better"; without these it was not.
- One power and a fill-rate range listed downwards asks for the top of the range,
  where operation_mode_conversion_efficiency declares the efficiency FlexMeasures
  plans with, so the instruction and the plan agree.
- FRBCSimple refuses a schedule_duration that is not a whole number of resolution
  steps: FlexMeasures answers with whole steps only, so the step, read as the
  duration over the number of values, would come out wrong.
- The handler's storage efficiency is what remains after one step of its own
  resolution, and FlexMeasures reads it per step of the leakage sensor's
  resolution, so the two have to agree. That is now said where the sensor is
  taken and where the value is posted.

Tests cover each of these, and each of the four mutations the review found
surviving, among them an on/off heat pump told to switch off: both modes ask for
factor 0, and only the mode tells them apart.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…ed, and simulated

The shipped example sent no instruction at all, for two reasons that only show
with both halves running (findings A1, A2). Its resource manager sent its storage
status before its system description, which planned nothing, since planning hangs
off a storage status and needs a description to turn a schedule into
instructions; and its description was dated 2026-01-15, while the server seeds
prices from the day it starts, so a schedule's window had no prices.

It also could not have shown steering if it had: one operation mode, with a power
range of -0.5 to 0.5 W against a fill rate of -0.5 to 0.5 per second, which is
1.8 MW, over a declared range of 0.4.

The resource manager is now a heat pump keeping a hot-water buffer warm. It is
idle, or heats at anything from 400 to 1500 W, at a COP of 4 while the buffer is
below 6 kWh and 3 above it. It declares its leakage, warmer losing more; a day of
hot-water use with showers in the morning and washing up in the evening; and a
comfort profile that pre-heats before the showers. It sends its description first,
dated now, and then simulates its buffer from S2's own definitions, written
separately from the CEM's code: it follows each instruction from the moment it is
due, and reports its fill level every so often, which asks for a new schedule.
That simulation is a second opinion on the CEM's arithmetic, and a test holds the
two to agreement across an element boundary.

The server gives the handler a sensor for every declaration (fill rate, band,
usage, leakage, charging efficiency), so each can be seen next to the schedule
planned against it, and plans from now rather than a day after the description.
Its connection handling is split out of the aiohttp handler, so a test can serve
the shipped resource manager with the shipped CEM, over an in-memory websocket,
with only FlexMeasures stood in for.

Two library changes the example needed:
- A system description that arrives after the latest storage status, or that
  changes what the device can do, is planned for against that status (finding
  A3), rather than waiting for a status that a resource manager whose fill level
  holds steady may never send. It plans without re-posting the old status as a
  new measurement.
- A schedule never starts before now. Planning happens on every storage status
  while a description can be days old, so a schedule anchored to the description
  alone slid into the past, and its instructions all arrived as one overdue burst.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…xMeasures

FRBCSimple passed its energy_unit straight through as the flex model's soc-unit,
while FlexMeasures validates soc-unit with OneOf(["kWh", "MWh"]). So four of the
six units fill_rate_in_watts documents, J, kJ, MJ and Wh, made every trigger a
422, and the storage got no schedule at all (finding B2).

The plan for this was to refuse those units when the handler is made. The carve
source answers it better, by converting joules at the edge, and that is what
happens now, for every supported unit: the state of charge and its two hard
bounds go to FlexMeasures in kWh, converted from the recorded unit with the table
fill_rate_in_watts already keeps, while kWh and MWh pass through as they are.
The band, the usage and the leakage are read from sensors, which FlexMeasures
converts itself, and the bounds the schedule is read back against stay in the
recorded unit, since the operation modes are stated in it.

An energy unit outside that table is now refused when the handler is made, where
it used to surface only at planning time, as a logged failure to record a
charging efficiency. The handler for that failure is gone, since nothing can now
reach it.

Not changed, and now pinned by a test: the measured state-of-charge sensor is
never named in the flex model (finding B13 stands). Named there, FlexMeasures
would save the state of charge it plans onto that sensor, beside the
measurements. The carve source does name it, and the carve will have to leave
that line behind.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…nges

A second round of review showed that following a step a minute at a time traded
one class of error for four others, and cost ten times the time:

- Usage was sampled at the start of each minute. Half-minute draws starting on
  the half minute were never seen at all, and the same draws starting on the
  minute were counted twice.
- A level settling on an element boundary, where filling stops above it, flipped
  across that boundary every step: 127 instructions in 144 steps, against one.
- An element narrower than a minute's travel was skipped: a level that stops at
  50 was carried to 80.
- A 48-hour schedule against the largest forecast S2 allows took 1.7 s, on the
  event loop, scanning the forecast once a minute.

Every rate is constant between the moments the usage changes and the moments
the level reaches an element boundary, of the mode or of the leakage behaviour,
or one of its bounds. So follow_operation_mode now steps from one such moment
to the next and is exact. A level that reaches a boundary whose far side would
push it back stays at that boundary, as a real storage does, and so does a level
at one of its bounds, which rises again from there once filling wins. The usage
forecast is read through a UsageProfile, built once per schedule and looked up
by bisection.

Also from the second round:
- FillRateBasedControlTUNES posted its usage forecast as an amount per quarter
  hour labelled MW. FlexMeasures reads soc-usage as a power and multiplies it by
  the step's share of an hour, so it planned a quarter of the real draw. It now
  posts per hour, and reads back accordingly.
- FRBCSimple refuses a resolution or schedule duration that is not positive,
  where a zero resolution raised a ZeroDivisionError.
- The factor tolerance's justification was wrong: FlexMeasures rounds a
  schedule to six decimals of MW, so what it absorbs is the noise of converting
  to watts, not a solver's rounding. The tolerance stays; the comment and the
  changelog now say what it is for.
- One of the first round's mutations had stopped applying after black split
  its line, so the gate's count overstated what was checked. The harness now
  rejects a mutant that cannot run, and every mutation is anchored to the
  formatted code.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…, and what review found

An adversarial review of the example, and of the two library changes it needed,
found the example resource manager following every plan it had ever been sent.
Each instruction ran in a task of its own, nothing cancelled the tasks of a
replaced batch, and the server sent no revocations. In the live demo against a
sandboxed FlexMeasures, thirty instructions for the same quarter hour fired at
once when it came, one from each replan.

The server now sets send_revocations, so a newer plan revokes what is left of the
one before, and the resource manager cancels a revoked instruction it has not yet
followed, and acknowledges the revocation.

Also from the review:
- The leakage, the usage forecast and the comfort profile were sent once, at
  connect, and the storage efficiency was recorded once, for twelve hours from
  the message, so planning went without leakage after twelve hours and without
  usage or a band after a day. The resource manager now sends all three again
  every S2_EXAMPLE_FORECAST_INTERVAL seconds, from then on, and the server seeds
  its demo prices again every twelve hours.
- In the library, a storage efficiency is now recorded each time a schedule is
  asked for, over the window it covers and as a share of the fill level it starts
  from, and named in the flex model only where it was recorded. On arrival, it
  covered the wrong hours, at a fill level nobody had reported yet, and as a share
  of the top of the range it understated the loss at lower fill levels.
- align_to_resolution floored the local clock, which in the repeated hour of an
  autumn DST change took the first pass of that hour, so a schedule could start
  67 minutes in the past. It now floors the instant, in UTC.
- A state of charge is posted as a reading at an instant, where it was posted over
  the minute after it, which FlexMeasures stored as a forecast.
- The example declares what it does: transitions between its two modes, which S2
  lists as the ones possible; no PowerForecast, since it sends none; and a
  consumer of electricity rather than a storage of it.
- Claims corrected: a description that differs from the one before it, its
  validity included, asks for a schedule, not only one that "changes what the
  device can do"; and a schedule starts in the current step, which has begun, not
  "never before now".
- The test that held the example's buffer to the CEM's integrator compared two
  copies of one discretisation. The CEM is now exact and the buffer steps a
  minute at a time, so the test bounds their difference by a minute of the rate
  difference at the crossing, which is what two correct discretisations owe each
  other.

Tests cover everything the review's thirteen mutations found unpinned.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
- The example now tells the CEM where each instruction stands: accepted on
  arrival, succeeded once carried out, revoked once revoked. Without that every
  instruction stayed NEW, so each replan revoked the whole previous batch, the
  instructions already carried out included.
- The mode a transition reports leaving is the one the last carried-out
  instruction put the heat pump in. It was the mode the previous instruction to
  arrive asked for, which after a revocation the heat pump was never in.
- A storage efficiency takes the loss as a share of the start fill level or of
  the middle of the storage's range, whichever is larger. A share of a near-empty
  start left almost nothing per step, and overstated the loss many times over
  once the plan filled the storage.
- A state of charge is posted as an instant only on a sensor that records
  instants: FlexMeasures refuses a single value of no duration on a sensor with
  a resolution, so there it is posted over a minute, as before.
- align_to_resolution floors on the moment's own clock, which is the grid
  FlexMeasures floors to, with the pass of an autumn DST hour taken from the
  moment. Flooring in UTC put hourly steps half an hour off in zones that are,
  and daily steps on the wrong midnight.
- The example server's price reseeding survives a failed ingestion job, a wrong
  content type and a wrong API version, not only a lost connection.
- Statuses about different steps that interleave within a second get belief
  times a second apart per step, where only the latest status was compared. The
  demo gate's first run had thirty statuses for one quarter hour arrive at once,
  and one was refused as a replacement.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
- In the repeated hour of an autumn DST change, a boundary is the later of the
  two candidates that is not after the moment. Taking the pass from the moment
  put 03:10 at two-hourly steps on 02:00 of the first pass, three hours back.
  The candidates are compared as pandas timestamps, which compare instants.
- The changelog says what the floor does: on the clock of the moment, in the
  offset it carries, which is how FlexMeasures floors a posted series. "The
  site's own clock" was not true for series a resource manager anchors in UTC.
- A state-of-charge sensor stating a resolution that is not a duration is
  posted over a minute and logged, where it aborted the status before planning.
  The docstring says the sensor is asked until it answers, which it is.
- An instruction that has become due is no longer pending, so a revocation
  arriving while it is carried out is not answered as REVOKED for an
  instruction the heat pump is following.
- The comment on the storage efficiency's reference says that below the middle
  of the range the loss is understated, and by how much at most.
- Tests pin the forgetting of belief times more than a day old.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…er way

The demo gate's second run, against a sandboxed FlexMeasures, heated the buffer
in the quarter hour the plan chose, but every replan inside that quarter
replanned the same quarter, at 0.72 kW, then 0.67, 0.63 and on down to 0.51 kW.

A schedule asked for mid-step started at the boundary that step began at, while
it was planned from the fill level reported now, which FlexMeasures takes to be
the level at the schedule's start. So each replan counted on the whole step
where only part of it was left, asked for less power over what remained, and
the step delivered less than any plan meant it to.

A schedule now starts at the next boundary, or at now where now is one. The
step under way keeps the instruction it is already following, which a newer
plan does not revoke, since it has been carried out.

The end-to-end test stands the CEM's clock on a boundary, so that its schedules
start there and the first instruction is due within the test.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Black had split the expression around the trailing comment after the merge.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Flix6x and others added 6 commits November 9, 2026 16:17
…cted request

FlexMeasures merges the asset's stored flex context in at the top level of
a request, and counts inflexible devices from there as well as from the
electricity entry, to which the request also carries them. So each was
counted twice: a house's planned aggregate power came out at twice its
inflexible load, and a community planner saw a breach that was never
realized (FlexMeasures/flexmeasures#2848).

The request now names the same inflexible-device fields empty at the top
level, which keeps FlexMeasures from merging the stored ones in there.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
S2 allows a negative expected usage, a gain. It was recorded as zero, which
hid what the resource manager stated, to keep plans feasible without a
usage correction. Recording and planning are separate questions: a
schedule request now carries the latest forecast, without its negative
values where no correction plans them, so the usage sensor can hold what
was stated. A FlexMeasures server before v0.23.0 takes a usage only as a
sensor, so there it is still recorded as zero.

Until a forecast arrives, a handler plans without usage, since the sensor
may hold negative usage recorded before it started.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
…escription

After a reconnection, a resource manager can send its actuator status
before its system description. Only a description says what power an
operation mode stands for, so such a status was logged and skipped, and
the power sensor showed nothing until the next status. It is now kept,
the latest per actuator, and recorded once a description arrives.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
… all operation modes

As asked in the review of #283.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
S2 states leakage as a rate per fill-level range. It was planned as a
storage efficiency, which FlexMeasures takes as a share of the stock, so the
storage leaked in proportion to how full it was, exact at one fill level
only. The leakage rate of the range the storage starts in is now added to
the usage, over the whole window, and sent with the schedule request. With
a usage correction, the correction covers usage and leakage together, so an
empty storage's leakage can be taken away like any other forced usage.

The leakage sensor records that rate, in the fill-rate unit, and the site
sensor is now called "leakage". It needs FlexMeasures v0.23.0, which takes a
usage with the request; an older server is planned without leakage.

As asked in the review of #283.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Flix6x and others added 8 commits November 9, 2026 17:41
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
S2 states these rates per second, so in kWh/s a heat buffer's leakage of some tens of watts reads about 0.00001.
They are now recorded in the power unit that goes with the energy unit (kW for kWh, W for J and Wh, MW for MJ and MWh), and the site's fill rate, usage forecast and leakage sensors are created in kW.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Like the energy and power units, the unit that rates of fill level are recorded in is the deployment's to set: W (the default), kW or MW.
Watts read a residential heat buffer's leakage of some tens of watts as such; the site's fill rate, usage forecast and leakage sensors are created in W.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Every power and rate of fill level the CEM sets up is now in W, the handler's default: power, measured-power and usage correction joined fill rate, usage forecast and leakage.
Watts read a residential site's small powers as such; a larger site passes its own sensors.
An existing site keeps its sensors, and one in another unit than a new site would get is logged as a warning.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
FlexMeasures converts what is posted into a sensor's own unit, so an older site's sensor in kW is a valid choice, not a fault.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
@Flix6x

Flix6x commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

Co-simulation results for this PR: two simulated homes with heat pumps, one day, steered as a community under a 2 kW limit. Each run uses deterministic solver settings.

Run Client Usage correction Breach (kWh) Peak (kW) Hours over the limit Fallbacks
C1 #279 off 0.058 2.21 0.5 0
C2 a31f1cf off 0.052 2.21 0.25 0
D7 a31f1cf on 0.745 2.55 3.5 0
D7b a31f1cf on (repeat of D7) 0.778 2.55 3.75 0

Comfort was kept in every run, and no run fell back to its last plan.

What this shows:

  • With the correction off, this PR behaves like fix(S2): the remaining findings (2 of 2) #279 (C2 vs C1). Its other changes, such as sending the usage with the request, do not change the outcome.
  • The correction makes the community limit hold worse, and this is not noise (D7 and D7b agree). The correction device itself is never used: all its planned values are zero. In the evening, the homes' own plans run both heat pumps at once, while the community steering reports a planned residual of 0.21 kW, as in C1.
  • Replaying one evening request in FlexMeasures gives about the same plan in every form tried. It was sent in the correction form, then without the correction device, then flattened as with the correction off. It was also replayed with FlexMeasures' default band breach price, and with usage clipped at zero. So a single home solves the same steering signals the same way either way. The difference arises over the community's iterations, in the steering the homes get back.
  • Each home's stored inflexible devices are now counted once (a31f1cf; replay of D7's last request). The planned residual went from 2.419 kW (D6) back to 0.21 kW.

Next: find why the community steering converges to signals that leave the homes' plans over the limit when the correction is on. Until then, the correction (opt-in) should not be switched on for steered communities.

Flix6x and others added 2 commits November 10, 2026 00:09
S2 states fill rates and usage per second, which in kWh/s puts a heat buffer's rates near 1e-5. Ported from #289 without its leakage-as-usage work, which stays with #286.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Consistent with the handler's defaults, and readable at residential scale. Sensors an existing site already has are reused by name and keep their units. Ported from #289.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
Base automatically changed from dev/s2-frbc-complete to main November 9, 2026 23:22
Flix6x added a commit that referenced this pull request Nov 9, 2026
#279)

Second of two PRs that finish the S2 FRBC Customer Energy Manager: #271 to #277, #283 to #285, #287, #288 and #290. The usage correction (#286, with #289) is held back.

Signed-off-by: F.N. Claessen <[email protected]>
@Flix6x
Flix6x changed the base branch from main to dev/s2-frbc-complete November 9, 2026 23:23
@Flix6x

Flix6x commented Nov 10, 2026

Copy link
Copy Markdown
Member Author

The steering issue is resolved: the cause was in the co-simulation's community controller, not in this PR. Lifting the hold.

Not a lack of rounds. In D7, every step reached a fixpoint, with no house re-planning, well inside the round cap.

The controller read stale plans.

  • With the correction on, each scheduling request differs from the previous one, since the correction's capacity and price change per job. FlexMeasures records each house's plan under a new scheduler data source whenever the request differs, so D7 left 7 sources per house over the evening; C2 left 1.
  • The controller took the latest belief per source, then combined the sources oldest first. It therefore steered against each house's first-round plan.
  • Rebuilt from the database for D7's evening: the controller saw a peak of 2.21 kW (0.21 kW over), while the houses' latest plans peaked at 2.55 kW and sat at 2.2 to 2.4 kW from 19:30 to 20:30. That is the 0.21 kW "unavoidable" residual it reported, and the realized breach.

With the controller fixed (per slot, the latest belief across scheduler sources wins), the same run:

Run Breach (kWh) Peak (kW) Hours over the limit Comfort violations
C2 (correction off) 0.052 2.21 0.25 0
D7 (correction on) 0.745 2.55 3.5 0
D7c (correction on, controller fixed) 0.052 2.21 0.25 0

No fallbacks in D7c. This PR needs no change. Anything else that reads a site's schedule should resolve the latest belief across scheduler sources too, rather than rank sources.

Flix6x and others added 2 commits November 10, 2026 19:13
…ection

#290 carried #289's unit changes on their own, so its conflicts resolve to this branch's side, which already has them along with the leakage as usage. #287's handling of an early actuator status and #288's changelog wording are kept, and the changelog links #290 next to #289.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
main holds the integration branch squashed (#279), with the same content, so this changes no file. It lets this PR target main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CcpawsJs74cSwFLXkz7ZwW
Signed-off-by: F.N. Claessen <[email protected]>
@Flix6x
Flix6x changed the base branch from dev/s2-frbc-complete to main November 10, 2026 18:13
@Flix6x
Flix6x merged commit 64cc634 into main Nov 10, 2026
20 of 23 checks passed
@Flix6x
Flix6x deleted the feat/s2-usage-correction branch November 10, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants