Skip to content

Feat/automations - #5

Open
TeaDrinkingProgrammer wants to merge 9 commits into
feat/lfenergy-demo-examplefrom
feat/automations
Open

TeaDrinkingProgrammer wants to merge 9 commits into
feat/lfenergy-demo-examplefrom
feat/automations

Conversation

@TeaDrinkingProgrammer

@TeaDrinkingProgrammer TeaDrinkingProgrammer commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

This PR moves the cron job polling logic to the new FlexMeasures Automations feature. It is not yet in a build of flexmeasures, so for now, this PR is based on main.

What changes

  • Polling schedule = automation: each polling schedule is an automation of type openadr3-fetch-events on the VEN client's asset. Its config holds the targets and the import-sensor/export-sensor ids. OAuth credentials stay in asset.secrets.
  • FlexMeasures saves the data: the fetcher returns the beliefs and FlexMeasures stores them under the OpenADR 3 VTN data source, now owned by the VEN asset's account. Fetching unchanged events again saves nothing new.
  • Cron and timezone instead of a daily UTC time: the schedule form now matches FlexMeasures' own automation form (five-field cron expression, timezone suggestions, defaulting to the asset's timezone). Times from the VTN are always stored in UTC.
  • Run now, (de)activate and run history: these live on the asset's FlexMeasures Automations page, which the polling schedules page links to.
  • Gone: the oadr3-run-cron-scheduler command, rq.cron and the pub/sub resync. The cron-scheduler compose service is replaced by an automation-runner that calls flexmeasures jobs run-automations once a minute.

Editing what a schedule fetches (targets or limit types) gives its automation a new data source, as FlexMeasures can't change an automation's config in place. The automation keeps its id and run history, and a renamed schedule keeps its sensors.

Breaking changes

Existing polling schedules are not migrated: recreate them after upgrading (their sensors are reused when the name stays the same). This is why the version goes to 2.0. I left out a migration because the plugin is still alpha; if anyone runs it with real schedules, let me know and I'll add a CLI command for it.

  • Deployments: replace the cron-scheduler process with flexmeasures jobs run-automations every minute. The ingestion worker now needs the plugin in FLEXMEASURES_PLUGINS too.
  • FlexMeasures main: plugin automations aren't in a release yet, so pyproject.toml pins FlexMeasures to main via [tool.uv.sources]. The Dockerfile starts from v1.1.0rc2 and installs main on top (this needs git, and the image's own /app/flexmeasures copy has to go, as it shadowed the installed package). Once 1.1.0 is out, this becomes a plain flexmeasures>=1.1.0.

Also in here

  • FlexMeasures main moved to psycopg 3, which broke the jsonb_path_exists queries used when deleting a schedule's sensors. The JSON paths are now cast explicitly.
  • FlexMeasures' test conftest now drops its schema after our Postgres testcontainer had stopped. The container now stops after that cleanup.
  • A .dockerignore, as the build context included .venv.

Testing

  • All tests pass: repository, automation, schedule form and UTC tests, plus the end-to-end test. That one fetches from an OpenLEADR VTN behind Keycloak, runs the automation through a worker and checks that a second run saves nothing new.
  • The examples (trigger_fetch.py, compose files, walkthroughs) are updated, but I haven't run the walkthroughs end to end yet.

Feedback on the schedule-to-automation mapping is the most welcome, especially storing the sensor ids in the automation's config. That's the part I'm least sure about.

Moving polling schedules to FlexMeasures automations breaks existing setups.

Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
FlexMeasures main moved to psycopg 3, after which deleting a polling schedule's sensors failed in jsonb_path_exists.

Signed-off-by: Stijn van Houwelingen <[email protected]>
FlexMeasures now schedules, queues and tracks jobs for plugin-defined automation types, so the plugin no longer needs a cron scheduler of its own.

BREAKING CHANGE: Replace the oadr3-run-cron-scheduler process with flexmeasures jobs run-automations, and recreate existing polling schedules, as they are not migrated.

Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>

@BelhsanHmida BelhsanHmida left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! one automation per polling schedule is the right mapping. Keep the sensor ids in the config, FlexMeasures uses them to check what a run may write.
I ran the walkthrough locally, and with the mount below and FlexMeasures v1.1.0rc6, polling works end to end.
Some things I found below, the first two should be fixed before merging, and the third once 1.1 is out.
Nit: the Fetched DR events log line is gone with ven_jobs.py, in the walkthrough README (line 178) and the lf-energy-demo one (line 488).


def delete_sensors_for_config(self, asset: GenericAsset, config_name: str) -> None:
"""Delete both import and export sensors associated with a config name."""
def delete_sensors_for_config(self, asset: GenericAsset, config_name: str, sensors: VenSensorPair) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Locally, renaming demo-poll to site-poll, adding a new demo-poll and deleting it removed both sensors and all their data, and site-poll then failed every minute.
A renamed schedule keeps sensors with the old name, so the new one picks up the same sensors.
Could we rename the sensors along with the schedule? That fixed it locally.

)
config = _fetch_events_config(form_data.targets, sensors)
if config != automation.generator.attributes.get("data_generator", {}).get("config"):
automation.generator = _fetch_events_source(ven_client.asset, config)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine for now, FlexMeasures/flexmeasures#2779 adds editing an automation's config, so this can move to that once it's merged.

"""Turn the payloads of OpenADR events into beliefs, per sensor they are to be recorded on."""
beliefs: dict[Sensor, list[tb.TimedBelief]] = {}
for event in events:
belief_time = _as_utc(event.modification_date_time)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this relies on the VTN changing modificationDateTime on every event change. OpenLEADR does, I checked.
With a mocked event that keeps the same time, a new value fails with a UniqueViolation on every poll, where db.session.merge used to overwrite it. Is that okay to rely on?

Comment thread pyproject.toml
]

# UV-specific configuration
[tool.uv.sources]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The locked commit is from before FlexMeasures/flexmeasures#2670, so with uv sync plugin runs stay pending, even failed ones.
With 1.1 out in a couple of days, could we drop [tool.uv.sources] then, depend on flexmeasures>=1.1.0 and update uv.lock?
The Dockerfile can then start from the 1.1.0 image, without the git install and the rm -rf. The "not in a release yet" notes here, in the Dockerfile and in the README can go too.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FlexMeasures 1.1.0 is out, so the pyproject.toml change can go ahead. I tried it locally with flexmeasures>=1.1.0, no [tool.uv.sources] and the v1.1.0 image: polling works, with the mount from the Dockerfile comment still needed.
One thing changed since v1.1.0rc6: a plugin job's result is now {"outputs": [{"sensor": ..., "num-computed-beliefs": ..., "num-recorded-beliefs": ...}]} instead of a list, so trigger_fetch.py (in both examples) crashes on line 74 and test_ven_fetch_e2e.py fails on lines 105 and 153.
Reading job.return_value()["outputs"] with those keys fixed all three locally.

Comment thread Dockerfile
# Install the plugin dependencies INTO the existing uv environment. The base image's own copy of the
# FlexMeasures source in /app would shadow the FlexMeasures this installs (/app is the working directory),
# so it is removed.
RUN uv pip install --python /app/.venv/bin/python /app/plugins/flexmeasures-openadr3 && \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This install makes FlexMeasures run from the venv, so its instance folder moves from /app/instance to /app/.venv/var/flexmeasures-instance and the mounted flexmeasures.cfg is never read.
Locally, creating a VEN client then gives a 500 in all three compose stacks: "No FLEXMEASURES_SECRETS_ENCRYPTION_KEYS set".
Mounting the folder there too fixed it, and the v1.1.0rc6 image needs that as well. Could we add that for every service with these mounts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants