Repository navigation
Feat/automations - #5
TeaDrinkingProgrammer wants to merge 9 commits into
Conversation
Moving polling schedules to FlexMeasures automations breaks existing setups. Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
FlexMeasures main moved to psycopg 3, after which deleting a polling schedule's sensors failed in jsonb_path_exists. Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
FlexMeasures now schedules, queues and tracks jobs for plugin-defined automation types, so the plugin no longer needs a cron scheduler of its own. BREAKING CHANGE: Replace the oadr3-run-cron-scheduler process with flexmeasures jobs run-automations, and recreate existing polling schedules, as they are not migrated. Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
Signed-off-by: Stijn van Houwelingen <[email protected]>
6a3f787 to
4e14a1c
Compare
BelhsanHmida
left a comment
There was a problem hiding this comment.
Looks good! one automation per polling schedule is the right mapping. Keep the sensor ids in the config, FlexMeasures uses them to check what a run may write.
I ran the walkthrough locally, and with the mount below and FlexMeasures v1.1.0rc6, polling works end to end.
Some things I found below, the first two should be fixed before merging, and the third once 1.1 is out.
Nit: the Fetched DR events log line is gone with ven_jobs.py, in the walkthrough README (line 178) and the lf-energy-demo one (line 488).
|
|
||
| def delete_sensors_for_config(self, asset: GenericAsset, config_name: str) -> None: | ||
| """Delete both import and export sensors associated with a config name.""" | ||
| def delete_sensors_for_config(self, asset: GenericAsset, config_name: str, sensors: VenSensorPair) -> None: |
There was a problem hiding this comment.
Locally, renaming demo-poll to site-poll, adding a new demo-poll and deleting it removed both sensors and all their data, and site-poll then failed every minute.
A renamed schedule keeps sensors with the old name, so the new one picks up the same sensors.
Could we rename the sensors along with the schedule? That fixed it locally.
| ) | ||
| config = _fetch_events_config(form_data.targets, sensors) | ||
| if config != automation.generator.attributes.get("data_generator", {}).get("config"): | ||
| automation.generator = _fetch_events_source(ven_client.asset, config) |
There was a problem hiding this comment.
Fine for now, FlexMeasures/flexmeasures#2779 adds editing an automation's config, so this can move to that once it's merged.
| """Turn the payloads of OpenADR events into beliefs, per sensor they are to be recorded on.""" | ||
| beliefs: dict[Sensor, list[tb.TimedBelief]] = {} | ||
| for event in events: | ||
| belief_time = _as_utc(event.modification_date_time) |
There was a problem hiding this comment.
Nit: this relies on the VTN changing modificationDateTime on every event change. OpenLEADR does, I checked.
With a mocked event that keeps the same time, a new value fails with a UniqueViolation on every poll, where db.session.merge used to overwrite it. Is that okay to rely on?
| ] | ||
|
|
||
| # UV-specific configuration | ||
| [tool.uv.sources] |
There was a problem hiding this comment.
The locked commit is from before FlexMeasures/flexmeasures#2670, so with uv sync plugin runs stay pending, even failed ones.
With 1.1 out in a couple of days, could we drop [tool.uv.sources] then, depend on flexmeasures>=1.1.0 and update uv.lock?
The Dockerfile can then start from the 1.1.0 image, without the git install and the rm -rf. The "not in a release yet" notes here, in the Dockerfile and in the README can go too.
There was a problem hiding this comment.
FlexMeasures 1.1.0 is out, so the pyproject.toml change can go ahead. I tried it locally with flexmeasures>=1.1.0, no [tool.uv.sources] and the v1.1.0 image: polling works, with the mount from the Dockerfile comment still needed.
One thing changed since v1.1.0rc6: a plugin job's result is now {"outputs": [{"sensor": ..., "num-computed-beliefs": ..., "num-recorded-beliefs": ...}]} instead of a list, so trigger_fetch.py (in both examples) crashes on line 74 and test_ven_fetch_e2e.py fails on lines 105 and 153.
Reading job.return_value()["outputs"] with those keys fixed all three locally.
| # Install the plugin dependencies INTO the existing uv environment. The base image's own copy of the | ||
| # FlexMeasures source in /app would shadow the FlexMeasures this installs (/app is the working directory), | ||
| # so it is removed. | ||
| RUN uv pip install --python /app/.venv/bin/python /app/plugins/flexmeasures-openadr3 && \ |
There was a problem hiding this comment.
This install makes FlexMeasures run from the venv, so its instance folder moves from /app/instance to /app/.venv/var/flexmeasures-instance and the mounted flexmeasures.cfg is never read.
Locally, creating a VEN client then gives a 500 in all three compose stacks: "No FLEXMEASURES_SECRETS_ENCRYPTION_KEYS set".
Mounting the folder there too fixed it, and the v1.1.0rc6 image needs that as well. Could we add that for every service with these mounts?
This PR moves the cron job polling logic to the new FlexMeasures Automations feature. It is not yet in a build of flexmeasures, so for now, this PR is based on main.
What changes
openadr3-fetch-eventson the VEN client's asset. Its config holds thetargetsand theimport-sensor/export-sensorids. OAuth credentials stay inasset.secrets.OpenADR 3 VTNdata source, now owned by the VEN asset's account. Fetching unchanged events again saves nothing new.oadr3-run-cron-schedulercommand, rq.cron and the pub/sub resync. Thecron-schedulercompose service is replaced by anautomation-runnerthat callsflexmeasures jobs run-automationsonce a minute.Editing what a schedule fetches (targets or limit types) gives its automation a new data source, as FlexMeasures can't change an automation's config in place. The automation keeps its id and run history, and a renamed schedule keeps its sensors.
Breaking changes
Existing polling schedules are not migrated: recreate them after upgrading (their sensors are reused when the name stays the same). This is why the version goes to 2.0. I left out a migration because the plugin is still alpha; if anyone runs it with real schedules, let me know and I'll add a CLI command for it.
cron-schedulerprocess withflexmeasures jobs run-automationsevery minute. Theingestionworker now needs the plugin inFLEXMEASURES_PLUGINStoo.pyproject.tomlpins FlexMeasures tomainvia[tool.uv.sources]. The Dockerfile starts fromv1.1.0rc2and installsmainon top (this needs git, and the image's own/app/flexmeasurescopy has to go, as it shadowed the installed package). Once 1.1.0 is out, this becomes a plainflexmeasures>=1.1.0.Also in here
jsonb_path_existsqueries used when deleting a schedule's sensors. The JSON paths are now cast explicitly..dockerignore, as the build context included.venv.Testing
trigger_fetch.py, compose files, walkthroughs) are updated, but I haven't run the walkthroughs end to end yet.Feedback on the schedule-to-automation mapping is the most welcome, especially storing the sensor ids in the automation's config. That's the part I'm least sure about.