diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index ac97731fc4..8e01db865f 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -8,7 +8,14 @@ API change log v3.0-35 | November xx, 2026 """""""""""""""""""""""""""" +**Automations** + - Renamed ``schedule-revision`` to ``revision`` in automation responses and in each run listed under ``run-stats``. The field counts the resets of an automation's recurrence, not anything about schedules; it was released under the old name in v3.0-34 (FlexMeasures v1.1.0), and is read-only, so only clients reading it need to switch. +- ``POST /api/v3_0/assets//automations`` and ``PATCH /api/v3_0/assets//automations/`` accept ``depends-on``, a list like ``[{"automation": 7, "allow-failure": false}]``, to have an automation's runs wait for the latest runs of other automations due at or before their own scheduled time, and automation responses report it, each upstream automation the caller may read coming with its ``name``, ``asset`` and ``asset-name``. +- ``POST /api/v3_0/assets//automations//trigger`` accepts ``with-upstream=true``, to first run the active automations this one waits for, or ``with-downstream=true``, to also run the active automations that wait for this one, in chain order, and ``exclude`` to exclude some of them; the response lists the runs under ``runs`` and the automations that do not run under ``excluded``, each with the reason. +- Added ``GET /api/v3_0/assets//automations//run-plan?include=upstream|downstream``, to see what running an automation along with what is upstream or downstream of it comes down to. +- Added ``GET /api/v3_0/assets/automations``, to list the automations on public assets (``scope=public``) or every automation the caller may read (``scope=all``), all at once, or one page at a time with ``page`` and ``per-page``. +- ``GET /api/v3_0/assets//automations/?include=waited-for-by`` reports ``waited-for-by``, the automations that wait for this one. v3.0-34 | November 05, 2026 """""""""""""""""""""""""""" diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 9f0ccb3aa9..f9463d6d8c 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -11,6 +11,7 @@ v1.2.0 | December xx, 2026 New features ------------- +* An automation can wait for the automations whose results it uses: each of its runs waits for their latest runs due at or before its own scheduled time, also on another schedule, and is skipped if one of them failed, from the CLI (``--depends-on``), the API (``depends-on``) and the *Automations* page, and a chain can be re-run on demand from an automation (``--with-downstream``) or up to it (``--with-upstream``), which the *Automations* page also draws [see `PR #2706 `_] Infrastructure / Support ------------------------- diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index 68cc8cf7ae..6f51589ce4 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -4,6 +4,15 @@ FlexMeasures CLI Changelog ************************** +since v1.2.0 | December XX, 2026 +================================= + +**Automations** + +* Add ``--depends-on`` and ``--allow-failure-of`` to ``flexmeasures add automation`` and ``flexmeasures edit automation`` (plus ``--no-dependencies`` on edit), to have an automation's runs wait for the latest runs of other automations due at or before their own scheduled time. +* Add ``--with-downstream`` to ``flexmeasures jobs run-automation``, to also run the automations that wait for it, each once the runs it waits for ended, for instance to re-run a chain after a backfill. +* Add ``--with-upstream`` to ``flexmeasures jobs run-automation``, to first run the automations it waits for, and ``--exclude`` to exclude one of them, along with what can only be reached through it. + since v1.1.0 | November 05, 2026 ================================= diff --git a/documentation/features/automations.rst b/documentation/features/automations.rst index 03df6cf34a..09e202db49 100644 --- a/documentation/features/automations.rst +++ b/documentation/features/automations.rst @@ -224,6 +224,81 @@ It runs every night at 1 AM, an hour after midnight, so that the day's last read .. _running_automations: +.. _automation_dependencies: + +Waiting for other automations +----------------------------- + +Some automations use what another automation produces. +For example, a report that computes flows from a stock is only correct once another report has snapped the raw meter readings of that stock to a regular grid. +Rather than staggering their cron strings and hoping the first one finishes in time, let the second automation wait for the first: + +.. code-block:: bash + + flexmeasures add automation --asset 3 --name "Flows" --cron "*/5 * * * *" --timezone UTC \ + --type reporting --reporter PandasReporter --config flows-config.yml --parameters flows-parameters.yml \ + --depends-on 7 + +Each run of the waiting automation then waits for the latest run of automation 7 that is due at or before its own scheduled time. +It queues its jobs as soon as that run has ended successfully, and is *skipped* if it failed, so that it never computes on input that was never produced. +A skipped run says which upstream run it was skipped for, and anything waiting for a skipped run is skipped, too. +A job that gets killed cannot report how it ended, so the runner looks that up in Redis on each tick: its run counts as failed, rather than leaving what waits for it waiting forever. +To have a run go ahead even when the run it waits for failed, also pass ``--allow-failure-of 7``. + +An automation can wait for several others (repeat ``--depends-on``), and these can be on any asset you can read. +They may recur on another schedule, as each run waits for the latest upstream run due at or before its own scheduled time, by the upstream automation's own cron string and timezone. +For instance, an hourly forecast that waits for meter data pulled every 15 minutes runs at 10:00 once the pull due at 10:00 ended. +The other way around, a forecast every 15 minutes that waits for an hourly pull runs at 10:15, 10:30 and 10:45 on the pull due at 10:00, so a failure of that one pull has all three of them skipped. +A dependency that would have an automation end up waiting for itself is refused. + +Some cases where the upstream run does not happen: + +- If the upstream automation missed that time while the runner was down, its missed runs were coalesced into the run it caught up with, so the waiting run waits for that one instead. +- If the upstream automation was never going to run at that time, because it is inactive, did not exist yet, or its recurrence was reset since (by reactivating it, or by giving it another cron string or timezone), the waiting run goes ahead without it. Its run record says so. +- If the upstream automation moved past that time without any run, the waiting run is skipped (unless failure is allowed for). + +Deactivating an automation cancels its scheduled runs that are still waiting, rather than having them go ahead late once it is reactivated, and what waits for them goes ahead without them. + +The API takes the same as ``depends-on``, a list like ``[{"automation": 7, "allow-failure": false}]``, and on the asset's *Automations* page the creation and edit forms let you pick the automations to wait for. +By default, you pick among those listed on the page, which are those on this asset and, if you include them, on the assets below it, and you can switch to this asset only, to public assets, or to every automation you can see. +The chain in an automation's details follows the same choice, and can show all of these automations rather than only those linked to this one. +Deleting an automation others wait for warns you first, as they then run without it. + +When assets are copied along with their automations, a copy waits for the copy of the automation its original waited for, if that was copied too, and for the original one otherwise. + +Runs always flow downstream, from the automations that are waited for to those that wait for them. +To re-run a whole chain, for instance once the data at its start was backfilled, run its first automation along with what is downstream of it: + +.. code-block:: bash + + flexmeasures jobs run-automation --automation 7 --with-downstream + +Automation 7 runs right away, and each active automation that waits for it, directly or through others, gets a run that waits for the runs started along with it, +and is skipped if one of those failed, just like scheduled runs. +These runs on demand do not affect the automations' recurrence, and the runner does not retry them if their jobs failed to queue, as whoever started them is told. + +To bring one automation up to date instead, run what is upstream of it first, and stop at this automation: + +.. code-block:: bash + + flexmeasures jobs run-automation --automation 9 --with-upstream + +The automations it waits for, directly or through others, run first, and each run waits for the runs started along with it, as above. +What waits for automation 9 does not run. + +Either way, the included automations run in chain order. +With several branches, all of them run, unless you exclude one with ``--exclude``. +An automation that does not run, because you excluded it, because it is inactive, or because you may not run it, such as one on a public asset, is excluded along with what can only be reached through it. +It still runs on its own schedule, just not because of this request, and what waits for it goes by the results it already has. +The API does the same with ``POST /api/v3_0/assets//automations//trigger?with-upstream=true`` (or ``with-downstream=true``) and ``exclude``, +and ``GET /api/v3_0/assets//automations//run-plan?include=upstream`` (or ``include=downstream``) tells what that would come down to. +On the *Automations* page, *Run up to here* and *Run from here* show this before anything runs. + +The details of an automation on the *Automations* page draw the chain it is part of, among the automations listed there: +what it waits for on the left, what waits for it on the right, and a dashed arrow where the waiting automation runs even when the one it waits for failed. +Hovering over an automation there tells its type, asset and recurrence, hovering over an arrow tells what a failure does, and clicking an automation opens its details, on the page of its own asset. +An automation that is not listed there, but that you may read, is named and can be clicked as well. + Running automations ------------------- @@ -280,7 +355,8 @@ a report run its reporting job, and a plugin run the job its plugin queued. A run which queued no jobs, such as a report run with nothing new to report on, has succeeded. A job which a schedule queues to fall back on a simpler scheduler is not one of the run's jobs, so a run whose schedule fell back has failed, with the error that made it fall back. -A run tracks two things separately: how far its *dispatch* got (``pending``, ``claimed``, ``partially_queued``, ``queued`` or ``failed``), and how its *execution* by the workers ended (``pending``, ``running``, ``succeeded``, ``failed`` or ``canceled``). +A run tracks two things separately: how far its *dispatch* got (``pending``, ``claimed``, ``waiting``, ``partially_queued``, ``queued`` or ``failed``), and how its *execution* by the workers ended (``pending``, ``running``, ``succeeded``, ``failed``, ``canceled`` or ``skipped``). +A run is ``waiting`` while the runs it depends on have not ended yet, and ``skipped`` if one of them failed (see :ref:`automation_dependencies`). Each attempt to dispatch a run is recorded too, with the runner which made it, what it queued, and why it failed if it did. This is what an operator needs to tell a run which failed before queueing anything, one which queued half its work, and one which queued everything but then failed while computing, apart from each other. @@ -309,6 +385,7 @@ That menu also lets you edit, copy, deactivate (or activate) and delete the auto The automation runs with the parameters it was created with, and the jobs it queues are recorded as its jobs, just like the jobs of a recurring run. An on-demand run does not affect the automation's recurrence: its cursor (see :ref:`automation_cursor`) stays where it was, so the next recurring run still happens as scheduled, and a run missed while the runner was down is still caught up. Inactive automations can be run this way, too, which is how you can try one out before activating it. +An on-demand run does not wait for the automations this one depends on, as it is meant to run now. Unlike a recurring run, an on-demand run is not protected against being started twice: asking for two runs in a row queues two runs. diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py index 39940cde22..cc93039699 100644 --- a/flexmeasures/api/v3_0/assets.py +++ b/flexmeasures/api/v3_0/assets.py @@ -64,8 +64,13 @@ get_asset_automations_job_stats, get_automation_job_stats, get_automation_run_stats, + get_automations_in_scope, + get_automations_waiting_for, + CHAIN_INCLUDES, + plan_automation_chain, resolve_automation_sensors, run_automation, + run_automation_chain, update_automation, ) from flexmeasures.data.models.charts import ASSET_CHART_BUILDERS @@ -123,6 +128,26 @@ asset_schema = AssetSchema() annotation_schema = AnnotationSchema() automation_schema = AutomationSchema() + +# Listing automations across assets has no asset to bound it, so it is capped. + + +def _user_has_access(asset: GenericAsset, permission: str) -> bool: + try: + check_access(asset, permission) + except (Forbidden, Unauthorized): + return False + return True + + +def _may_read_automation(automation: Automation) -> bool: + return _user_has_access(automation.asset, "read") + + +def _may_manage_automation(automation: Automation) -> bool: + return _user_has_access(automation.asset, "manage-automations") + + # creating this once to avoid recreating it on every request default_list_assets_schema = AssetSchema(many=True, only=default_response_fields) sensor_schema = SensorSchema() @@ -1608,6 +1633,104 @@ def get_automations( "redis-connection-err": redis_connection_err, }, 200 + @route("/automations", methods=["GET"]) + @use_kwargs( + { + "scope": fields.Str( + required=True, + validate=validate.OneOf(["public", "all"]), + metadata={ + "description": "`public` for the automations on public assets, or `all` for every automation you may read." + }, + ), + "page": fields.Int(required=False, validate=validate.Range(min=1)), + "per_page": fields.Int( + data_key="per-page", + required=False, + validate=validate.Range(min=1), + load_default=10, + ), + }, + location="query", + ) + @as_json + def get_automations_across_assets( + self, scope: str, page: int | None = None, per_page: int = 10 + ): + """ + .. :quickref: Assets; List automations across assets. + + --- + get: + summary: List automations across assets. + description: | + List the automations on public assets, or every automation you may read, such as to pick one to wait for. + Job counts are left out, as those are looked up per asset. + The automations are listed by ID. + Without `page`, all of them are listed. + Pass `page`, and optionally `per-page` (default 10), to list one page of them, along with `num-records`, how many there are in all. + security: + - ApiKeyAuth: [] + parameters: + - in: query + name: scope + required: true + description: "`public` for the automations on public assets, or `all` for every automation you may read." + schema: + type: string + enum: [public, all] + - in: query + name: page + required: false + description: The page to list, starting at 1. + schema: + type: integer + - in: query + name: per-page + required: false + description: How many automations to list per page. + schema: + type: integer + default: 10 + responses: + 200: + description: PROCESSED + 401: + description: UNAUTHORIZED + 422: + description: UNPROCESSABLE_ENTITY + tags: + - Assets + """ + # Read access is decided per asset, so what may not be read is dropped before paging, + # and each page holds as many automations as asked for while there are more. + readable_assets: dict[int, bool] = {} + readable = [] + for automation in get_automations_in_scope( + scope, [account.id for account in get_accessible_accounts()] + ): + if automation.asset_id not in readable_assets: + readable_assets[automation.asset_id] = _may_read_automation(automation) + if readable_assets[automation.asset_id]: + readable.append(automation) + listed = ( + readable + if page is None + else readable[(page - 1) * per_page : page * per_page] # noqa: E203 + ) + automations_data = [] + for automation in listed: + automation_data = automation_schema.dump(automation) + automation_data["recurrence-description"] = describe_cronstr( + automation.cronstr + ) + automation_data["asset-name"] = automation.asset.name + automations_data.append(automation_data) + response: dict = {"automations": automations_data} + if page is not None: + response["num-records"] = len(readable) + return response, 200 + @route("//automations/", methods=["GET"]) @use_kwargs( { @@ -1616,9 +1739,27 @@ def get_automations( }, location="path", ) + @use_kwargs( + { + "include": fields.List( + fields.Str(validate=validate.OneOf(["waited-for-by"])), + load_default=list, + metadata={ + "description": "Extra fields to include: `waited-for-by`, the automations that wait for this one." + }, + ) + }, + location="query", + ) @permission_required_for_context("read", ctx_arg_name="asset") @as_json - def get_automation(self, id: int, automation_id: int, asset: GenericAsset): + def get_automation( + self, + id: int, + automation_id: int, + asset: GenericAsset, + include: list[str] | None = None, + ): """ .. :quickref: Assets; Get details of one automation defined on an asset. @@ -1634,6 +1775,9 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset): Note that jobs in Redis have a limited TTL, so not all past jobs will be counted, while durable run status records queueing attempts and outcomes even after those jobs expire. The cursor is the time of the most recent run the automation committed to, in the automation's own timezone; runs at or before it are never queued again. It advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded. + + Pass `include=waited-for-by` to also list the automations that wait for this one, such as before deleting it. + They are looked up across all automations, and one you may not read is listed as a bare ID. security: - ApiKeyAuth: [] parameters: @@ -1649,6 +1793,15 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset): description: ID of the automation. schema: type: integer + - in: query + name: include + required: false + description: Extra fields to include, namely `waited-for-by`. + schema: + type: array + items: + type: string + enum: [waited-for-by] responses: 200: description: PROCESSED @@ -1758,6 +1911,18 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset): {"id": sensor.id, "name": sensor.name} for sensor in automation_sensors[key] ] + if "waited-for-by" in (include or []): + automation_data["waited-for-by"] = [ + { + "automation": waiting.id, + **( + {"name": waiting.name, "asset": waiting.asset_id} + if _may_read_automation(waiting) + else {} + ), + } + for waiting in get_automations_waiting_for(automation) + ] redis_connection_err = None try: automation_data["job-stats"] = get_automation_job_stats(automation) @@ -1815,6 +1980,11 @@ def post_automation(self, automation_data: dict, id: int, asset: GenericAsset): The automation can only involve sensors that you have access to yourself: read access to the sensors it reads data from, and permission to record data on the sensors it writes to. + + An automation can wait for other automations that it consumes the results of, listed in `depends-on`. + Each of its runs then waits for their latest runs due at or before its own scheduled time, and queues its jobs once those ended, + or is skipped if one of them failed, unless that dependency has `allow-failure` set. + The automations it waits for can be on any asset you can read, and may recur on another schedule. security: - ApiKeyAuth: [] parameters: @@ -1912,15 +2082,17 @@ def patch_automation( self, automation_data: dict, id: int, automation_id: int, asset: GenericAsset ): """ - .. :quickref: Assets; Update an automation's name, cron string, timezone or activation status. + .. :quickref: Assets; Update an automation's name, cron string, timezone, activation status or dependencies. --- patch: - summary: Update an automation's name, cron string, timezone or activation status. + summary: Update an automation's name, cron string, timezone, activation status or dependencies. description: | - Any subset of the fields `name`, `cronstr`, `timezone` and `active` can be sent. + Any subset of the fields `name`, `cron`, `timezone`, `active` and `depends-on` can be sent. Changing the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change, so runs from before it are not caught up on, while a run due in that very minute still is. + An automation's runs wait for the latest runs of the automations in `depends-on` that are due at or before their own scheduled time, + so either side of a dependency can change its recurrence or timezone. Other automation fields cannot be updated; instead, create a new automation. Requires permission to add data under the asset. security: @@ -1947,6 +2119,13 @@ def patch_automation( summary: Deactivate the automation value: active: false + wait_for_another_automation: + summary: Wait for the runs of automation 4 + description: >- + Each run waits for the latest run of automation 4 due at or before its own time, and is skipped if that one failed. + value: + depends-on: + - automation: 4 responses: 200: description: PROCESSED @@ -1966,7 +2145,13 @@ def patch_automation( return { "message": f"Asset {asset.id} has no automation with id {automation_id}." }, 404 - update_automation(automation, origin="API", **automation_data) + try: + update_automation( + automation, origin="API", check_permissions=True, **automation_data + ) + except ValidationError as e: + db.session.rollback() + return unprocessable_entity(e.messages) db.session.commit() response = automation_schema.dump(automation) response["recurrence-description"] = describe_cronstr(automation.cronstr) @@ -2040,12 +2225,46 @@ def delete_automation(self, id: int, automation_id: int, asset: GenericAsset): }, location="path", ) + @use_kwargs( + { + "with_downstream": fields.Bool( + data_key="with-downstream", + load_default=False, + metadata={ + "description": "Also run the active automations downstream of this one, which wait for it, directly or through others, each once the runs it waits for ended." + }, + ), + "with_upstream": fields.Bool( + data_key="with-upstream", + load_default=False, + metadata={ + "description": "First run the active automations upstream of this one, which it waits for, directly or through others, as far as you may run them." + }, + ), + "exclude": fields.List( + fields.Int(), + load_default=list, + metadata={ + "description": "With `with-upstream` or `with-downstream`, IDs of automations not to run, along with what can only be reached through them." + }, + ), + }, + location="query", + ) # Running an automation writes data under the asset, which is what create-children means here. # The sensors it writes to were checked against the same permission when the automation was created, # and its output scope is checked again on each run (see validate_automation_output_scope). @permission_required_for_context("manage-automations", ctx_arg_name="asset") @as_json - def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): + def trigger_automation( + self, + id: int, + automation_id: int, + asset: GenericAsset, + with_downstream: bool = False, + with_upstream: bool = False, + exclude: list[int] | None = None, + ): """ .. :quickref: Assets; Trigger a single run of an automation. @@ -2063,6 +2282,18 @@ def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): An on-demand run does not affect the automation's recurrence: its cursor stays where it was, so the next recurring run still happens as scheduled, and a missed run is still caught up. Inactive automations can be triggered, too, which is how you can try one out before activating it. + + Pass `with-upstream=true` to first run the active automations upstream of this one, which it waits for, directly or through others, and then this one, + for instance to bring it up to date with what its results are based on. + Pass `with-downstream=true` to also run the active automations downstream of this one, which wait for it, directly or through others, + for instance to re-run a chain of automations once the data at its start was backfilled. + The included automations run in chain order: each waits for the runs triggered along with it, + and is skipped if one of those failed, unless its dependency allows for failure. + An automation that is inactive, or that you may not run, is excluded, along with what can only be reached through it. + Pass `exclude` to exclude automations yourself, such as one of several branches. + See the `run-plan` endpoint for what this would come down to. + The response lists the runs under `runs`, in chain order, and the excluded automations under `excluded`, each with the reason; + its `job` is that of this automation's own run, if that started right away. security: - ApiKeyAuth: [] parameters: @@ -2078,6 +2309,28 @@ def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): description: ID of the automation to run. schema: type: integer + - in: query + name: with-upstream + required: false + description: First run the active automations upstream of this one, which it waits for, directly or through others. Cannot be combined with `with-downstream`. + schema: + type: boolean + default: false + - in: query + name: with-downstream + required: false + description: Also run the active automations downstream of this one, which wait for it, directly or through others. + schema: + type: boolean + default: false + - in: query + name: exclude + required: false + description: With `with-upstream` or `with-downstream`, IDs of automations not to run. + schema: + type: array + items: + type: integer responses: 202: description: PROCESSING @@ -2096,6 +2349,39 @@ def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): job-url: "/api/v3_0/jobs/364bfd06-c1fa-430b-8d25-8f5a547651fb" n-jobs: 2 message: "Request has been accepted for processing." + triggered-with-downstream: + summary: Automation run accepted, along with the automations downstream of it + value: + status: ACCEPTED + job: "364bfd06-c1fa-430b-8d25-8f5a547651fb" + job-url: "/api/v3_0/jobs/364bfd06-c1fa-430b-8d25-8f5a547651fb" + n-jobs: 1 + message: "Started 1 run(s), and 1 run(s) wait for them." + runs: + - automation: 7 + run: 411 + dispatch-state: queued + - automation: 8 + run: 412 + dispatch-state: waiting + excluded: [] + triggered-with-upstream: + summary: Automation run accepted, after the automations upstream of it + value: + status: ACCEPTED + job: null + n-jobs: 1 + message: "Started 1 run(s), and 1 run(s) wait for them." + runs: + - automation: 7 + run: 411 + dispatch-state: queued + - automation: 8 + run: 412 + dispatch-state: waiting + excluded: + - automation: 5 + reason: You may not run this automation. 401: description: UNAUTHORIZED 403: @@ -2112,6 +2398,21 @@ def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): return { "message": f"Asset {asset.id} has no automation with id {automation_id}." }, 404 + if with_downstream and with_upstream: + return unprocessable_entity( + "Use either with-upstream or with-downstream, not both." + ) + if with_upstream or with_downstream: + return self._trigger_automation_chain( + asset, + automation, + "upstream" if with_upstream else "downstream", + exclude or [], + ) + if exclude: + return unprocessable_entity( + "Use exclude along with with-upstream or with-downstream." + ) try: returns = run_automation(automation) except ( @@ -2147,6 +2448,198 @@ def trigger_automation(self, id: int, automation_id: int, asset: GenericAsset): response["n-jobs"] = returns.get("n_jobs") return response, status_code + @route("//automations//run-plan", methods=["GET"]) + @use_kwargs( + { + "asset": AssetIdField(data_key="id"), + "automation_id": fields.Int(), + }, + location="path", + ) + @use_kwargs( + { + "include": fields.Str( + required=True, + validate=validate.OneOf(CHAIN_INCLUDES), + metadata={ + "description": "Which part of the chain to include: the automations upstream of this one, which it waits for, or those downstream of it, which wait for it." + }, + ), + "exclude": fields.List( + fields.Int(), + load_default=list, + metadata={"description": "IDs of automations not to run."}, + ), + }, + location="query", + ) + @permission_required_for_context("read", ctx_arg_name="asset") + @as_json + def get_automation_run_plan( + self, + id: int, + automation_id: int, + asset: GenericAsset, + include: str, + exclude: list[int] | None = None, + ): + """ + .. :quickref: Assets; See what running an automation along with what is upstream or downstream of it comes down to. + + --- + get: + summary: See what running an automation along with what is upstream or downstream of it comes down to. + description: | + Nothing is run. + Runs always flow downstream; `include` says which part of the chain is included. + The automations that will run are listed under `steps`, in chain order, each saying what it waits for among these steps. + The automations that will not run are listed under `excluded`, each with the reason, + which of the steps wait for it, and, if you may read it, when it last succeeded (null if it never did). + They still run on their own schedule, just not because of this request, + and the steps that wait for them go by the results they last produced. + An automation you may not read stays a bare ID. + security: + - ApiKeyAuth: [] + parameters: + - in: path + name: id + required: true + description: ID of the asset the automation is defined on. + schema: + type: integer + - in: path + name: automation-id + required: true + description: ID of the automation. + schema: + type: integer + - in: query + name: include + required: true + description: Which part of the chain to include, upstream or downstream of this automation. + schema: + type: string + enum: [upstream, downstream] + - in: query + name: exclude + required: false + description: IDs of automations not to run. + schema: + type: array + items: + type: integer + responses: + 200: + description: PROCESSED + content: + application/json: + examples: + run-plan: + summary: One automation does not run + value: + steps: + - automation: 7 + waits-for: [] + name: Forecast + asset: 3 + asset-name: Site + excluded: + - automation: 5 + reason: You may not run this automation. + waited-for-by: [7] + name: Day-ahead prices + asset: 2 + asset-name: Market + last-succeeded-at: null + 401: + description: UNAUTHORIZED + 403: + description: INVALID_SENDER + 404: + description: NOT_FOUND + 422: + description: UNPROCESSABLE_ENTITY + tags: + - Assets + """ + automation = db.session.get(Automation, automation_id) + if automation is None or automation.asset_id != asset.id: + return { + "message": f"Asset {asset.id} has no automation with id {automation_id}." + }, 404 + return ( + plan_automation_chain( + automation, + include, + exclude or [], + may_read=_may_read_automation, + may_run=_may_manage_automation, + ), + 200, + ) + + def _trigger_automation_chain( + self, + asset: GenericAsset, + automation: Automation, + include: str, + exclude: list[int], + ) -> tuple[dict, int]: + """Run an automation along with what is upstream or downstream of it (see `run_automation_chain`).""" + try: + # The automations in a chain may be on other assets, where their runs write data, too. + # One the user may not run is excluded, rather than refusing the whole chain, + # as another organisation may wait for this automation, which reading it is enough for. + runs, returns, excluded = run_automation_chain( + automation, include, exclude, may_run=_may_manage_automation + ) + except ( + NotImplementedError, + ValueError, + ValidationError, + AutomationSensorsUnknown, + ) as e: + db.session.rollback() + return unprocessable_entity( + e.messages if isinstance(e, ValidationError) else str(e) + ) + for run in runs: + AssetAuditLog.add_record( + run.automation.asset, + ( + f"Triggered a run of automation '{run.automation.name}' ({run.automation_id}) and of the automations {include} of it." + if run.automation_id == automation.id + else f"Triggered a run of automation '{run.automation.name}' ({run.automation_id}), {'upstream' if include == 'upstream' else 'downstream'} of automation {automation.id}." + ), + ) + db.session.commit() + for run in runs: + db.session.refresh(run) + job_id = (returns or {}).get("job_id") + if job_id is not None: + response, status_code = request_accepted_for_processing(job_id) + else: + response, status_code = {"status": "ACCEPTED", "job": None}, 202 + n_waiting = sum(run.dispatch_state == "waiting" for run in runs) + response["message"] = ( + f"Started {len(runs) - n_waiting} run(s), and {n_waiting} run(s) wait for them." + ) + if returns is not None and job_id is None and returns.get("message"): + response["message"] = f"{returns['message']} {response['message']}" + response["n-jobs"] = sum(len(run.job_intents) for run in runs) + response["runs"] = [ + { + "automation": run.automation_id, + "run": run.id, + "dispatch-state": run.dispatch_state, + } + for run in runs + ] + response["excluded"] = [ + {"automation": step.id, "reason": reason} for step, reason in excluded + ] + return response, status_code + @route("//jobs", methods=["GET"]) @use_kwargs( {"asset": AssetIdField(data_key="id")}, diff --git a/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py b/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py index 4fba308614..90e3aa3f7f 100644 --- a/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py +++ b/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py @@ -686,6 +686,7 @@ def test_a_schedule_automation_is_skipped_with_a_reason_of_its_own( ] assert len(skipped) == 1 assert "cannot be copied yet" in skipped[0].reason + assert "flex config" in skipped[0].reason def test_a_copy_records_under_a_data_source_of_its_own_organisation( @@ -756,3 +757,53 @@ def test_a_copy_keeps_no_reference_its_maker_cannot_read( ] assert len(skipped) == 1 assert str(client_power.id) in skipped[0].reason + + +def test_a_copy_waits_for_the_copies_of_what_its_original_waits_for( + fresh_db, automated_site +): + """Within the copied subtree, a dependency points at the copied automation; outside it, at the very automation the original waits for.""" + from flexmeasures.data.models.automations import AutomationDependency + + neighbouring_automation = _add_automation( + fresh_db, + asset=automated_site["neighbour"], + name="Neighbouring power forecasts", + parameters={"sensor": automated_site["neighbouring_power"].id}, + ) + site_automation = automated_site["site_automation"] + fresh_db.session.add_all( + [ + AutomationDependency( + automation=site_automation, + upstream=automated_site["meter_automation"], + ), + AutomationDependency( + automation=site_automation, + upstream=neighbouring_automation, + allow_failure=True, + ), + ] + ) + fresh_db.session.commit() + + site_copy = copy_asset(automated_site["site"]).asset + + copied_site_automation = _automations_of(fresh_db, site_copy)[0] + copied_meter_automation = _automations_of(fresh_db, _child_of(fresh_db, site_copy))[ + 0 + ] + assert sorted( + (dependency.upstream_id, dependency.allow_failure) + for dependency in copied_site_automation.dependencies + ) == sorted( + [ + (copied_meter_automation.id, False), + (neighbouring_automation.id, True), + ] + ) + # The original keeps waiting for what it waited for. + assert {dependency.upstream_id for dependency in site_automation.dependencies} == { + automated_site["meter_automation"].id, + neighbouring_automation.id, + } diff --git a/flexmeasures/api/v3_0/tests/test_automation_dependencies_api_fresh_db.py b/flexmeasures/api/v3_0/tests/test_automation_dependencies_api_fresh_db.py new file mode 100644 index 0000000000..9b7987e307 --- /dev/null +++ b/flexmeasures/api/v3_0/tests/test_automation_dependencies_api_fresh_db.py @@ -0,0 +1,544 @@ +"""Naming the automations an automation waits for, through the API.""" + +from datetime import timedelta + +import pytest +from flask import url_for +from sqlalchemy import select + +from flexmeasures.data.models.automations import ( + Automation, + AutomationDependency, + AutomationRun, +) +from flexmeasures.data.models.data_sources import DataSource +from flexmeasures.data.models.time_series import Sensor + + +@pytest.fixture +def upstream_automations( + fresh_db, setup_roles_users_fresh_db, setup_generic_assets_fresh_db +): + """An automation on the prosumer's own battery, one on a public asset, and one on the supplier's wind turbine, all recurring at 06:00 UTC.""" + assets = setup_generic_assets_fresh_db + battery = assets["test_battery"] + sensor = Sensor( + name="battery power", + unit="MW", + event_resolution=timedelta(minutes=15), + generic_asset=battery, + ) + generator = DataSource( + name="forecaster", type="forecaster", model="TrainPredictPipeline" + ) + fresh_db.session.add_all([sensor, generator]) + fresh_db.session.flush() + automations = { + key: Automation( + asset=assets[asset_key], + generator=generator, + type="forecasting", + name=f"Upstream on {asset_key}", + cronstr="0 6 * * *", + timezone="UTC", + parameters={}, + ) + for key, asset_key in ( + ("own", "test_battery"), + ("public", "troposphere"), + ("foreign", "test_wind_turbine"), + ) + } + fresh_db.session.add_all(automations.values()) + fresh_db.session.commit() + return battery, sensor, automations + + +def _waits_for(response) -> list[dict]: + """What a response says an automation waits for, without what it tells about those automations.""" + return [ + {key: dependency[key] for key in ("automation", "allow-failure")} + for dependency in response.json["depends-on"] + ] + + +def _post(client, battery, sensor, depends_on, cron="0 6 * * *"): + return client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={ + "name": "Downstream forecasts", + "cron": cron, + "timezone": "UTC", + "type": "forecasting", + "parameters": {"sensor": sensor.id}, + "depends-on": depends_on, + }, + ) + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_post_automation_waiting_for_readable_automations( + app, fresh_db, upstream_automations, requesting_user +): + """Upstream automations can be on the user's own assets and on public ones, and the response names them.""" + battery, sensor, upstream = upstream_automations + with app.test_client() as client: + response = _post( + client, + battery, + sensor, + [ + {"automation": upstream["own"].id}, + {"automation": upstream["public"].id, "allow-failure": True}, + ], + ) + assert response.status_code == 201, response.json + assert _waits_for(response) == [ + {"automation": upstream["own"].id, "allow-failure": False}, + {"automation": upstream["public"].id, "allow-failure": True}, + ] + listing = client.get(url_for("AssetAPI:get_automations", id=battery.id)) + listed = { + automation["id"]: automation for automation in listing.json["automations"] + } + assert listed[response.json["id"]]["depends-on"] == response.json["depends-on"] + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_an_upstream_automation_is_only_described_to_who_may_read_it( + app, fresh_db, upstream_automations, requesting_user +): + """One a user may read comes with its name and asset, while one they may not read gives nothing away but its ID.""" + battery, sensor, upstream = upstream_automations + with app.test_client() as client: + response = _post(client, battery, sensor, [{"automation": upstream["own"].id}]) + assert response.status_code == 201, response.json + # The API refuses to wait for an automation one may not read, but access to one can be lost later on. + fresh_db.session.add( + AutomationDependency( + automation_id=response.json["id"], + upstream_id=upstream["foreign"].id, + allow_failure=False, + ) + ) + fresh_db.session.commit() + listing = client.get(url_for("AssetAPI:get_automations", id=battery.id)) + listed = { + automation["id"]: automation for automation in listing.json["automations"] + } + described = { + dependency["automation"]: dependency + for dependency in listed[response.json["id"]]["depends-on"] + } + assert described[upstream["own"].id] == { + "automation": upstream["own"].id, + "allow-failure": False, + "name": "Upstream on test_battery", + "asset": battery.id, + "asset-name": battery.name, + } + assert described[upstream["foreign"].id] == { + "automation": upstream["foreign"].id, + "allow-failure": False, + } + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_post_automation_refuses_to_wait_for_an_unreadable_automation( + app, fresh_db, upstream_automations, requesting_user +): + """An automation one may not read gets the answer of one that does not exist, so that IDs reveal nothing.""" + battery, sensor, upstream = upstream_automations + foreign_id = upstream["foreign"].id + unknown_id = foreign_id + 1000 + with app.test_client() as client: + foreign = _post(client, battery, sensor, [{"automation": foreign_id}]) + unknown = _post(client, battery, sensor, [{"automation": unknown_id}]) + assert foreign.status_code == unknown.status_code == 422, foreign.json + assert foreign.get_data(as_text=True).replace( + str(foreign_id), "" + ) == unknown.get_data(as_text=True).replace(str(unknown_id), "") + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_post_automation_waiting_for_one_on_another_schedule( + app, fresh_db, upstream_automations, requesting_user +): + """A run waits for the latest upstream run due at or before its own time, so the schedules need not match.""" + battery, sensor, upstream = upstream_automations + with app.test_client() as client: + response = _post( + client, + battery, + sensor, + [{"automation": upstream["own"].id}], + cron="0 7 * * *", + ) + assert response.status_code == 201, response.json + assert _waits_for(response) == [ + {"automation": upstream["own"].id, "allow-failure": False} + ] + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_patch_automation_dependencies( + app, fresh_db, upstream_automations, requesting_user +): + """Dependencies can be replaced and cleared, and the schedule can change while they stay.""" + battery, sensor, upstream = upstream_automations + with app.test_client() as client: + created = _post(client, battery, sensor, [{"automation": upstream["own"].id}]) + assert created.status_code == 201, created.json + url = url_for( + "AssetAPI:patch_automation", id=battery.id, automation_id=created.json["id"] + ) + + moved = client.patch(url, json={"cron": "0 7 * * *"}) + assert moved.status_code == 200, moved.json + assert _waits_for(moved) == [ + {"automation": upstream["own"].id, "allow-failure": False} + ] + + replaced = client.patch( + url, + json={ + "depends-on": [ + {"automation": upstream["public"].id, "allow-failure": True} + ] + }, + ) + assert replaced.status_code == 200, replaced.json + assert _waits_for(replaced) == [ + {"automation": upstream["public"].id, "allow-failure": True} + ] + + cleared = client.patch(url, json={"depends-on": []}) + assert cleared.status_code == 200, cleared.json + assert cleared.json["depends-on"] == [] + assert cleared.json["cron"] == "0 7 * * *" + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_patch_automation_refuses_to_wait_for_an_unreadable_automation( + app, fresh_db, upstream_automations, requesting_user +): + battery, sensor, upstream = upstream_automations + with app.test_client() as client: + response = client.patch( + url_for( + "AssetAPI:patch_automation", + id=battery.id, + automation_id=upstream["own"].id, + ), + json={"depends-on": [{"automation": upstream["foreign"].id}]}, + ) + assert response.status_code == 422, response.json + fresh_db.session.expire_all() + assert fresh_db.session.get(Automation, upstream["own"].id).dependencies == [] + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_listing_automations_across_assets( + app, fresh_db, upstream_automations, requesting_user +): + """Public automations can be listed apart, and everything a user may read comes together, but nothing they may not read.""" + _, _, upstream = upstream_automations + with app.test_client() as client: + public = client.get( + url_for("AssetAPI:get_automations_across_assets"), + query_string={"scope": "public"}, + ) + everything = client.get( + url_for("AssetAPI:get_automations_across_assets"), + query_string={"scope": "all"}, + ) + unknown = client.get( + url_for("AssetAPI:get_automations_across_assets"), + query_string={"scope": "nonsense"}, + ) + assert public.status_code == 200, public.json + assert [a["id"] for a in public.json["automations"]] == [upstream["public"].id] + assert public.json["automations"][0]["asset-name"] == "troposphere" + assert "num-records" not in public.json + assert {a["id"] for a in everything.json["automations"]} == { + upstream["own"].id, + upstream["public"].id, + } + assert unknown.status_code == 422 + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_listing_automations_across_assets_by_page( + app, fresh_db, upstream_automations, requesting_user, monkeypatch +): + """Pages are cut from what the user may read, so an automation they may not read takes no place on a page.""" + _, _, upstream = upstream_automations + extra = Automation( + name="Another public one", + asset=upstream["public"].asset, + type=upstream["public"].type, + cronstr="0 * * * *", + generator=upstream["public"].generator, + parameters=dict(upstream["public"].parameters), + ) + fresh_db.session.add(extra) + fresh_db.session.commit() + # Keep the user's own automation, which comes first by ID, from them. + hidden = upstream["own"].id + monkeypatch.setattr( + "flexmeasures.api.v3_0.assets._may_read_automation", + lambda automation: automation.id != hidden, + ) + url = url_for("AssetAPI:get_automations_across_assets") + with app.test_client() as client: + pages = [ + client.get(url, query_string={"scope": "all", "page": page, "per-page": 1}) + for page in (1, 2, 3) + ] + assert [page.status_code for page in pages] == [200] * 3 + assert [[a["id"] for a in page.json["automations"]] for page in pages] == [ + [upstream["public"].id], + [extra.id], + [], + ] + assert {page.json["num-records"] for page in pages} == {2} + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_automation_details_say_what_waits_for_it( + app, fresh_db, upstream_automations, requesting_user +): + """Before deleting an automation, one can ask what waits for it, also on another asset, without learning about what one may not read.""" + battery, sensor, upstream = upstream_automations + downstream = _downstream_of(fresh_db, upstream, sensor, upstream["own"]) + fresh_db.session.add( + AutomationDependency( + automation_id=upstream["foreign"].id, + upstream_id=upstream["own"].id, + allow_failure=False, + ) + ) + fresh_db.session.commit() + with app.test_client() as client: + response = client.get( + url_for( + "AssetAPI:get_automation", + id=battery.id, + automation_id=upstream["own"].id, + ), + query_string={"include": "waited-for-by"}, + ) + plain = client.get( + url_for( + "AssetAPI:get_automation", + id=battery.id, + automation_id=upstream["own"].id, + ) + ) + assert "waited-for-by" not in plain.json + assert response.status_code == 200, response.json + waiting = {item["automation"]: item for item in response.json["waited-for-by"]} + assert waiting[downstream.id] == { + "automation": downstream.id, + "name": "Downstream", + "asset": battery.id, + } + assert waiting[upstream["foreign"].id] == {"automation": upstream["foreign"].id} + + +def _downstream_of(fresh_db, upstream, sensor, *upstreams) -> Automation: + """An automation on the prosumer's battery that waits for these automations.""" + downstream = Automation( + asset=upstream["own"].asset, + generator=upstream["own"].generator, + type="forecasting", + name="Downstream", + cronstr="0 6 * * *", + timezone="UTC", + parameters={"sensor": sensor.id}, + ) + fresh_db.session.add(downstream) + fresh_db.session.flush() + for automation in upstreams: + fresh_db.session.add( + AutomationDependency( + automation_id=downstream.id, + upstream_id=automation.id, + allow_failure=False, + ) + ) + fresh_db.session.commit() + return downstream + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_run_plan_tells_what_will_run_and_what_will_not( + app, fresh_db, upstream_automations, requesting_user +): + """A public automation can be waited for but not run, which the plan says, and the plan runs nothing.""" + battery, sensor, upstream = upstream_automations + downstream = _downstream_of( + fresh_db, + upstream, + sensor, + upstream["own"], + upstream["public"], + upstream["foreign"], + ) + with app.test_client() as client: + response = client.get( + url_for( + "AssetAPI:get_automation_run_plan", + id=battery.id, + automation_id=downstream.id, + ), + query_string={"include": "upstream"}, + ) + assert response.status_code == 200, response.json + assert [step["automation"] for step in response.json["steps"]] == [ + upstream["own"].id, + downstream.id, + ] + excluded = {step["automation"]: step for step in response.json["excluded"]} + assert ( + excluded[upstream["public"].id]["reason"] == "You may not run this automation." + ) + assert excluded[upstream["public"].id]["name"] == "Upstream on troposphere" + assert excluded[upstream["foreign"].id] == { + "automation": upstream["foreign"].id, + "reason": "You may not run this automation.", + "waited-for-by": [downstream.id], + } + assert excluded[upstream["public"].id]["last-succeeded-at"] is None + assert fresh_db.session.scalars(select(AutomationRun)).all() == [] + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_trigger_automation_with_upstream( + app, fresh_db, upstream_automations, requesting_user +): + """What the user may run runs first, what they may not is left out and reported, and what they leave out does not run.""" + battery, sensor, upstream = upstream_automations + upstream["own"].parameters = {"sensor": sensor.id} + downstream = _downstream_of( + fresh_db, upstream, sensor, upstream["own"], upstream["public"] + ) + url = url_for( + "AssetAPI:trigger_automation", id=battery.id, automation_id=downstream.id + ) + with app.test_client() as client: + response = client.post(url, query_string={"with-upstream": "true"}) + assert response.status_code == 202, response.json + assert [ + (run["automation"], run["dispatch-state"]) for run in response.json["runs"] + ] == [(upstream["own"].id, "queued"), (downstream.id, "waiting")] + assert response.json["excluded"] == [ + { + "automation": upstream["public"].id, + "reason": "You may not run this automation.", + } + ] + + excluded = client.post( + url, + query_string={ + "with-upstream": "true", + "exclude": [upstream["own"].id], + }, + ) + assert excluded.status_code == 202, excluded.json + assert [run["automation"] for run in excluded.json["runs"]] == [downstream.id] + assert {"automation": upstream["own"].id, "reason": "You left it out."} in ( + excluded.json["excluded"] + ) + + both = client.post( + url, query_string={"with-upstream": "true", "with-downstream": "true"} + ) + exclude_alone = client.post(url, query_string={"exclude": [upstream["own"].id]}) + assert both.status_code == 422, both.json + assert exclude_alone.status_code == 422, exclude_alone.json + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_trigger_automation_with_downstream( + app, fresh_db, upstream_automations, requesting_user +): + """Triggering with downstream runs the automations waiting for this one, too, and lists the runs in chain order.""" + battery, sensor, upstream = upstream_automations + upstream["own"].parameters = {"sensor": sensor.id} + fresh_db.session.commit() + with app.test_client() as client: + created = _post(client, battery, sensor, [{"automation": upstream["own"].id}]) + assert created.status_code == 201, created.json + response = client.post( + url_for( + "AssetAPI:trigger_automation", + id=battery.id, + automation_id=upstream["own"].id, + **{"with-downstream": "true"}, + ) + ) + assert response.status_code == 202, response.json + assert response.json["job"] is not None + assert [ + (run["automation"], run["dispatch-state"]) for run in response.json["runs"] + ] == [(upstream["own"].id, "queued"), (created.json["id"], "waiting")] + + +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user@seita.nl"], indirect=True +) +def test_trigger_automation_with_downstream_excludes_what_the_user_may_not_run( + app, fresh_db, upstream_automations, requesting_user +): + """Another organisation's automation waiting for this one does not run on the user's say, and does not keep the rest from running.""" + battery, sensor, upstream = upstream_automations + upstream["own"].parameters = {"sensor": sensor.id} + fresh_db.session.add( + AutomationDependency(automation=upstream["foreign"], upstream=upstream["own"]) + ) + fresh_db.session.commit() + with app.test_client() as client: + response = client.post( + url_for( + "AssetAPI:trigger_automation", + id=battery.id, + automation_id=upstream["own"].id, + **{"with-downstream": "true"}, + ) + ) + assert response.status_code == 202, response.json + assert [run["automation"] for run in response.json["runs"]] == [upstream["own"].id] + assert response.json["excluded"] == [ + { + "automation": upstream["foreign"].id, + "reason": "You may not run this automation.", + } + ] + runs = fresh_db.session.scalars(select(AutomationRun)).all() + assert [run.automation_id for run in runs] == [upstream["own"].id] diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py index fb8afad42f..865f25d323 100755 --- a/flexmeasures/cli/data_add.py +++ b/flexmeasures/cli/data_add.py @@ -45,10 +45,12 @@ DeprecatedOptionsCommand, LoggedClickExceptionGroup, add_cli_options_from_schema, + assemble_automation_dependencies, split_commas, ) from flexmeasures.data import db from flexmeasures.data.automations import get_automation_handler, get_automation_types +from flexmeasures.data.models.automations import Automation from flexmeasures.data.scripts.data_gen import ( add_transmission_zone_asset, populate_initial_structure, @@ -86,7 +88,7 @@ ) from flexmeasures.data.models.data_sources import DataSource, DEFAULT_DATASOURCE_TYPES from flexmeasures.data.models.annotations import Annotation, get_or_create_annotation -from flexmeasures.data.schemas.automations import CronField +from flexmeasures.data.schemas.automations import AutomationIdField, CronField from flexmeasures.data.schemas import ( AccountIdField, AwareDateTimeField, @@ -1872,6 +1874,24 @@ def add_forecast( # noqa: C901 help="How long the period each run covers lasts, as an ISO 8601 duration, such as P1D." " For a forecast or schedule automation, the period then starts at the run time, unless --start-offset or --end-offset says otherwise.", ) +@click.option( + "--depends-on", + "depends_on", + multiple=True, + type=AutomationIdField(), + help="ID of an automation whose runs each run of this one waits for: its latest run due at or before the run's own time." + " A run queues its jobs once the runs it waits for ended, and is skipped if one of them failed (but see --allow-failure-of)." + " The automation waited for may recur on another schedule." + " Repeat to wait for several automations.", +) +@click.option( + "--allow-failure-of", + "allow_failure_of", + multiple=True, + type=AutomationIdField(), + help="ID of an automation given with --depends-on whose failed (or missed) run should not stop this automation's run." + " Repeat for several.", +) @add_cli_options_from_schema( ForecasterParametersSchema(), hidden=True, @@ -1893,6 +1913,8 @@ def add_automation( # noqa: C901 source: DataSource | None = None, config_file: TextIOBase | None = None, parameters_file: TextIOBase | None = None, + depends_on: tuple[Automation, ...] = (), + allow_failure_of: tuple[Automation, ...] = (), **kwargs, ): """ @@ -1921,6 +1943,9 @@ def add_automation( # noqa: C901 Alternatively, pass an existing data source (--source) to reuse the forecaster and configuration stored on it. + An automation that consumes what another automation produces can wait for it (--depends-on), + so that each of its runs queues its jobs only once the other automation's latest run due at or before that time ended. + Every forecaster and pipeline option that `flexmeasures add forecast` accepts is accepted here, too, but is left out of the help text above to keep it focused on the automation itself; run `flexmeasures add forecast --help` to see them. @@ -1933,6 +1958,7 @@ def add_automation( # noqa: C901 except (ValueError, NotImplementedError) as e: available = ", ".join(get_automation_types()) raise click.UsageError(f"{e} Available types: {available}.") + dependencies = assemble_automation_dependencies(depends_on, allow_failure_of) # Only a forecast automation has a default generator: # a report automation has to name its reporter, and the service says so, while a schedule automation resolves its own. @@ -1992,6 +2018,7 @@ def add_automation( # noqa: C901 config=config, source=source, origin="CLI", + dependencies=dependencies, ) except ValidationError as e: # The messages name the part of the request at fault, which is not always the parameters. diff --git a/flexmeasures/cli/data_delete.py b/flexmeasures/cli/data_delete.py index 74eaeee209..6e03afc390 100644 --- a/flexmeasures/cli/data_delete.py +++ b/flexmeasures/cli/data_delete.py @@ -22,6 +22,7 @@ from flexmeasures.data.schemas.automations import AutomationIdField from flexmeasures.data.services.automations import ( delete_automation as remove_automation, + get_automations_waiting_for, get_automations_involving_sensor, ) from flexmeasures.data.models.time_series import Sensor, TimedBelief @@ -298,6 +299,10 @@ def delete_automation(automation: Automation, force: bool): """ if not force: prompt = f"Delete automation '{automation.name}' (ID: {automation.id}) of asset '{automation.asset.name}'?" + waiting = get_automations_waiting_for(automation) + if waiting: + names = ", ".join(f"'{other.name}' (ID: {other.id})" for other in waiting) + prompt = f"{names} wait(s) for this automation, and will run without it from now on. {prompt}" click.confirm(prompt, abort=True) remove_automation(automation, origin="CLI") db.session.commit() diff --git a/flexmeasures/cli/data_edit.py b/flexmeasures/cli/data_edit.py index 5b33ad400c..b9ea807eed 100644 --- a/flexmeasures/cli/data_edit.py +++ b/flexmeasures/cli/data_edit.py @@ -8,6 +8,7 @@ import pandas as pd from flask import current_app as app from flask.cli import with_appcontext +from marshmallow import ValidationError import json from flexmeasures.data.models.user import Account, Plan from flexmeasures.data.schemas.account import AccountIdField @@ -35,6 +36,7 @@ DeprecatedOption, DeprecatedOptionsCommand, abort, + assemble_automation_dependencies, validate_rate_limit_cli, ) from flexmeasures.utils.flexmeasures_inflection import pluralize @@ -98,22 +100,66 @@ def fm_edit_data(): default=None, help="Activate or deactivate the automation.", ) +@click.option( + "--depends-on", + "depends_on", + multiple=True, + type=AutomationIdField(), + help="ID of an automation whose runs each run of this one waits for: its latest run due at or before the run's own time." + " Replaces the automations it waited for. Repeat to wait for several automations.", +) +@click.option( + "--allow-failure-of", + "allow_failure_of", + multiple=True, + type=AutomationIdField(), + help="ID of an automation given with --depends-on whose failed (or missed) run should not stop this automation's run." + " Repeat for several.", +) +@click.option( + "--no-dependencies", + "no_dependencies", + is_flag=True, + help="Stop waiting for other automations.", +) def edit_automation( automation: Automation, name: str | None = None, cronstr: str | None = None, timezone: str | None = None, active: bool | None = None, + depends_on: tuple[Automation, ...] = (), + allow_failure_of: tuple[Automation, ...] = (), + no_dependencies: bool = False, ): - """Edit the name, recurrence, timezone or activation status of an automation.""" - changes = update_automation( - automation, - name=name, - cronstr=cronstr, - timezone=timezone, - active=active, - origin="CLI", - ) + """Edit the name, recurrence, timezone, activation status or dependencies of an automation.""" + if no_dependencies and (depends_on or allow_failure_of): + raise click.UsageError( + "--no-dependencies cannot be combined with --depends-on or --allow-failure-of." + ) + if allow_failure_of and not depends_on: + raise click.UsageError( + "--allow-failure-of needs --depends-on, as it replaces all the automations this one waits for." + ) + if no_dependencies: + dependencies: list[dict] | None = [] + elif depends_on: + dependencies = assemble_automation_dependencies(depends_on, allow_failure_of) + else: + dependencies = None + try: + changes = update_automation( + automation, + name=name, + cronstr=cronstr, + timezone=timezone, + active=active, + origin="CLI", + dependencies=dependencies, + ) + except ValidationError as e: + db.session.rollback() + abort(f"Cannot update automation {automation.id}: {e.messages}") if not changes: click.secho("Nothing to change.", **MsgStyle.WARN) return diff --git a/flexmeasures/cli/data_show.py b/flexmeasures/cli/data_show.py index cf1fd99add..3372d83149 100644 --- a/flexmeasures/cli/data_show.py +++ b/flexmeasures/cli/data_show.py @@ -41,6 +41,7 @@ from flexmeasures.data.services.automations import ( AutomationSensorsUnknown, describe_cronstr, + get_automations_waiting_for, resolve_automation_sensors, ) from flexmeasures.data.services.time_series import simplify_index @@ -554,7 +555,7 @@ def list_automations( def _show_automation(automation: Automation): - """Show one automation in detail, including the sensors it reads from and writes to.""" + """Show one automation in detail, including the automations it waits for and that wait for it, and the sensors it reads and writes.""" title = f"Automation {automation.id}: {automation.name}" click.echo("=" * len(title)) click.echo(title) @@ -583,6 +584,24 @@ def _show_automation(automation: Automation): ), ), ("Parameters", json.dumps(automation.parameters, indent=4)), + ( + "Waits for", + # Their recurrence says which of their runs this automation's runs wait for. + "\n".join( + f"{dependency.upstream.name} (ID: {dependency.upstream_id}, {dependency.upstream.cronstr} in {dependency.upstream.timezone})" + + (", also when it fails" if dependency.allow_failure else "") + for dependency in automation.dependencies + ) + or "none", + ), + ( + "Waited for by", + "\n".join( + f"{waiting.name} (ID: {waiting.id})" + for waiting in get_automations_waiting_for(automation) + ) + or "none", + ), ], headers=["Field", "Value"], ) @@ -606,8 +625,8 @@ def _show_automation(automation: Automation): ) return for role, header in ( - ("input_sensors", "Reads from"), - ("output_sensors", "Writes to"), + ("input_sensors", "Sensors it reads"), + ("output_sensors", "Sensors it writes"), ): click.echo(f"\n{header}:\n") if not sensors[role]: diff --git a/flexmeasures/cli/jobs.py b/flexmeasures/cli/jobs.py index 96e732d84a..347e49c878 100644 --- a/flexmeasures/cli/jobs.py +++ b/flexmeasures/cli/jobs.py @@ -47,6 +47,7 @@ floor_to_minute, get_dispatchable_automation_runs, run_automation, + run_automation_chain, ) from flexmeasures.data.services.data_sources import ( find_duplicate_sources, @@ -100,11 +101,32 @@ def run_automations(): Durable automation run records claim each scheduled run and make queueing resumable. Failed dispatch attempts are retried safely by reusing the original run plan and deterministic job IDs. + + A run of an automation that depends on other automations waits for their latest runs due at or before its own scheduled time, + and queues its jobs as soon as the last of them ended, or is skipped if one of them failed. + This command also dispatches the waiting runs whose upstream runs ended without them being released. """ now = floor_to_minute(server_now()) - claimed_runs = get_dispatchable_automation_runs(now) + report: list[dict] = [] + claimed_runs = get_dispatchable_automation_runs(now, report=report) + # A killed job's run, or an abandoned run on demand, is failed here, which can skip or set off the runs waiting for it, too. + for event in report: + run = event["run"] + failed = run.execution_state in ("failed", "skipped", "canceled") + click.secho( + f"Automation {run.automation_id} ('{run.automation.name}') run {run.id}, scheduled for {run.scheduled_at}, is {run.execution_state if failed else run.dispatch_state}" + + { + "reconciled": ", as its job ended without saying so", + "abandoned": ", as its dispatch was abandoned", + "released": ", as the runs it waits for ended", + }[event["event"]] + + "." + + (f" {event['message']}" if event["message"] else ""), + **(MsgStyle.WARN if failed else MsgStyle.SUCCESS), + ) if not claimed_runs: - click.secho(f"No automations due at {now}.", **MsgStyle.SUCCESS) + if not report: + click.secho(f"No automations due at {now}.", **MsgStyle.SUCCESS) return n_run = 0 @@ -114,6 +136,17 @@ def run_automations(): try: returns = dispatch_automation_run(claimed_run) n_jobs = returns["n_jobs"] + if returns.get("dispatch_state") in ("waiting", "skipped"): + click.secho( + f"Automation {automation.id} ('{automation.name}') run {claimed_run.run.id}, scheduled for {claimed_run.run.scheduled_at}, queued no jobs. {returns['message']}", + **( + MsgStyle.WARN + if returns["dispatch_state"] == "skipped" + else MsgStyle.SUCCESS + ), + ) + n_run += 1 + continue click.secho( f"Automation {automation.id} ('{automation.name}') run {claimed_run.run.id} queued {n_jobs} {automation.type} job(s) for asset {automation.asset_id}, scheduled for {claimed_run.run.scheduled_at}.", **MsgStyle.SUCCESS, @@ -144,7 +177,34 @@ def run_automations(): required=True, help="ID of the automation to run.", ) -def run_one_automation(automation: Automation): +@click.option( + "--with-upstream", + "with_upstream", + is_flag=True, + help="First run the active automations upstream of this one, which it waits for, directly or through others, and then this one," + " each once the runs it waits for ended, for instance to bring it up to date with what its results are based on.", +) +@click.option( + "--with-downstream", + "with_downstream", + is_flag=True, + help="Also run the active automations downstream of this one, which wait for it, directly or through others," + " each once the runs it waits for ended, for instance to re-run a chain after backfilling the data at its start.", +) +@click.option( + "--exclude", + "exclude", + type=int, + multiple=True, + help="With --with-upstream or --with-downstream, the ID of an automation not to run, along with what can only be reached through it." + " Use multiple times to exclude several.", +) +def run_one_automation( + automation: Automation, + with_upstream: bool = False, + with_downstream: bool = False, + exclude: tuple[int, ...] = (), +): """ Queue the jobs for a single run of one automation, now. @@ -159,7 +219,29 @@ def run_one_automation(automation: Automation): This does not affect the automation's recurrence: its cursor stays where it was, so the next recurring run still happens as scheduled (see `flexmeasures jobs run-automations`). Inactive automations can be run this way, too, which is how you can try one out before activating it. + A run on demand does not wait for the automations this one depends on, as it is meant to run now. + + With --with-upstream, the automations this one waits for run first, and this one runs once they ended; + with --with-downstream, the automations that wait for this one run, too, once it ended. + The included automations run in chain order: each waits for the runs asked for along with it, + and is skipped if one of those failed, unless its dependency allows for failure: + + \b + flexmeasures jobs run-automation --automation 4 --with-upstream + flexmeasures jobs run-automation --automation 4 --with-upstream --exclude 2 + flexmeasures jobs run-automation --automation 4 --with-downstream """ + if with_upstream and with_downstream: + raise click.UsageError("Use either --with-upstream or --with-downstream.") + if exclude and not (with_upstream or with_downstream): + raise click.UsageError( + "--exclude goes along with --with-upstream or --with-downstream." + ) + if with_upstream or with_downstream: + _run_automation_chain( + automation, "upstream" if with_upstream else "downstream", exclude + ) + return try: returns = run_automation(automation) except ( @@ -195,6 +277,61 @@ def run_one_automation(automation: Automation): ) +def _run_automation_chain( + automation: Automation, include: str, exclude: tuple[int, ...] +): + """Run an automation along with what is upstream or downstream of it, and say what each of them does.""" + try: + runs, returns, excluded = run_automation_chain(automation, include, exclude) + except ( + NotImplementedError, + ValueError, + ValidationError, + AutomationSensorsUnknown, + Forbidden, + ) as e: + db.session.rollback() + click.secho( + f"Automation {automation.id} ('{automation.name}') failed to queue jobs, so the automations waiting for it are skipped: {e}", + **MsgStyle.ERROR, + ) + raise click.Abort() + for run in runs: + AssetAuditLog.add_record( + run.automation.asset, + ( + f"Triggered a run of automation '{run.automation.name}' ({run.automation_id}) and of the automations {include} of it, via CLI." + if run.automation_id == automation.id + else f"Triggered a run of automation '{run.automation.name}' ({run.automation_id}), {include} of automation {automation.id}, via CLI." + ), + ) + db.session.commit() + for run in runs: + db.session.refresh(run) + n_jobs = ( + f", with {len(run.job_intents)} job(s)" + if run.dispatch_state != "waiting" + else "" + ) + click.secho( + f"Automation {run.automation_id} ('{run.automation.name}') run {run.id} is {run.dispatch_state}{n_jobs}" + + (f" ({run.execution_state})" if run.execution_state == "skipped" else "") + + ".", + **(MsgStyle.WARN if run.execution_state == "skipped" else MsgStyle.SUCCESS), + ) + if returns is not None and returns.get("job_id") is None and returns.get("message"): + click.secho(returns["message"], **MsgStyle.WARN) + for step, reason in excluded: + click.secho( + f"Automation {step.id} ('{step.name}') does not run. {reason}", + **MsgStyle.WARN, + ) + if len(runs) == 1 and not excluded: + click.secho( + f"No active automations are {include} of this one.", **MsgStyle.WARN + ) + + @fm_jobs.command("stats") @with_appcontext @click.option( diff --git a/flexmeasures/cli/tests/test_automations.py b/flexmeasures/cli/tests/test_automations.py index 5f0d44255a..bb573fb50b 100644 --- a/flexmeasures/cli/tests/test_automations.py +++ b/flexmeasures/cli/tests/test_automations.py @@ -11,7 +11,7 @@ from flexmeasures import Sensor from flexmeasures.data.models.audit_log import AssetAuditLog -from flexmeasures.data.models.automations import Automation +from flexmeasures.data.models.automations import Automation, AutomationRun from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType from flexmeasures.cli.tests.utils import to_flags from flexmeasures.data.services.generator_results import ( @@ -2399,12 +2399,17 @@ def test_run_automations( assert "No automations due" in result.output, result.output assert len(app.queues["forecasting"].jobs) == n_jobs - # inactive automations are not due + # inactive automations are not due, while the jobs of their runs that left Redis without saying how they ended are noticed, and reported for automation in automations: automation.active = False fresh_db.session.commit() app.redis_connection.flushdb() result = runner.invoke(run_automations) + assert "queued" not in result.output, result.output + assert ( + result.output.count("is failed, as its job ended without saying so") == 2 + ), result.output + result = runner.invoke(run_automations) assert "No automations due" in result.output, result.output @@ -2777,9 +2782,11 @@ def test_show_automations_lists_all_and_shows_one( f"{automation.generator.name} (ID: {automation.generator.id}, model: {automation.generator.model})" in result.output ) - assert "Reads from" in result.output - assert "Writes to" in result.output - reads_from, writes_to = result.output.split("Reads from:")[1].split("Writes to:") + assert "Sensors it reads" in result.output + assert "Sensors it writes" in result.output + reads_from, writes_to = result.output.split("Sensors it reads:")[1].split( + "Sensors it writes:" + ) assert "sensor 1" in writes_to assert "sensor 1" in reads_from and "sensor 2" in reads_from @@ -2914,7 +2921,7 @@ def test_show_automations_handles_unresolved_sensors( f"Could not determine the sensors of automation {automation.id}" in result.output ) - assert "Reads from" not in result.output + assert "Sensors it reads" not in result.output def test_show_automations_names_the_automation_when_schedule_sensors_are_unknown( @@ -2953,7 +2960,7 @@ def test_show_automations_names_the_automation_when_schedule_sensors_are_unknown assert result.exit_code == 0, result.output assert f"Automation {automation.id} ('Broken schedules')" in result.output assert "Could not determine the sensors of schedule automation" in result.output - assert "Reads from" not in result.output + assert "Sensors it reads" not in result.output def test_show_automations_shows_a_schedule_automation( @@ -2991,3 +2998,268 @@ def test_show_automations_shows_a_schedule_automation( assert result.exit_code == 0, result.output assert re.search(r"^Type\s+scheduling$", result.output, re.MULTILINE) assert automation.parameters["duration"] in result.output + + +def _add_hourly_automation(runner, sensor_id: int, name: str, *extra_args): + from flexmeasures.cli.data_add import add_automation + + return runner.invoke( + add_automation, + [ + "--asset", + "1", + "--name", + name, + "--cron", + "0 * * * *", + "--timezone", + "UTC", + "--sensor", + str(sensor_id), + *extra_args, + ], + ) + + +def test_add_edit_and_show_automation_dependencies( + app, fresh_db, setup_dummy_data_fresh_db +): + """An automation can be told to wait for another with --depends-on, which edit replaces or clears, and show lists.""" + from flexmeasures.cli.data_edit import edit_automation + from flexmeasures.cli.data_show import list_automations + + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + upstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + + result = _add_hourly_automation( + runner, + sensor_id, + "Downstream", + "--depends-on", + str(upstream.id), + "--allow-failure-of", + str(upstream.id), + ) + assert result.exit_code == 0, result.output + downstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Downstream") + ).one() + assert [ + (dependency.upstream_id, dependency.allow_failure) + for dependency in downstream.dependencies + ] == [(upstream.id, True)] + + shown = runner.invoke(list_automations, ["--id", downstream.id]) + assert shown.exit_code == 0, shown.output + assert ( + f"Upstream (ID: {upstream.id}, {upstream.cronstr} in {upstream.timezone}), also when it fails" + in shown.output + ) + shown_upstream = runner.invoke(list_automations, ["--id", upstream.id]) + assert shown_upstream.exit_code == 0, shown_upstream.output + assert "Waited for by" in shown_upstream.output + assert f"Downstream (ID: {downstream.id})" in shown_upstream.output + + # The upstream automation can move to another schedule while another one waits for it. + moved = runner.invoke( + edit_automation, ["--id", str(upstream.id), "--cron", "*/15 * * * *"] + ) + assert moved.exit_code == 0, moved.output + + cleared = runner.invoke( + edit_automation, ["--id", str(downstream.id), "--no-dependencies"] + ) + assert cleared.exit_code == 0, cleared.output + fresh_db.session.expire_all() + assert fresh_db.session.get(Automation, downstream.id).dependencies == [] + + +def test_run_automations_reports_what_it_reconciled( + app, fresh_db, setup_dummy_data_fresh_db, monkeypatch +): + """A tick that fails a killed job's run and skips what waits for it says so, rather than that no automations were due.""" + from flexmeasures.cli import jobs + from flexmeasures.cli.jobs import run_automations + + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + automation = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + run = AutomationRun( + automation=automation, + scheduled_at=automation.cursor, + schedule_revision=automation.schedule_revision, + automation_type=automation.type, + generator_id=automation.generator_id, + dispatch_state="queued", + execution_state="failed", + attempt_count=1, + parameters={}, + plan={}, + last_error_message="The job left the queue without saying how it ended.", + ) + fresh_db.session.add(run) + fresh_db.session.commit() + + def tick(now, report): + report.append( + {"run": run, "event": "reconciled", "message": run.last_error_message} + ) + return [] + + monkeypatch.setattr(jobs, "get_dispatchable_automation_runs", tick) + result = runner.invoke(run_automations) + + assert result.exit_code == 0, result.output + assert "No automations due" not in result.output + assert ( + f"Automation {automation.id} ('Upstream') run {run.id}" in result.output + and "is failed, as its job ended without saying so. The job left the queue without saying how it ended." + in result.output + ) + + +def test_run_automation_with_downstream( + app, fresh_db, setup_dummy_data_fresh_db, clean_redis +): + """--with-downstream runs the automations waiting for this one, too, each holding back until the run it waits for ended.""" + from flexmeasures.cli.jobs import run_one_automation + + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + upstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + added = _add_hourly_automation( + runner, sensor_id, "Downstream", "--depends-on", str(upstream.id) + ) + assert added.exit_code == 0, added.output + + result = runner.invoke( + run_one_automation, ["--automation", str(upstream.id), "--with-downstream"] + ) + + assert result.exit_code == 0, result.output + assert "('Downstream') run" in result.output and "is waiting" in result.output + fresh_db.session.expire_all() + runs = fresh_db.session.scalars( + select(AutomationRun).order_by(AutomationRun.id) + ).all() + assert [(run.automation.name, run.trigger, run.dispatch_state) for run in runs] == [ + ("Upstream", "on-demand", "queued"), + ("Downstream", "on-demand", "waiting"), + ] + + +def test_deleting_an_automation_warns_about_what_waits_for_it( + app, fresh_db, setup_dummy_data_fresh_db +): + """Without --force, the prompt names the automations that lose what they wait for.""" + from flexmeasures.cli.data_delete import delete_automation + + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + upstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + added = _add_hourly_automation( + runner, sensor_id, "Downstream", "--depends-on", str(upstream.id) + ) + assert added.exit_code == 0, added.output + + result = runner.invoke(delete_automation, ["--id", str(upstream.id)], input="n\n") + + assert "'Downstream'" in result.output + assert "will run without it" in result.output + assert fresh_db.session.get(Automation, upstream.id) is not None + + +def test_run_automation_with_upstream( + app, fresh_db, setup_dummy_data_fresh_db, clean_redis +): + """--with-upstream runs what this automation waits for first, and --exclude keeps one of those from running.""" + from flexmeasures.cli.jobs import run_one_automation + + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + upstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + added = _add_hourly_automation( + runner, sensor_id, "Downstream", "--depends-on", str(upstream.id) + ) + assert added.exit_code == 0, added.output + downstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Downstream") + ).one() + + result = runner.invoke( + run_one_automation, + ["--automation", str(downstream.id), "--with-upstream"], + ) + assert result.exit_code == 0, result.output + assert f"Automation {upstream.id} ('Upstream')" in result.output + assert f"Automation {downstream.id} ('Downstream')" in result.output + assert "is waiting" in result.output + + excluded = runner.invoke( + run_one_automation, + [ + "--automation", + str(downstream.id), + "--with-upstream", + "--exclude", + str(upstream.id), + ], + ) + assert excluded.exit_code == 0, excluded.output + assert ( + f"Automation {upstream.id} ('Upstream') does not run. You left it out." + in excluded.output + ) + assert f"Automation {upstream.id} ('Upstream') run" not in excluded.output + + both = runner.invoke( + run_one_automation, + ["--automation", str(upstream.id), "--with-upstream", "--with-downstream"], + ) + assert both.exit_code != 0 + assert "either" in both.output + + alone = runner.invoke( + run_one_automation, ["--automation", str(upstream.id), "--exclude", "1"] + ) + assert alone.exit_code != 0 + + +def test_add_automation_allow_failure_of_needs_depends_on( + app, fresh_db, setup_dummy_data_fresh_db +): + runner = app.test_cli_runner() + sensor_id = setup_dummy_data_fresh_db[0] + assert _add_hourly_automation(runner, sensor_id, "Upstream").exit_code == 0 + upstream = fresh_db.session.scalars( + select(Automation).filter_by(name="Upstream") + ).one() + + result = _add_hourly_automation( + runner, sensor_id, "Downstream", "--allow-failure-of", str(upstream.id) + ) + + assert result.exit_code == 2, result.output + assert "which --depends-on does not" in result.output + assert ( + fresh_db.session.scalars( + select(Automation).filter_by(name="Downstream") + ).one_or_none() + is None + ) diff --git a/flexmeasures/cli/utils.py b/flexmeasures/cli/utils.py index b6e6f17d3b..070ab7362d 100644 --- a/flexmeasures/cli/utils.py +++ b/flexmeasures/cli/utils.py @@ -285,6 +285,26 @@ def validate_unique(ctx, param, value): return value +def assemble_automation_dependencies( + depends_on: tuple, allow_failure_of: tuple +) -> list[dict[str, Any]]: + """Combine the automations given with --depends-on and --allow-failure-of into the dependencies an automation is given. + + :raises click.UsageError: if --allow-failure-of names an automation that --depends-on does not. + """ + upstream_ids = {upstream.id for upstream in depends_on} + stray_ids = sorted({upstream.id for upstream in allow_failure_of} - upstream_ids) + if stray_ids: + raise click.UsageError( + f"--allow-failure-of names automation(s) {', '.join(map(str, stray_ids))}, which --depends-on does not." + ) + allowed_ids = {upstream.id for upstream in allow_failure_of} + return [ + {"upstream": upstream, "allow_failure": upstream.id in allowed_ids} + for upstream in depends_on + ] + + def abort(message: str): click.secho(message, **MsgStyle.ERROR) raise click.Abort() diff --git a/flexmeasures/data/migrations/versions/b8e4c2d7f913_add_automation_dependencies.py b/flexmeasures/data/migrations/versions/b8e4c2d7f913_add_automation_dependencies.py new file mode 100644 index 0000000000..6247b53987 --- /dev/null +++ b/flexmeasures/data/migrations/versions/b8e4c2d7f913_add_automation_dependencies.py @@ -0,0 +1,98 @@ +"""add automation dependencies + +Let an automation's runs wait for the latest run of another automation due at or before their own scheduled time. +A run that waits is 'waiting' to be dispatched, and a run whose upstream run failed is 'skipped'. + +Revision ID: b8e4c2d7f913 +Revises: bf22b21d0990 +Create Date: 2026-10-14 10:00:00.000000 + +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "b8e4c2d7f913" +down_revision = "bf22b21d0990" +branch_labels = None +depends_on = None + +DISPATCH_STATE_CK = "automation_run_automation_run_dispatch_state_ck" +EXECUTION_STATE_CK = "automation_run_automation_run_execution_state_ck" + + +def upgrade(): + op.create_table( + "automation_dependency", + sa.Column("automation_id", sa.Integer(), nullable=False), + sa.Column("upstream_id", sa.Integer(), nullable=False), + sa.Column( + "allow_failure", + sa.Boolean(), + nullable=False, + server_default=sa.false(), + ), + sa.ForeignKeyConstraint( + ["automation_id"], + ["automation.id"], + name=op.f("automation_dependency_automation_id_automation_fkey"), + ondelete="CASCADE", + ), + sa.ForeignKeyConstraint( + ["upstream_id"], + ["automation.id"], + name=op.f("automation_dependency_upstream_id_automation_fkey"), + ondelete="CASCADE", + ), + sa.PrimaryKeyConstraint( + "automation_id", "upstream_id", name=op.f("automation_dependency_pkey") + ), + ) + op.alter_column("automation_dependency", "allow_failure", server_default=None) + op.create_index( + op.f("automation_dependency_upstream_id_idx"), + "automation_dependency", + ["upstream_id"], + ) + + op.drop_constraint(op.f(DISPATCH_STATE_CK), "automation_run", type_="check") + op.create_check_constraint( + op.f(DISPATCH_STATE_CK), + "automation_run", + "dispatch_state IN ('pending', 'claimed', 'waiting', 'partially_queued', 'queued', 'failed')", + ) + op.drop_constraint(op.f(EXECUTION_STATE_CK), "automation_run", type_="check") + op.create_check_constraint( + op.f(EXECUTION_STATE_CK), + "automation_run", + "execution_state IN ('pending', 'running', 'succeeded', 'failed', 'canceled', 'skipped')", + ) + + +def downgrade(): + # The narrower constraints below would refuse these runs, so give them the states they had before a run could wait. + # A waiting run was claimed and never queued anything, and a skipped run never ran. + op.execute( + "UPDATE automation_run SET dispatch_state = 'failed' WHERE dispatch_state = 'waiting'" + ) + op.execute( + "UPDATE automation_run SET execution_state = 'canceled' WHERE execution_state = 'skipped'" + ) + op.drop_constraint(op.f(EXECUTION_STATE_CK), "automation_run", type_="check") + op.create_check_constraint( + op.f(EXECUTION_STATE_CK), + "automation_run", + "execution_state IN ('pending', 'running', 'succeeded', 'failed', 'canceled')", + ) + op.drop_constraint(op.f(DISPATCH_STATE_CK), "automation_run", type_="check") + op.create_check_constraint( + op.f(DISPATCH_STATE_CK), + "automation_run", + "dispatch_state IN ('pending', 'claimed', 'partially_queued', 'queued', 'failed')", + ) + op.drop_index( + op.f("automation_dependency_upstream_id_idx"), + table_name="automation_dependency", + ) + op.drop_table("automation_dependency") diff --git a/flexmeasures/data/migrations/versions/c3d9f6a2e481_record_on_demand_automation_runs.py b/flexmeasures/data/migrations/versions/c3d9f6a2e481_record_on_demand_automation_runs.py new file mode 100644 index 0000000000..3720bc89c1 --- /dev/null +++ b/flexmeasures/data/migrations/versions/c3d9f6a2e481_record_on_demand_automation_runs.py @@ -0,0 +1,61 @@ +"""record on-demand automation runs + +Say whether an automation run was due by its schedule or asked for on demand, +so that a chain of automations run on demand can wait for each other's runs, too. +Only one run per scheduled occurrence is allowed, as before, while on-demand runs can share a time with any other run. + +Revision ID: c3d9f6a2e481 +Revises: b8e4c2d7f913 +Create Date: 2026-10-30 03:00:00.000000 + +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "c3d9f6a2e481" +down_revision = "b8e4c2d7f913" +branch_labels = None +depends_on = None + +TRIGGER_CK = "automation_run_automation_run_trigger_ck" + + +def upgrade(): + op.add_column( + "automation_run", + sa.Column( + "trigger", + sa.String(16), + nullable=False, + server_default="schedule", + ), + ) + op.alter_column("automation_run", "trigger", server_default=None) + op.create_check_constraint( + op.f(TRIGGER_CK), + "automation_run", + "trigger IN ('schedule', 'on-demand')", + ) + op.drop_constraint("automation_run_occurrence_uq", "automation_run", type_="unique") + op.create_index( + "automation_run_occurrence_uq", + "automation_run", + ["automation_id", "scheduled_at", "schedule_revision"], + unique=True, + postgresql_where=sa.text("trigger = 'schedule'"), + ) + + +def downgrade(): + # Without the trigger, on-demand runs could clash with scheduled runs at the same time, so they go. + op.execute("DELETE FROM automation_run WHERE trigger = 'on-demand'") + op.drop_index("automation_run_occurrence_uq", table_name="automation_run") + op.create_unique_constraint( + "automation_run_occurrence_uq", + "automation_run", + ["automation_id", "scheduled_at", "schedule_revision"], + ) + op.drop_constraint(op.f(TRIGGER_CK), "automation_run", type_="check") + op.drop_column("automation_run", "trigger") diff --git a/flexmeasures/data/migrations/versions/d5b8f3a1c962_record_automation_revision_start.py b/flexmeasures/data/migrations/versions/d5b8f3a1c962_record_automation_revision_start.py new file mode 100644 index 0000000000..bf58f55b21 --- /dev/null +++ b/flexmeasures/data/migrations/versions/d5b8f3a1c962_record_automation_revision_start.py @@ -0,0 +1,52 @@ +"""record when an automation's recurrence revision started + +Remember the first minute the current revision of an automation's recurrence is due in, +so that a run waiting for the automation can tell a time from before a reset of its recurrence, +at which it was not due, from a time it was due at but moved past. + +An automation still on its first revision started in the minute it was created. +For one whose recurrence was reset since, the start is not recorded anywhere, +so it is taken to be its first scheduled run of the current revision, or else the minute after its cursor, which a reset puts just before the start. + +Revision ID: d5b8f3a1c962 +Revises: c3d9f6a2e481 +Create Date: 2026-11-15 23:30:00.000000 + +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "d5b8f3a1c962" +down_revision = "c3d9f6a2e481" +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column( + "automation", + sa.Column("revision_started_at", sa.DateTime(timezone=True), nullable=True), + ) + op.execute(""" + UPDATE automation + SET revision_started_at = CASE + WHEN schedule_revision = 1 THEN date_trunc('minute', created_at) + ELSE COALESCE( + ( + SELECT min(automation_run.scheduled_at) + FROM automation_run + WHERE automation_run.automation_id = automation.id + AND automation_run.schedule_revision = automation.schedule_revision + AND automation_run.trigger = 'schedule' + ), + cursor + interval '1 minute' + ) + END + """) + op.alter_column("automation", "revision_started_at", nullable=False) + + +def downgrade(): + op.drop_column("automation", "revision_started_at") diff --git a/flexmeasures/data/models/automations.py b/flexmeasures/data/models/automations.py index cbba34f811..34ef922cd5 100644 --- a/flexmeasures/data/models/automations.py +++ b/flexmeasures/data/models/automations.py @@ -39,9 +39,12 @@ def get_default_automation_timezone(asset=None) -> str: def get_initial_cursor() -> datetime: """Return a cursor which keeps the automation's creation minute eligible.""" - return server_now().astimezone(timezone.utc).replace( - second=0, microsecond=0 - ) - timedelta(minutes=1) + return get_revision_start() - timedelta(minutes=1) + + +def get_revision_start() -> datetime: + """Return the minute an automation's recurrence starts from, when it is created or reset, which is the current minute.""" + return server_now().astimezone(timezone.utc).replace(second=0, microsecond=0) class Automation(db.Model, AuthModelMixin): @@ -94,6 +97,13 @@ class Automation(db.Model, AuthModelMixin): default=get_initial_cursor, ) schedule_revision = db.Column(db.Integer, nullable=False, default=1) + # The first minute the current revision of the recurrence is due in, set on creation and on each reset (see `update_automation`). + # A time before it was not due by this recurrence, so a run waiting for this automation at that time goes ahead without it. + revision_started_at = db.Column( + db.DateTime(timezone=True), + nullable=False, + default=get_revision_start, + ) active = db.Column(db.Boolean, nullable=False, default=True) generator_id = db.Column( db.Integer, db.ForeignKey("data_source.id"), nullable=False @@ -123,6 +133,16 @@ class Automation(db.Model, AuthModelMixin): passive_deletes=True, order_by="desc(AutomationRun.scheduled_at)", ) + # The automations whose latest run due at or before its own scheduled time each run of this one waits for (see `AutomationDependency`). + dependencies = db.relationship( + "AutomationDependency", + foreign_keys="AutomationDependency.automation_id", + back_populates="automation", + lazy=True, + cascade="all, delete-orphan", + passive_deletes=True, + order_by="AutomationDependency.upstream_id", + ) @validates("timezone") def validate_timezone(self, key: str, timezone: str) -> str: @@ -183,6 +203,42 @@ def output_sensors(self) -> list: return get_automation_sensors(self)["output_sensors"] +class AutomationDependency(db.Model): + """One automation that another automation's runs wait for. + + A run of the dependent automation is dispatched only once the upstream automation's latest run due at or before its own scheduled time has ended, + and is skipped if that run failed, unless the dependency allows for failure. + """ + + __tablename__ = "automation_dependency" + + automation_id = db.Column( + db.Integer, + db.ForeignKey("automation.id", ondelete="CASCADE"), + primary_key=True, + ) + upstream_id = db.Column( + db.Integer, + db.ForeignKey("automation.id", ondelete="CASCADE"), + primary_key=True, + index=True, + ) + # Whether the dependent run goes ahead when the upstream run failed, rather than being skipped. + allow_failure = db.Column(db.Boolean, nullable=False, default=False) + + automation = db.relationship( + "Automation", foreign_keys=[automation_id], back_populates="dependencies" + ) + upstream = db.relationship("Automation", foreign_keys=[upstream_id]) + + def __repr__(self): + return "" % ( + self.automation_id, + self.upstream_id, + ", also when it fails" if self.allow_failure else "", + ) + + # A job intent counts as dispatched from this status onwards: it is in Redis, whatever became of it since. AUTOMATION_RUN_JOB_QUEUED_OR_LATER = ( "queued", @@ -194,22 +250,29 @@ def output_sensors(self) -> list: class AutomationRun(db.Model): - """Durable execution record for one scheduled automation occurrence.""" + """Durable execution record for one run of an automation, scheduled or on demand.""" __tablename__ = "automation_run" __table_args__ = ( - db.UniqueConstraint( + # One run per scheduled occurrence, while on-demand runs can share a time with any other run. + db.Index( + "automation_run_occurrence_uq", "automation_id", "scheduled_at", "schedule_revision", - name="automation_run_occurrence_uq", + unique=True, + postgresql_where=db.text("trigger = 'schedule'"), + ), + db.CheckConstraint( + "trigger IN ('schedule', 'on-demand')", + name="automation_run_trigger_ck", ), db.CheckConstraint( - "dispatch_state IN ('pending', 'claimed', 'partially_queued', 'queued', 'failed')", + "dispatch_state IN ('pending', 'claimed', 'waiting', 'partially_queued', 'queued', 'failed')", name="automation_run_dispatch_state_ck", ), db.CheckConstraint( - "execution_state IN ('pending', 'running', 'succeeded', 'failed', 'canceled')", + "execution_state IN ('pending', 'running', 'succeeded', 'failed', 'canceled', 'skipped')", name="automation_run_execution_state_ck", ), ) @@ -231,9 +294,13 @@ class AutomationRun(db.Model): ) scheduled_at = db.Column(db.DateTime(timezone=True), nullable=False) schedule_revision = db.Column(db.Integer, nullable=False) + # Whether the run was due by the automation's schedule, or asked for on demand along with a chain of runs (see `run_automation_chain`). + trigger = db.Column(db.String(16), nullable=False, default="schedule") automation_type = db.Column(db.String(80), nullable=False) generator_id = db.Column(db.Integer, nullable=True) + # A run that is 'waiting' holds back its jobs until the upstream runs it depends on have ended (see `AutomationDependency`). dispatch_state = db.Column(db.String(32), nullable=False, default="pending") + # A run that is 'skipped' queued no jobs, because an upstream run it depends on did not succeed. execution_state = db.Column(db.String(32), nullable=False, default="pending") claim_owner = db.Column(db.String(128), nullable=True) claimed_at = db.Column(db.DateTime(timezone=True), nullable=True) diff --git a/flexmeasures/data/schemas/automations.py b/flexmeasures/data/schemas/automations.py index b272d83a80..e92c894d91 100644 --- a/flexmeasures/data/schemas/automations.py +++ b/flexmeasures/data/schemas/automations.py @@ -67,6 +67,58 @@ def _serialize(self, automation, attr, data, **kwargs): return automation.id +class ReadableAutomationIdField(AutomationIdField): + """Field that only finds an automation the current user may read. + + One they may not read gets the answer of one that does not exist, so that trying IDs does not reveal which ones do. + Nobody is logged in on the command line, where this makes no difference. + """ + + def _deserialize(self, value, attr, obj, **kwargs) -> Automation: + from flask_security import current_user + from werkzeug.exceptions import Forbidden, Unauthorized + + from flexmeasures.auth.policy import check_access + + automation = super()._deserialize(value, attr, obj, **kwargs) + if current_user and current_user.is_authenticated: + try: + check_access(automation, "read") + except (Forbidden, Unauthorized): + raise FMValidationError(f"No automation found with id {automation.id}.") + return automation + + +class AutomationDependencySchema(Schema): + """One automation whose runs another automation's runs wait for.""" + + # The loaded names are the ones `set_automation_dependencies` takes. + upstream = ReadableAutomationIdField( + required=True, + data_key="automation", + metadata={ + "description": "ID of the automation to wait for. It may recur on another schedule.", + "example": 4, + }, + ) + allow_failure = fields.Bool( + load_default=False, + data_key="allow-failure", + metadata={ + "description": "Whether a run goes ahead when the upstream run failed or did not happen, rather than being skipped.", + "example": False, + }, + ) + + +DEPENDS_ON_DESCRIPTION = ( + "The automations whose runs each run of this automation waits for: their latest runs due at or before its own scheduled time." + " A run queues its jobs once these have ended, and is skipped if one of them failed, unless its dependency allows for failure." + " Upstream automations can be on any asset you can read, and may recur on another schedule." + " A run on demand does not wait." +) + + class AutomationCreationSchema(Schema): """Request schema for creating an automation (the asset comes from the URL path). @@ -122,6 +174,15 @@ class AutomationCreationSchema(Schema): "example": 6, }, ) + dependencies = fields.List( + fields.Nested(AutomationDependencySchema), + data_key="depends-on", + load_default=list, + metadata={ + "description": DEPENDS_ON_DESCRIPTION, + "example": [{"automation": 4, "allow-failure": False}], + }, + ) @validates_schema def validate_generator_is_named_once(self, data: dict, **kwargs): @@ -151,7 +212,7 @@ def validate_generator_is_named_once(self, data: dict, **kwargs): class AutomationUpdateSchema(Schema): - """Request schema for updating an automation's name, recurrence, timezone and/or activation status. + """Request schema for updating an automation's name, recurrence, timezone, activation status and/or dependencies. The parameters cannot be updated, so the sensors an automation involves stay the ones its creator was checked against. """ @@ -165,6 +226,15 @@ class AutomationUpdateSchema(Schema): } ) active = fields.Bool() + dependencies = fields.List( + fields.Nested(AutomationDependencySchema), + data_key="depends-on", + metadata={ + "description": DEPENDS_ON_DESCRIPTION + + " Replaces the automations it depended on; send an empty list to have its runs wait for none.", + "example": [{"automation": 4, "allow-failure": True}], + }, + ) class AutomationSchema(ma.SQLAlchemySchema): @@ -211,6 +281,24 @@ class Meta: }, ) active = ma.auto_field() + depends_on = fields.Method( + serialize="dump_depends_on", + data_key="depends-on", + dump_only=True, + metadata={ + "description": DEPENDS_ON_DESCRIPTION + + " Each upstream automation you may read comes with its `name`, `asset` and `asset-name`.", + "example": [ + { + "automation": 4, + "allow-failure": False, + "name": "Pull meter data", + "asset": 2, + "asset-name": "Site", + } + ], + }, + ) @staticmethod def _in_automation_timezone( @@ -237,6 +325,14 @@ def dump_cursor(self, automation: Automation) -> str | None: """Render the cursor in the automation's own timezone.""" return self._in_automation_timezone(automation, automation.cursor) + def dump_depends_on(self, automation: Automation) -> list[dict]: + """Render the automations this automation's runs wait for.""" + from flexmeasures.data.services.automations import ( + serialize_automation_dependencies, + ) + + return serialize_automation_dependencies(automation) + @validates("type") def validate_type(self, type: str, **kwargs): validate_automation_type(type) diff --git a/flexmeasures/data/services/automation_copies.py b/flexmeasures/data/services/automation_copies.py index 2d0eba027a..f8df865d53 100644 --- a/flexmeasures/data/services/automation_copies.py +++ b/flexmeasures/data/services/automation_copies.py @@ -29,6 +29,7 @@ from flexmeasures.data.services.automations import ( WINDOW_FIELDS, get_forecast_output_sensor, + set_automation_dependencies, validate_automation_output_scope, ) from flexmeasures.data.services.data_sources import get_or_create_source @@ -92,6 +93,8 @@ def copy_automations( with its references to sensors inside the subtree pointed at the new sensors. A copy keeps its name, type, cron expression and timezone, but starts inactive and with a fresh cursor, so that it inherits neither the original's run history nor its queued jobs. + A copy waits for the copies of the automations its original waits for, where those were copied, too, + and for the very automations its original waits for otherwise. An automation that cannot be copied safely is skipped rather than failing the asset copy: the copied asset, its sensors and the other automations are kept. @@ -120,10 +123,11 @@ def copy_automations( ) skipped: list[SkippedAutomation] = [] + copies: dict[int, Automation] = {} for automation in source_automations: try: with db.session.begin_nested(): - _copy_automation( + copies[automation.id] = _copy_automation( automation=automation, asset_id_map=asset_id_map, sensor_id_map=sensor_id_map, @@ -148,9 +152,37 @@ def copy_automations( reason=str(e), ) ) + _copy_dependencies(source_automations, copies) return skipped +def _copy_dependencies( + source_automations: list[Automation], copies: dict[int, Automation] +) -> None: + """Have each copy wait for what its original waits for, pointing at the copies where there are any. + + This happens once all automations were copied, as an automation may wait for one that was copied after it. + The copies recur on the schedules of their originals, and a copied set of dependencies cannot form a cycle the originals did not, + so the dependencies need no validation beyond what the originals had. + """ + for automation in source_automations: + copied_automation = copies.get(automation.id) + if copied_automation is None or not automation.dependencies: + continue + set_automation_dependencies( + copied_automation, + [ + { + "upstream": copies.get(dependency.upstream_id, dependency.upstream), + "allow_failure": dependency.allow_failure, + } + for dependency in automation.dependencies + ], + origin="asset copy", + ) + db.session.flush() + + def _copy_automation( automation: Automation, asset_id_map: dict[int, int], @@ -164,7 +196,7 @@ def _copy_automation( if automation.type not in COPYABLE_AUTOMATION_TYPES: raise AutomationNotCopyable( f"An automation of type '{automation.type}' cannot be copied yet:" - " its parameters hold sensor references that a copy cannot point at the copied sensors." + " its flex config can hold sensor references that a copy cannot point at the copied sensors." ) data_generator = _load_data_generator(automation) remapper = _ReferenceRemapper( diff --git a/flexmeasures/data/services/automations.py b/flexmeasures/data/services/automations.py index 5ef282abf5..66fc497f43 100644 --- a/flexmeasures/data/services/automations.py +++ b/flexmeasures/data/services/automations.py @@ -4,9 +4,12 @@ from __future__ import annotations +from collections import defaultdict +from contextvars import ContextVar from contextlib import contextmanager import functools from copy import copy, deepcopy +from collections.abc import Callable, Collection from dataclasses import dataclass from datetime import datetime, timedelta, timezone import os @@ -23,20 +26,22 @@ from flask import current_app from marshmallow import ValidationError from rq import get_current_job +from redis.exceptions import RedisError from rq.job import Job, JobStatus from sqlalchemy import func, or_, select, update -from sqlalchemy.exc import IntegrityError +from sqlalchemy.exc import IntegrityError, SQLAlchemyError from sqlalchemy.orm import selectinload from flexmeasures import Forecaster, Reporter from flexmeasures.data import db from flexmeasures.data.models.automations import ( Automation, + AutomationDependency, AutomationRun, AutomationRunAttempt, AutomationRunJob, get_default_automation_timezone, - get_initial_cursor, + get_revision_start, ) from flexmeasures.data.models.data_sources import DataSource from flexmeasures.data.models.time_series import Sensor @@ -54,6 +59,32 @@ from flexmeasures.utils.time_utils import apply_offset_chain, get_timezone, server_now AUTOMATION_RUN_CLAIM_LEASE = timedelta(minutes=10) +# The error type recorded on a waiting run canceled as its automation was deactivated (see `_cancel_waiting_scheduled_runs`). +AUTOMATION_DEACTIVATED = "AutomationDeactivated" +# What a runner's tick did beyond claiming runs, while it collects that (see `get_dispatchable_automation_runs`). +_tick_report: ContextVar[list[dict[str, Any]] | None] = ContextVar( + "automation_tick_report", default=None +) + + +def _tick_report_position() -> int: + """Say where to note what becomes of a run in a runner's tick, before what follows from it is noted (see `_report_to_tick`).""" + report = _tick_report.get() + return len(report) if report is not None else 0 + + +def _report_to_tick( + run: AutomationRun, event: str, message: str | None, position: int +) -> None: + """Note what became of a run during a runner's tick, if a tick is collecting that. + + Ending a run releases what waits for it, which is noted first, so this goes back to where the run's turn was (see `_tick_report_position`). + """ + report = _tick_report.get() + if report is not None: + report.insert(position, {"run": run, "event": event, "message": message}) + + # How often a runner may attempt one run's dispatch, and how long it waits between attempts. # A dispatch which keeps failing usually fails for a reason no retry fixes, such as a sensor that was deleted, # so the attempts are spaced further apart each time, and run out. @@ -73,6 +104,10 @@ "queued", "failed", ) +# The automation types whose runs are dispatched a second time, after an attempt that failed. +# A forecast run's jobs carry IDs derived from the run, so a retry recognizes the ones it already queued. +# A schedule run's jobs get a fresh ID on every dispatch, so retrying one would duplicate its schedules, +# which is why such a run is recorded and reported, but left where it failed. # The automation types whose runs are dispatched again after a failed dispatch. @@ -266,12 +301,16 @@ def claim_existing_automation_run( run: AutomationRun, owner: str | None = None, lease: timedelta = AUTOMATION_RUN_CLAIM_LEASE, + dispatch_states: tuple[str, ...] = AUTOMATION_RUN_RESUMABLE_DISPATCH_STATES, ) -> ClaimedAutomationRun | None: """Claim a durable automation run whose dispatch is unfinished and unclaimed. A run is only up for grabs once no other runner holds a live claim on it, because the dispatch state turns to 'partially_queued' while the owning runner is still queueing the rest of its jobs. A runner which fails releases its own claim, so its run is immediately retryable. + + :param dispatch_states: the dispatch states the run may be in to be claimed; + a run that is 'waiting' for its upstream runs is claimed only once they ended (see `release_automation_runs_waiting_for`). """ owner = owner or _runner_owner() now = _now_utc() @@ -279,7 +318,7 @@ def claim_existing_automation_run( update(AutomationRun) .where( AutomationRun.id == run.id, - AutomationRun.dispatch_state.in_(AUTOMATION_RUN_RESUMABLE_DISPATCH_STATES), + AutomationRun.dispatch_state.in_(dispatch_states), AutomationRun.dispatch_completed_at.is_(None), AutomationRun.attempt_count < AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS, _claim_is_available(now), @@ -308,11 +347,28 @@ def claim_existing_automation_run( def get_dispatchable_automation_runs( now: datetime | None = None, owner: str | None = None, + report: list[dict[str, Any]] | None = None, ) -> list[ClaimedAutomationRun]: - """Claim new due occurrences and resumable durable runs for dispatch.""" + """Claim new due occurrences, resumable durable runs, and waiting runs whose upstream runs ended, for dispatch. + + A waiting run is normally dispatched as soon as the last upstream run it waits for ends (see `release_automation_runs_waiting_for`). + Looking at the waiting runs here, too, catches the ones whose release was missed, + for instance because the worker that ended the upstream run went down before it got to them. + A run whose job was killed never gets to say so, so that is looked up first (see `reconcile_unfinished_automation_jobs`). + + :param report: a list to add what else became of runs along the way to, + each as a dict with the 'run', the 'event' ('reconciled', 'abandoned' or 'released') and a 'message', + such as a run failed for a killed job, and the run waiting for it, skipped as a result. + """ if now is None: now = _now_utc() now = floor_to_minute(now) + token = _tick_report.set(report) + try: + reconcile_unfinished_automation_jobs() + give_up_abandoned_on_demand_runs(now) + finally: + _tick_report.reset(token) claimed_runs: list[ClaimedAutomationRun] = [] for due_automation in get_due_automations(now): claimed = claim_due_automation_run(due_automation, owner=owner) @@ -324,6 +380,8 @@ def get_dispatchable_automation_runs( .join(Automation) .where( Automation.active.is_(True), + # Whoever asked for a run on demand is told when it fails, and can ask again. + AutomationRun.trigger == "schedule", AutomationRun.automation_type.in_(AUTOMATION_RUN_RETRIED_TYPES), AutomationRun.dispatch_state.in_(AUTOMATION_RUN_RESUMABLE_DISPATCH_STATES), AutomationRun.dispatch_completed_at.is_(None), @@ -340,9 +398,456 @@ def get_dispatchable_automation_runs( claimed = claim_existing_automation_run(run, owner=owner) if claimed is not None: claimed_runs.append(claimed) + + waiting_runs = db.session.scalars( + select(AutomationRun) + .join(Automation) + .where( + # A run on demand is asked for, so it goes ahead for an inactive automation, too. + or_(Automation.active.is_(True), AutomationRun.trigger == "on-demand"), + AutomationRun.dispatch_state == "waiting", + _claim_is_available(now), + ) + .order_by(AutomationRun.scheduled_at, AutomationRun.id) + ).all() + for run in waiting_runs: + if check_upstream_runs(run)[0] == "wait": + continue + claimed = claim_existing_automation_run( + run, owner=owner, dispatch_states=("waiting",) + ) + if claimed is not None: + claimed_runs.append(claimed) return claimed_runs +def give_up_abandoned_on_demand_runs(now: datetime) -> None: + """Fail the runs on demand whose dispatch was abandoned, so that what waits for them is not left waiting forever. + + Whoever asked for a run on demand claims it, and lets go of it once its jobs are queued. + One that stopped halfway, such as an API process that went down between recording a chain of runs and queueing the first, + leaves its claim to expire, and as the runner does not retry runs on demand (see `get_dispatchable_automation_runs`), nothing else would end it. + """ + for run in db.session.scalars( + select(AutomationRun) + .where( + AutomationRun.trigger == "on-demand", + AutomationRun.dispatch_state.in_(("claimed", "partially_queued")), + AutomationRun.dispatch_completed_at.is_(None), + AutomationRun.claim_owner.is_not(None), + AutomationRun.claim_expires_at <= now, + ) + .order_by(AutomationRun.id) + ).all(): + position = _tick_report_position() + if run.dispatch_state == "claimed": + run.dispatch_state = "failed" + run.claim_owner = None + run.claim_expires_at = None + run.last_error_type = "DispatchAbandoned" + run.last_error_message = "Whoever asked for this run stopped before it queued all of its jobs, and a run on demand is not retried." + db.session.commit() + _report_to_tick(run, "abandoned", run.last_error_message, position) + _release_runs_waiting_for_ended_run(run) + + +def _dispatch_was_given_up(run: AutomationRun) -> bool: + """Whether a run's dispatch failed for good, so that it will never queue (the rest of) its jobs.""" + return ( + run.dispatch_completed_at is None + and run.dispatch_state in ("failed", "partially_queued") + # A runner that is still queueing the jobs of a run holds a claim on it, and may yet finish. + and run.claim_owner is None + and ( + run.trigger == "on-demand" + or run.automation_type not in AUTOMATION_RUN_RETRIED_TYPES + or run.attempt_count >= AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS + ) + ) + + +def _upstream_run_state(run: AutomationRun) -> str: + """Say whether an upstream run 'succeeded', 'failed', or is still 'pending', as far as the runs waiting for it are concerned. + + A run that was skipped, or canceled, did not deliver what the runs waiting for it need, so it counts as failed. + So does a run whose dispatch was given up, as it will never end otherwise. + A run canceled as its automation was deactivated counts as 'not-scheduled', as if the automation had been inactive at that time. + """ + if run.last_error_type == AUTOMATION_DEACTIVATED: + return "not-scheduled" + if run.execution_completed_at is not None: + return "succeeded" if run.execution_state == "succeeded" else "failed" + if _dispatch_was_given_up(run): + return "failed" + return "pending" + + +def _state_without_upstream_run(upstream: Automation, due_at: datetime) -> str: + """Say what it means that an upstream automation has no run for the time it was due at. + + - 'pending': the upstream automation is due at that time, but no runner claimed its run yet. + - 'passed-over': the upstream automation was due at that time, but moved past it without a run, + as missed runs are coalesced or left out (see `get_due_automations`). + - 'not-scheduled': the upstream automation was never going to run at that time, + as it is inactive, or as the time comes before its current recurrence started, + which is when it was created, or when its recurrence was last reset (see `update_automation`). + """ + if not upstream.active or due_at < upstream.revision_started_at: + return "not-scheduled" + if upstream.cursor is None or upstream.cursor < due_at: + return "pending" + return "passed-over" + + +def _get_catch_up_run(upstream: Automation, due_at: datetime) -> AutomationRun | None: + """Find the scheduled run that an upstream automation caught up with after the time it was due at, if any. + + Runs missed while the runner was down are coalesced into the latest one (see `get_due_automations`), + which is the first run of the same revision after the missed time. + """ + return db.session.scalars( + select(AutomationRun) + .where( + AutomationRun.automation_id == upstream.id, + AutomationRun.trigger == "schedule", + AutomationRun.schedule_revision == upstream.schedule_revision, + AutomationRun.scheduled_at > due_at, + ) + .order_by(AutomationRun.scheduled_at) + .limit(1) + ).first() + + +def get_upstream_due_at( + upstream: Automation, scheduled_at: datetime +) -> datetime | None: + """Return when the upstream run that a run scheduled at this time waits for is due. + + That is the latest time the upstream automation is due at, by its own schedule, up to and including the time the waiting run is scheduled at. + For instance, an hourly run at 10:00 waits for the run of a quarter-hourly upstream automation due at 10:00, + and a quarter-hourly run at 10:15 waits for the run of an hourly upstream automation due at 10:00. + + :returns: None if the upstream automation's schedule cannot be read, such as for a stale timezone. + """ + try: + return get_latest_scheduled_run(upstream, scheduled_at) + except (CroniterError, ValueError, ZoneInfoNotFoundError): + return None + + +def check_upstream_runs(run: AutomationRun) -> tuple[str, list[dict[str, Any]]]: + """Decide whether a run can be dispatched yet, going by the runs of the automations it waits for. + + Each upstream automation's latest run due at or before the run's own scheduled time counts (see `get_upstream_due_at`), + so the automations need not recur on the same schedule. + The run is skipped as soon as one of those failed or will not happen, unless its dependency allows for that, + as its jobs would compute with input that was never produced. + Otherwise, it waits for as long as one of them has yet to end. + An upstream automation that was never going to run at that time is left out, and the run goes ahead without it. + + A run on demand only waits for the runs that were asked for along with it (see `run_automation_chain`). + + :returns: 'wait', 'skip' or 'go', and for each dependency, what was found: + the upstream automation, whether its failure is allowed for, its run (if any) and the state of that run. + """ + if run.trigger == "on-demand": + found = _find_chain_upstream_runs(run) + return _upstream_verdict(found), found + found = [] + for dependency in run.automation.dependencies: + due_at = get_upstream_due_at(dependency.upstream, run.scheduled_at) + upstream_run = ( + db.session.scalars( + select(AutomationRun) + .where( + AutomationRun.automation_id == dependency.upstream_id, + AutomationRun.trigger == "schedule", + AutomationRun.scheduled_at == due_at, + ) + .order_by(AutomationRun.schedule_revision.desc()) + .limit(1) + ).first() + if due_at is not None + else None + ) + state = None + if upstream_run is None and due_at is not None: + state = _state_without_upstream_run(dependency.upstream, due_at) + if state == "passed-over": + # After an outage, the run it caught up with does what the missed run would have, so it is waited for instead. + upstream_run = _get_catch_up_run(dependency.upstream, due_at) + if upstream_run is not None: + state = _upstream_run_state(upstream_run) + elif due_at is None: + state = "not-scheduled" + found.append( + { + "automation": dependency.upstream_id, + "allow-failure": dependency.allow_failure, + "due-at": due_at.isoformat() if due_at is not None else None, + "run": upstream_run.id if upstream_run is not None else None, + "state": state, + "summary": _summarize_upstream_run( + dependency.upstream, state, due_at, upstream_run + ), + } + ) + return _upstream_verdict(found), found + + +def _format_due_at(upstream: Automation, due_at: datetime) -> str: + """Write a time on an upstream automation's own clock, for a message.""" + try: + return due_at.astimezone(ZoneInfo(upstream.timezone)).strftime( + "%Y-%m-%d %H:%M %Z" + ) + except ZoneInfoNotFoundError: + return due_at.isoformat() + + +def _summarize_upstream_run( + upstream: Automation | None, + state: str, + due_at: datetime | None, + upstream_run: AutomationRun | None, +) -> str: + """Say in a sentence what became of the upstream run that a run waits for, such as "Solar forecast (automation 3, run 4) failed.". + + :param upstream: the upstream automation, or None if it is gone. + :param state: the state of the upstream run, as the waiting run sees it (see `_upstream_run_state`). + :param due_at: when the upstream run was due, if known. + :param upstream_run: the upstream run, if there is one. + """ + if upstream is None: + return "An automation that no longer exists did not run." + label = f"{upstream.name} (automation {upstream.id}" + label += f", run {upstream_run.id})" if upstream_run is not None else ")" + due = f" at {_format_due_at(upstream, due_at)}" if due_at is not None else "" + if upstream_run is None: + return _summarize_missing_upstream_run(upstream, label, state, due) + if due_at is not None and upstream_run.scheduled_at > due_at: + label += f", which caught up with its run due{due}," + if state == "succeeded": + return f"{label} succeeded." + if state == "pending": + return f"{label} has yet to end." + if state == "not-scheduled": + return f"{label} was deactivated while its run waited." + # The run ended without delivering what the waiting run needs (see `_upstream_run_state`). + return f"{label} {_describe_unsuccessful_run(upstream_run)}." + + +def _summarize_missing_upstream_run( + upstream: Automation, label: str, state: str, due: str +) -> str: + """Say in a sentence why there is no upstream run for a run to wait for (see `_summarize_upstream_run`).""" + if state == "pending": + return f"{label} is due{due}, and has yet to run." + if state != "not-scheduled": + return f"{label} did not run{due}." + if not due: + return f"{label} has a recurrence that cannot be read, so it is not waited for." + if not upstream.active: + return f"{label} is inactive, so it was not due{due}." + if upstream.schedule_revision == 1: + return f"{label} was not due yet{due}, as it was created after that." + return f"{label} was not due{due}, as its recurrence was reset after that." + + +def _describe_unsuccessful_run(run: AutomationRun) -> str: + """Say how a run ended that did not deliver what the runs waiting for it need.""" + if run.execution_state == "skipped": + return "was skipped" + if run.execution_state == "canceled": + return "was canceled" + if run.execution_completed_at is None: + return "failed to queue its jobs" + return "failed" + + +def _find_chain_upstream_runs(run: AutomationRun) -> list[dict[str, Any]]: + """Find the runs asked for on demand along with this run, which it waits for, as recorded when they were created.""" + allow_failure = { + dependency.upstream_id: dependency.allow_failure + for dependency in run.automation.dependencies + } + found = [] + for upstream in (run.plan or {}).get("chain-upstream-runs", []): + upstream_run = db.session.get(AutomationRun, upstream["run"]) + # A run that is gone, along with its automation, will not produce anything. + state = ( + _upstream_run_state(upstream_run) if upstream_run is not None else "failed" + ) + found.append( + { + "automation": upstream["automation"], + "allow-failure": allow_failure.get(upstream["automation"], False), + "due-at": run.scheduled_at.isoformat(), + "run": upstream["run"], + "state": state, + "summary": _summarize_upstream_run( + db.session.get(Automation, upstream["automation"]), + state, + None, + upstream_run, + ), + } + ) + return found + + +def _upstream_verdict(found: list[dict[str, Any]]) -> str: + """Say whether a run should 'skip', 'wait' or 'go', given the state of each upstream run it waits for.""" + if any( + upstream["state"] in ("failed", "passed-over") and not upstream["allow-failure"] + for upstream in found + ): + return "skip" + if any(upstream["state"] == "pending" for upstream in found): + return "wait" + return "go" + + +def _describe_upstream_runs( + found: list[dict[str, Any]], states: tuple[str, ...] +) -> str: + """Say what became of the upstream runs in one of these states, for a message.""" + return " ".join( + upstream["summary"] for upstream in found if upstream["state"] in states + ) + + +def _hold_automation_run( + claimed_run: ClaimedAutomationRun, found: list[dict[str, Any]] +) -> dict[str, Any]: + """Have a run wait for its upstream runs, releasing the claim on it, as no runner has to hold on to it meanwhile.""" + run = claimed_run.run + run.dispatch_state = "waiting" + run.claim_owner = None + run.claim_expires_at = None + _finish_attempt(claimed_run.attempt, "waiting", 0) + db.session.commit() + message = f"Waiting: {_describe_upstream_runs(found, ('pending',))}" + # An upstream run may have ended between deciding to wait and saying so, + # in which case nothing else would come back to this run before the next runner tick. + released = release_waiting_automation_run(run) + if released is not None: + return released + return { + "run_id": run.id, + "job_id": None, + "n_jobs": 0, + "dispatch_state": "waiting", + "message": message, + } + + +def _skip_automation_run( + claimed_run: ClaimedAutomationRun, found: list[dict[str, Any]] +) -> dict[str, Any]: + """End a run without queueing any jobs, as an upstream run it waits for did not succeed.""" + run = claimed_run.run + now = _now_utc() + # Only the upstream runs whose failure is not allowed for are why the run is skipped. + reason = f"Skipped: {_describe_upstream_runs([upstream for upstream in found if not upstream['allow-failure']], ('failed', 'passed-over'))}" + run.dispatch_state = "queued" + run.dispatch_completed_at = now + run.execution_state = "skipped" + run.execution_completed_at = now + run.last_error_type = "UpstreamRunFailed" + run.last_error_message = reason + run.claim_owner = None + run.claim_expires_at = None + _finish_attempt(claimed_run.attempt, "skipped", 0) + db.session.commit() + # Whatever waits for this run cannot have its input either. + release_automation_runs_waiting_for(run) + return { + "run_id": run.id, + "job_id": None, + "n_jobs": 0, + "dispatch_state": "skipped", + "message": reason, + } + + +def release_waiting_automation_run(run: AutomationRun) -> dict[str, Any] | None: + """Dispatch a waiting run, or skip it, if the upstream runs it waits for ended. + + Another worker or runner may be releasing the same run at the same time, so it is claimed first, and only one of them gets it. + A failure to dispatch is recorded on the run (see `dispatch_automation_run`), rather than handed to whoever ended the upstream run, + whose own job succeeded or failed regardless. + + :returns: what the dispatch returned, or None if the run was left waiting. + """ + db.session.refresh(run) + if run.dispatch_state != "waiting" or ( + # A run on demand is asked for, so it goes ahead for an inactive automation, too. + not run.automation.active + and run.trigger == "schedule" + ): + return None + if check_upstream_runs(run)[0] == "wait": + return None + claimed = claim_existing_automation_run(run, dispatch_states=("waiting",)) + if claimed is None: + return None + try: + position = _tick_report_position() + returns = dispatch_automation_run(claimed) + _report_to_tick(run, "released", returns.get("message"), position) + return returns + except Exception as exc: + # Any error, whatever its kind, was recorded on the run by the dispatch, which is what the run's owner gets to see. + # The upstream run ended either way, so its job does not fail on account of the run that waited for it. + current_app.logger.warning( + f"Automation run {run.id} failed to dispatch once its upstream runs ended: {exc}" + ) + return None + + +def release_automation_runs_waiting_for(upstream_run: AutomationRun) -> None: + """Dispatch, or skip, the runs that wait for this upstream run, now that it ended. + + This is called wherever a run ends: when its last job ends, when it queued no jobs, when its dispatch failed for good, and when it was skipped. + A runner also looks at the waiting runs on every tick (see `get_dispatchable_automation_runs`), in case a release here was missed. + """ + waiting_runs = db.session.scalars( + select(AutomationRun) + .join( + AutomationDependency, + AutomationDependency.automation_id == AutomationRun.automation_id, + ) + .where( + AutomationDependency.upstream_id == upstream_run.automation_id, + # A run usually waits for the latest upstream run due at or before its own scheduled time (see `get_upstream_due_at`), + # but after an outage it can wait for a later run, which the upstream automation caught up with (see `_get_catch_up_run`). + # So every waiting run of the automations that wait for this one is checked, and those waiting for another run are left waiting. + AutomationRun.dispatch_state == "waiting", + ) + .order_by(AutomationRun.id) + ).all() + for run in waiting_runs: + release_waiting_automation_run(run) + + +def _release_runs_waiting_for_ended_run(run: AutomationRun) -> None: + """Release the runs waiting for this run, if it ended, without letting a database error there reach whoever ended it. + + The callers record how a job or a dispatch ended, which stands whatever becomes of the runs waiting for it; + a runner picks those up on its next tick instead (see `get_dispatchable_automation_runs`). + """ + if run.execution_completed_at is None and not _dispatch_was_given_up(run): + return + try: + release_automation_runs_waiting_for(run) + except SQLAlchemyError as exc: + db.session.rollback() + current_app.logger.warning( + f"Could not release the automation runs waiting for run {run.id}, so a runner will on its next tick: {exc}" + ) + + def pin_automation_run_effective_start( run_id: int, start: datetime, prior: datetime | None = None ) -> None: @@ -488,6 +993,7 @@ def mark_automation_run_dispatch_failed( run.claim_expires_at = _now_utc() + _retry_delay(run.attempt_count) _finish_attempt(attempt, run.dispatch_state, queued_count, error) db.session.commit() + _release_runs_waiting_for_ended_run(run) def record_automation_job_started( @@ -555,6 +1061,7 @@ def record_automation_job_succeeded( intent.finished_at = now _refresh_run_execution_state(intent.run, now) db.session.commit() + _release_runs_waiting_for_ended_run(intent.run) def record_automation_job_failed( @@ -592,10 +1099,12 @@ def _record_automation_job_failure( ).one_or_none() if intent is None: return - if intent.status == "failed": + if intent.status in ("failed", "succeeded"): # A job's failure reaches this twice: # the job says so itself, and the queue's exception handler (see `handle_forecasting_exception`) says so for a job that could not. # The first of them is the one that counts, so that the moment recorded is the moment the job failed. + # A job that recorded its success did its work, and can only fail after that while releasing the runs waiting for its run, + # such as on its timeout, which is no reason to fail the run (a runner releases what it did not get to, see `get_dispatchable_automation_runs`). db.session.rollback() return intent.status = "failed" @@ -607,6 +1116,7 @@ def _record_automation_job_failure( run.last_error_type = error_type run.last_error_message = error_message db.session.commit() + _release_runs_waiting_for_ended_run(run) def reconcile_automation_job_intent(intent: AutomationRunJob) -> bool: @@ -628,6 +1138,73 @@ def reconcile_automation_job_intent(intent: AutomationRunJob) -> bool: return False +def reconcile_unfinished_automation_jobs() -> None: + """Record how jobs ended that could not say so themselves, as the runs waiting for theirs would wait forever otherwise. + + A job reports its own outcome, which it cannot do once it is killed, for instance for running out of memory. + Redis does know: RQ fails a job whose work-horse died, and one whose worker disappeared altogether once its heartbeat lapsed (see `StartedJobRegistry.cleanup`). + Every job reports before it leaves Redis, so one that left without reporting is lost, and counts as failed. + A job that is still queued or running is left alone, however long that takes. + """ + intents_per_queue: dict[str, list[AutomationRunJob]] = defaultdict(list) + for intent in db.session.scalars( + select(AutomationRunJob) + .join(AutomationRun) + .where( + AutomationRun.execution_completed_at.is_(None), + AutomationRunJob.status.in_(("queued", "running")), + ) + .order_by(AutomationRunJob.id) + ): + intents_per_queue[intent.queue].append(intent) + for queue_name, intents in intents_per_queue.items(): + queue = current_app.queues.get(queue_name) + if queue is None: + continue + try: + queue.started_job_registry.cleanup() + jobs = Job.fetch_many( + [intent.rq_job_id for intent in intents], + connection=queue.connection, + serializer=queue.serializer, + ) + except RedisError as exc: + # The next tick looks again, and a runner without Redis has more pressing problems. + current_app.logger.warning( + f"Could not look up the unfinished jobs on queue '{queue_name}': {exc}" + ) + continue + for intent, job in zip(intents, jobs): + run = intent.run + was_completed = run.execution_completed_at is not None + position = _tick_report_position() + _record_outcome_known_to_redis(intent.run_id, intent.logical_job_key, job) + db.session.refresh(run) + if not was_completed and run.execution_completed_at is not None: + _report_to_tick(run, "reconciled", run.last_error_message, position) + + +def _record_outcome_known_to_redis( + run_id: int, logical_job_key: str, job: Job | None +) -> None: + """Record how a job ended, going by what Redis says about it, if it ended.""" + if job is None: + _record_automation_job_failure( + run_id, + logical_job_key, + "JobLost", + "The job left the queue without saying how it ended.", + ) + return + status = job.get_status(refresh=False) + if status == JobStatus.FINISHED: + record_automation_job_succeeded(run_id, logical_job_key) + elif status in (JobStatus.FAILED, JobStatus.STOPPED, JobStatus.CANCELED): + _record_automation_job_failure( + run_id, logical_job_key, *_error_of_failed_job(job) + ) + + def record_automation_run_jobs(run_id: int, jobs: list[tuple[str, str, Job]]) -> None: """Record the jobs that a report or plugin run queued, so that the run follows what becomes of them. @@ -647,6 +1224,7 @@ def record_automation_run_jobs(run_id: int, jobs: list[tuple[str, str, Job]]) -> run.execution_state = "succeeded" run.execution_completed_at = now db.session.commit() + _release_runs_waiting_for_ended_run(run) return run.plan = { **dict(run.plan or {}), @@ -670,11 +1248,8 @@ def record_automation_run_jobs(run_id: int, jobs: list[tuple[str, str, Job]]) -> run.first_enqueued_at = run.first_enqueued_at or now db.session.commit() for key, _, job in jobs: - status = job.get_status(refresh=True) - if status == JobStatus.FINISHED: - record_automation_job_succeeded(run_id, key) - elif status == JobStatus.FAILED: - _record_automation_job_failure(run_id, key, *_error_of_failed_job(job)) + job.refresh() + _record_outcome_known_to_redis(run_id, key, job) def _error_of_failed_job(job: Job) -> tuple[str | None, str]: @@ -2024,8 +2599,12 @@ def create_automation( source: DataSource | None = None, origin: str = "API", check_permissions: bool = False, + dependencies: list[dict[str, Any]] | None = None, ) -> tuple[Automation, list[str]]: - """Create an automation through its registered handler, without committing.""" + """Create an automation through its registered handler, without committing. + + :param dependencies: the automations whose runs this automation's runs wait for (see `set_automation_dependencies`). + """ from flask_security import current_user from flexmeasures.data.automations import ( get_automation_handler, @@ -2033,6 +2612,11 @@ def create_automation( ) validate_automation_type(automation_type) + if dependencies: + # Check these first, so that nothing is set up for an automation that is refused. + validate_automation_dependencies( + None, dependencies, check_permissions=check_permissions + ) automation, warnings = get_automation_handler(automation_type).create( asset=asset, name=name, @@ -2053,6 +2637,8 @@ def create_automation( raise Unauthorized() automation.execution_user_id = current_user.id + if dependencies: + set_automation_dependencies(automation, dependencies, origin=origin) return automation, warnings @@ -2242,18 +2828,29 @@ def update_automation( timezone: str | None = None, active: bool | None = None, origin: str = "API", + dependencies: list[dict[str, Any]] | None = None, + check_permissions: bool = False, ) -> list[str]: - """Update an automation's name, cron string, timezone and/or activation status (not committed yet). + """Update an automation's name, cron string, timezone, activation status and/or dependencies (not committed yet). Anything that changes which runs are due, namely the recurrence, the timezone and reactivation, also resets the cursor to just before the minute of the change, so that runs from before it are not caught up on, while a run due in that very minute still is. An audit log record is added to the asset. + :param dependencies: the automations whose runs this automation's runs wait for, replacing the ones it had; None leaves them as they are. + :param check_permissions: whether to require that the current user may read the upstream automations named in `dependencies`. + :raises marshmallow.ValidationError: if the dependencies are not possible. :returns: a list of (human-readable) changes; empty if nothing changed. """ from flexmeasures.data.models.audit_log import AssetAuditLog + # Check everything before changing anything, so that a refused update leaves the automation as it was. + if dependencies is not None: + validate_automation_dependencies( + automation, dependencies, check_permissions=check_permissions + ) + changes = [] reset_cursor = False if name is not None and name != automation.name: @@ -2271,18 +2868,217 @@ def update_automation( changes.append("activated" if active else "deactivated") if active: reset_cursor = True + else: + _cancel_waiting_scheduled_runs(automation) automation.active = active if reset_cursor: # A new revision keeps the durable runs of the old and the new schedule apart, # even where they fall on the same scheduled UTC time. automation.schedule_revision += 1 - automation.cursor = get_initial_cursor() + # Runs from before the reset are not due by the new recurrence, which is also what the runs waiting for this automation go by (see `_state_without_upstream_run`). + automation.revision_started_at = get_revision_start() + automation.cursor = automation.revision_started_at - timedelta(minutes=1) + if dependencies is not None: + # This records its own audit log line, as creating an automation with dependencies does. + dependency_changes = set_automation_dependencies( + automation, dependencies, origin=origin + ) + else: + dependency_changes = [] if changes: AssetAuditLog.add_record( automation.asset, f"Updated automation '{automation.name}' ({automation.id}): {'; '.join(changes)}. Via {origin}.", ) - return changes + return changes + dependency_changes + + +def _cancel_waiting_scheduled_runs(automation: Automation) -> None: + """End the scheduled runs of an automation that wait for others, as it is deactivated (not committed yet). + + Such a run would otherwise go ahead, late, once the automation is reactivated. + What waits for one of these runs goes ahead without it, as for an inactive automation (see `_upstream_run_state`), + once a runner looks at it again (see `get_dispatchable_automation_runs`). + Runs on demand are left alone, as they were asked for. + """ + now = _now_utc() + for run in db.session.scalars( + select(AutomationRun).where( + AutomationRun.automation_id == automation.id, + AutomationRun.trigger == "schedule", + AutomationRun.dispatch_state == "waiting", + ) + ): + run.dispatch_state = "queued" + run.dispatch_completed_at = now + run.execution_state = "canceled" + run.execution_completed_at = now + run.last_error_type = AUTOMATION_DEACTIVATED + run.last_error_message = "Canceled, as its automation was deactivated while this run waited for the runs of other automations." + + +def _describe_dependency(upstream: Automation, allow_failure: bool) -> str: + """Name an upstream automation in a message, and say whether its failure is allowed for.""" + return f"{upstream.id}{' (also when it fails)' if allow_failure else ''}" + + +def _upstream_ids_reachable_from(automation_ids: set[int]) -> set[int]: + """Return every automation that the given automations wait for, directly or through others.""" + reached: set[int] = set() + frontier = set(automation_ids) + while frontier: + upstream_ids = set( + db.session.scalars( + select(AutomationDependency.upstream_id).where( + AutomationDependency.automation_id.in_(frontier) + ) + ).all() + ) + frontier = upstream_ids - reached + reached |= upstream_ids + return reached + + +def validate_automation_dependencies( + automation: Automation | None, + dependencies: list[dict[str, Any]], + check_permissions: bool = False, +) -> None: + """Check that an automation may wait for these upstream automations. + + The automations need not recur on the same schedule, as a run waits for the latest upstream run due at or before its own scheduled time (see `get_upstream_due_at`). + An automation waiting for itself, directly or through others, would never run, so such a cycle is refused. + + :param automation: the automation that is to wait, or None for one that is about to be created. + :param dependencies: dicts with the upstream automation under "upstream", and whether its failure is allowed for under "allow_failure". + :param check_permissions: whether to require that the current user may read each upstream automation (and so its asset). + :raises marshmallow.ValidationError: if a dependency is not possible, under the "depends-on" field. + :raises werkzeug.exceptions.Forbidden: if the current user may not read an upstream automation. + """ + from flexmeasures.auth.policy import check_access + + automation_id = automation.id if automation is not None else None + errors = [] + upstream_ids = [dependency["upstream"].id for dependency in dependencies] + duplicates = sorted( + { + upstream_id + for upstream_id in upstream_ids + if upstream_ids.count(upstream_id) > 1 + } + ) + if duplicates: + errors.append( + f"Automation(s) {', '.join(map(str, duplicates))} are named more than once." + ) + for dependency in dependencies: + upstream = dependency["upstream"] + if check_permissions: + check_access(upstream, "read") + if automation_id is not None and upstream.id == automation_id: + errors.append(f"Automation {upstream.id} cannot wait for itself.") + continue + # An automation that is about to be created has nothing waiting for it yet, so it cannot close a cycle. + if automation_id is not None and automation_id in _upstream_ids_reachable_from( + {upstream_id for upstream_id in upstream_ids if upstream_id != automation_id} + ): + errors.append( + f"Automation {automation_id} would end up waiting for itself, as one of the automations it waits for already waits for it, directly or through others." + ) + if errors: + raise ValidationError({"depends-on": errors}) + + +def set_automation_dependencies( + automation: Automation, + dependencies: list[dict[str, Any]], + origin: str = "API", +) -> list[str]: + """Replace the automations that an automation's runs wait for (not committed yet). + + Validate them first (see `validate_automation_dependencies`). + An audit log record is added to the asset if anything changed. + + :param dependencies: dicts with the upstream automation under "upstream", and whether its failure is allowed for under "allow_failure". + :returns: a list of (human-readable) changes; empty if nothing changed. + """ + from flexmeasures.data.models.audit_log import AssetAuditLog + + current = { + dependency.upstream_id: dependency.allow_failure + for dependency in automation.dependencies + } + wanted = { + dependency["upstream"].id: bool(dependency.get("allow_failure", False)) + for dependency in dependencies + } + if current == wanted: + return [] + upstreams = { + dependency["upstream"].id: dependency["upstream"] for dependency in dependencies + } + automation.dependencies = [ + AutomationDependency( + automation=automation, + upstream=upstreams[upstream_id], + allow_failure=allow_failure, + ) + for upstream_id, allow_failure in sorted(wanted.items()) + ] + described = ( + ", ".join( + _describe_dependency(upstreams[upstream_id], allow_failure) + for upstream_id, allow_failure in sorted(wanted.items()) + ) + or "none" + ) + change = f"depends on: {described}" + if automation.id is not None: + AssetAuditLog.add_record( + automation.asset, + f"Updated automation '{automation.name}' ({automation.id}): {change}. Via {origin}.", + ) + return [change] + + +def serialize_automation_dependencies(automation: Automation) -> list[dict[str, Any]]: + """Render the automations an automation waits for, as the API names them. + + Each one comes with its name and asset, so that it can be told apart without being listed alongside this automation. + An automation can go on waiting for one its users may no longer read, which then stays a bare ID. + """ + return [ + { + "automation": dependency.upstream_id, + "allow-failure": dependency.allow_failure, + **( + { + "name": dependency.upstream.name, + "asset": dependency.upstream.asset_id, + "asset-name": dependency.upstream.asset.name, + } + if _current_user_may_read(dependency.upstream) + else {} + ), + } + for dependency in automation.dependencies + ] + + +def _current_user_may_read(automation: Automation) -> bool: + """Whether the current user may read this automation, which nobody on the command line is logged in to do.""" + from flask_security import current_user + from werkzeug.exceptions import Forbidden, Unauthorized + + from flexmeasures.auth.policy import check_access + + if not current_user or not current_user.is_authenticated: + return False + try: + check_access(automation, "read") + except (Forbidden, Unauthorized): + return False + return True def delete_automation(automation: Automation, origin: str = "API"): @@ -2303,6 +3099,7 @@ def serialize_automation_run(run: AutomationRun) -> dict[str, Any]: "id": run.id, "scheduled-at": run.scheduled_at.isoformat(), "revision": run.schedule_revision, + "trigger": run.trigger, "dispatch-state": run.dispatch_state, "execution-state": run.execution_state, "attempt-count": run.attempt_count, @@ -2319,6 +3116,8 @@ def serialize_automation_run(run: AutomationRun) -> dict[str, Any]: if run.execution_completed_at else None ), + # What the run found of the upstream runs it waits for, when it last looked (see `check_upstream_runs`). + "upstream-runs": list((run.plan or {}).get("upstream-runs", [])), "claim-owner": run.claim_owner, "claim-expires-at": ( run.claim_expires_at.isoformat() if run.claim_expires_at else None @@ -2508,6 +3307,15 @@ def dispatch_automation_run( raise AutomationRunClaimLost( f"Automation run {run.id} is no longer claimed by {owner}, so another runner is dispatching it." ) + if run.automation.dependencies and not run.job_intents: + # A run that queued jobs before was past its upstream runs already, so a retry does not wait for them again. + verdict, found = check_upstream_runs(run) + run.plan = {**dict(run.plan or {}), "upstream-runs": found} + if verdict == "wait": + return _hold_automation_run(claimed_run, found) + if verdict == "skip": + return _skip_automation_run(claimed_run, found) + db.session.commit() try: returns = run_automation(run.automation, automation_run=run) except Exception as exc: @@ -2522,6 +3330,338 @@ def dispatch_automation_run( } +def get_automations_waiting_for(automation: Automation) -> list[Automation]: + """Return the automations that wait for this one directly, active or not.""" + return list( + db.session.scalars( + select(Automation) + .join( + AutomationDependency, + AutomationDependency.automation_id == Automation.id, + ) + .where(AutomationDependency.upstream_id == automation.id) + .order_by(Automation.id) + ).unique() + ) + + +def get_automations_in_scope( + scope: str, account_ids: Collection[int] +) -> list[Automation]: + """Return automations across assets, by ID. + + :param scope: 'public' for those on public assets, or 'all' to add those on the assets of these accounts. + :param account_ids: the accounts whose assets count, for scope 'all'. + """ + from flexmeasures.data.models.generic_assets import GenericAsset + + on_public_asset = GenericAsset.account_id.is_(None) + return list( + db.session.scalars( + select(Automation) + .join(GenericAsset, GenericAsset.id == Automation.asset_id) + .where( + on_public_asset + if scope == "public" + else on_public_asset | GenericAsset.account_id.in_(account_ids) + ) + .order_by(Automation.id) + ).unique() + ) + + +CHAIN_INCLUDES = ("upstream", "downstream") + + +def _reason_not_to_run( + candidate: Automation, + exclude: Collection[int], + may_run: Callable[[Automation], bool] | None, +) -> str | None: + """Say why an automation in a chain does not run along with it, if it does not.""" + if candidate.id in exclude: + return "You left it out." + if not candidate.active: + return "It is inactive." + if may_run is not None and not may_run(candidate): + return "You may not run this automation." + return None + + +def _in_chain_order(automations: list[Automation]) -> list[Automation]: + """Order automations so that each comes after the ones among them it waits for, those that can run side by side by ID.""" + by_id = {automation.id: automation for automation in automations} + waits_for = { + automation.id: { + dependency.upstream_id + for dependency in automation.dependencies + if dependency.upstream_id in by_id + } + for automation in automations + } + ordered: list[Automation] = [] + while waits_for: + # Dependencies form no cycle (see `validate_automation_dependencies`), so there is always one ready. + ready = sorted( + automation_id + for automation_id, upstream in waits_for.items() + if not upstream + ) or sorted(waits_for) + for automation_id in ready: + ordered.append(by_id[automation_id]) + del waits_for[automation_id] + for upstream in waits_for.values(): + upstream.difference_update(ready) + return ordered + + +def get_chain_automations( + automation: Automation, + include: str, + exclude: Collection[int] = (), + may_run: Callable[[Automation], bool] | None = None, +) -> tuple[list[Automation], list[tuple[Automation, str]]]: + """Return the automations to run along with this one, upstream or downstream of it, and those that do not run, with the reason. + + Upstream are the automations this one waits for, directly or through others; downstream are those that wait for it. + An automation that does not run takes along what can only be reached through it: + upstream, nothing would use their results, and downstream, they would compute with results that were not renewed. + + :param include: 'upstream' or 'downstream'. + :param exclude: IDs of automations not to run, as asked for. + :param may_run: whether an automation may be run, by default all of them. + :returns: the automations that run, this one included, in the order they run (see `_in_chain_order`), + and those that do not, for being left out, inactive or not allowed. + """ + reached = {automation.id} + frontier = [automation] + running = [automation] + excluded: list[tuple[Automation, str]] = [] + while frontier: + found = [] + for current in frontier: + neighbours = ( + sorted( + (dependency.upstream for dependency in current.dependencies), + key=lambda upstream: upstream.id, + ) + if include == "upstream" + else get_automations_waiting_for(current) + ) + for neighbour in neighbours: + if neighbour.id in reached: + continue + reached.add(neighbour.id) + reason = _reason_not_to_run(neighbour, exclude, may_run) + if reason is not None: + excluded.append((neighbour, reason)) + continue + running.append(neighbour) + found.append(neighbour) + frontier = found + return _in_chain_order(running), excluded + + +def plan_automation_chain( + automation: Automation, + include: str, + exclude: Collection[int] = (), + may_read: Callable[[Automation], bool] | None = None, + may_run: Callable[[Automation], bool] | None = None, +) -> dict[str, list[dict[str, Any]]]: + """Say what running an automation along with what is upstream or downstream of it comes down to (see `get_chain_automations`). + + :returns: under 'steps', the automations that run, in the order they run, each with what it waits for among them, + and under 'excluded', those that do not run, each with the reason, + which of the steps wait for it, and when it last succeeded (null if never), + as the steps that wait for it go by the results it produced then. + One the user may not read stays a bare ID. + """ + chain, excluded = get_chain_automations(automation, include, exclude, may_run) + involved = {step.id for step in chain} + + def describe(step: Automation) -> dict[str, Any]: + if may_read is not None and not may_read(step): + return {} + return { + "name": step.name, + "asset": step.asset_id, + "asset-name": step.asset.name, + } + + def describe_excluded(step: Automation, reason: str) -> dict[str, Any]: + described = { + "automation": step.id, + "reason": reason, + "waited-for-by": [ + other.id + for other in chain + if any( + dependency.upstream_id == step.id + for dependency in other.dependencies + ) + ], + **describe(step), + } + if "name" in described: + last_succeeded_at = get_last_succeeded_at(step) + described["last-succeeded-at"] = ( + last_succeeded_at.isoformat() if last_succeeded_at is not None else None + ) + return described + + return { + "steps": [ + { + "automation": step.id, + "waits-for": [ + dependency.upstream_id + for dependency in step.dependencies + if dependency.upstream_id in involved + ], + **describe(step), + } + for step in chain + ], + "excluded": [describe_excluded(step, reason) for step, reason in excluded], + } + + +def get_last_succeeded_at(automation: Automation) -> datetime | None: + """Return when the latest successful run of an automation ended, scheduled or on demand, or None if none did.""" + return db.session.scalar( + select(func.max(AutomationRun.execution_completed_at)).where( + AutomationRun.automation_id == automation.id, + AutomationRun.execution_state == "succeeded", + ) + ) + + +def _new_on_demand_run( + automation: Automation, scheduled_at: datetime, dispatch_state: str +) -> AutomationRun: + """Make a durable record for a run asked for on demand (not committed yet).""" + run = AutomationRun( + automation=automation, + scheduled_at=scheduled_at, + schedule_revision=automation.schedule_revision, + trigger="on-demand", + automation_type=automation.type, + generator_id=automation.generator_id, + dispatch_state=dispatch_state, + execution_state="pending", + attempt_count=0, + parameters=dict(automation.parameters or {}), + plan=_run_snapshot(automation, scheduled_at), + ) + db.session.add(run) + return run + + +def _start_on_demand_runs( + automations: list[Automation], + scheduled_at: datetime, + owner: str, + claimed_at: datetime, +) -> tuple[dict[int, AutomationRun], list[ClaimedAutomationRun]]: + """Record a run for each of these automations, asked for together, and claim the ones that can start. + + An automation that waits for none of the others can start, and the others wait, for the runs asked for along with them only. + + :returns: the runs per automation ID, and the claimed runs for the caller to dispatch. + """ + involved = {automation.id for automation in automations} + runs: dict[int, AutomationRun] = {} + starting: list[AutomationRun] = [] + for automation in automations: + waits = any( + dependency.upstream_id in involved for dependency in automation.dependencies + ) + run = _new_on_demand_run( + automation, scheduled_at, "waiting" if waits else "claimed" + ) + runs[automation.id] = run + if not waits: + run.claim_owner = owner + run.claimed_at = claimed_at + run.claim_expires_at = _claim_expires_at( + claimed_at, AUTOMATION_RUN_CLAIM_LEASE + ) + run.attempt_count = 1 + starting.append(run) + db.session.flush() + for automation in automations: + run = runs[automation.id] + if run not in starting: + run.plan = { + **dict(run.plan or {}), + "chain-upstream-runs": [ + { + "automation": dependency.upstream_id, + "run": runs[dependency.upstream_id].id, + } + for dependency in automation.dependencies + if dependency.upstream_id in runs + ], + } + claimed_runs = [ + ClaimedAutomationRun(run=run, attempt=_new_attempt(run, owner, claimed_at)) + for run in starting + ] + db.session.commit() + return runs, claimed_runs + + +def run_automation_chain( + automation: Automation, + include: str, + exclude: Collection[int] = (), + may_run: Callable[[Automation], bool] | None = None, + now: datetime | None = None, + owner: str | None = None, +) -> tuple[list[AutomationRun], dict[str, Any] | None, list[tuple[Automation, str]]]: + """Run an automation now, along with what is upstream or downstream of it (see `get_chain_automations`), each once the runs it waits for ended. + + Including what is downstream is how a chain is re-run, for instance once the data at its start was backfilled. + Including what is upstream is how an automation is brought up to date, with what its results are based on. + The runs that wait for none of the others start right away; each of the others waits for the runs asked for along with it only, + and is skipped if one of those failed, as a scheduled run is, unless its dependency allows for failure. + An automation that does not run is simply not waited for, so what waits for it goes by the results it already has. + None of these runs affect the automations' recurrence, and the runner does not retry them if their jobs failed to queue. + + :param include: 'upstream' or 'downstream'. + :param exclude: IDs of automations not to run. + :param may_run: whether an automation may be run, by default all of them. + :returns: the runs in the order they run, + what this automation's own run dispatched if it started right away (see `dispatch_automation_run`), + and the automations that do not run, with the reason. + :raises: whatever this automation's own run raised while queueing its jobs, if it started right away. + """ + claimed_at = _now_utc() + chain, excluded = get_chain_automations(automation, include, exclude, may_run) + runs, claimed_runs = _start_on_demand_runs( + chain, + floor_to_minute(now or claimed_at), + owner or _runner_owner(), + claimed_at, + ) + returns = None + for claimed_run in claimed_runs: + if claimed_run.run.automation_id == automation.id: + # Whoever asked for this automation to run is told right away if it failed to queue its jobs, as for a single run. + returns = dispatch_automation_run(claimed_run) + continue + try: + dispatch_automation_run(claimed_run) + except Exception as exc: + # Any error, whatever its kind, was recorded on the run by the dispatch, which ends it (see `_dispatch_was_given_up`). + # The other runs that can start have nothing to do with it, and a runner settles what waits for this one. + current_app.logger.warning( + f"Automation run {claimed_run.run.id} failed to dispatch: {exc}" + ) + return [runs[step.id] for step in chain], returns, excluded + + def run_automation( automation: Automation, automation_run: AutomationRun | None = None, diff --git a/flexmeasures/data/tests/test_automation_dependencies_fresh_db.py b/flexmeasures/data/tests/test_automation_dependencies_fresh_db.py new file mode 100644 index 0000000000..95a7fe6928 --- /dev/null +++ b/flexmeasures/data/tests/test_automation_dependencies_fresh_db.py @@ -0,0 +1,1030 @@ +"""Automations that wait for the runs of other automations: validation, and how the runner holds, releases and skips their runs.""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +import pytest +from marshmallow import ValidationError +from redis.exceptions import RedisError +from rq.job import Job, JobStatus +from sqlalchemy import select + +from flexmeasures.data.automations import register_automation_handler +from flexmeasures.data.models.automations import Automation, AutomationRun +from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType +from flexmeasures.data.models.time_series import Sensor +from flexmeasures.data.services import automations as automation_services +from flexmeasures.data.services.automations import ( + create_automation, + dispatch_automation_run, + get_dispatchable_automation_runs, + run_automation, + get_chain_automations, + plan_automation_chain, + run_automation_chain, + update_automation, +) +from flexmeasures.data.tests.plugins.mock_ingestion import ( + MockIngestionGenerator, + __automation_types__, +) +from flexmeasures.utils.job_utils import work_on_rq + +# Every automation here recurs hourly and was created well before the run at 01:00, which a runner ticking at 01:00 dispatches. +CREATED_AT = datetime(2026, 8, 5, 0, 0, tzinfo=timezone.utc) +SCHEDULED_AT = datetime(2026, 8, 5, 1, 0, tzinfo=timezone.utc) +TICK = datetime(2026, 8, 5, 1, 0, 30, tzinfo=timezone.utc) + + +@pytest.fixture +def ingestion_plugin(app, monkeypatch): + monkeypatch.setattr(app, "automation_handlers", dict(app.automation_handlers)) + monkeypatch.setattr( + app, + "data_generators", + {key: dict(value) for key, value in app.data_generators.items()}, + ) + register_automation_handler(app, __automation_types__[0]) + app.redis_connection.flushdb() + yield + app.redis_connection.flushdb() + + +@pytest.fixture +def failing_value(monkeypatch): + """Have an ingestion job fail when its automation's parameters ask for a negative value.""" + compute = MockIngestionGenerator._compute + + def _compute(self, value): + if value < 0: + raise ValueError("The external system was unreachable.") + return compute(self, value) + + monkeypatch.setattr(MockIngestionGenerator, "_compute", _compute) + + +@pytest.fixture +def make_automation(fresh_db, ingestion_plugin): + """Create hourly ingestion automations on one asset, each due at 01:00, optionally waiting for others.""" + asset = GenericAsset( + name="Dependency site", + generic_asset_type=GenericAssetType(name="Dependency asset type"), + ) + sensor = Sensor(name="measurement", generic_asset=asset, unit="kWh") + fresh_db.session.add(sensor) + fresh_db.session.commit() + + def _make_automation( + name: str, + depends_on: list[Automation] = (), + allow_failure: bool = False, + value: float = 1.0, + cronstr: str = "0 * * * *", + tz: str = "UTC", + ) -> Automation: + automation, _ = create_automation( + asset=asset, + name=name, + cronstr=cronstr, + timezone=tz, + automation_type="mock-ingestion", + config={"sensor": sensor.id}, + parameters={"value": value}, + dependencies=[ + {"upstream": upstream, "allow_failure": allow_failure} + for upstream in depends_on + ], + ) + automation.created_at = CREATED_AT + automation.revision_started_at = CREATED_AT + automation.cursor = CREATED_AT + fresh_db.session.commit() + return automation + + return _make_automation + + +def _tick(now: datetime = TICK) -> dict[int, dict]: + """Claim and dispatch what a runner would at this moment, dependents first, which is the order that makes them wait.""" + claimed_runs = get_dispatchable_automation_runs(now) + claimed_runs.sort(key=lambda claimed: -claimed.run.automation_id) + return { + claimed.run.automation_id: dispatch_automation_run(claimed) + for claimed in claimed_runs + } + + +def _run_of(db, automation: Automation) -> AutomationRun: + db.session.expire_all() + return db.session.scalars( + select(AutomationRun).where(AutomationRun.automation_id == automation.id) + ).one() + + +def _work(app, run: AutomationRun) -> None: + """Have a worker run the jobs a run recorded.""" + for intent in run.job_intents: + work_on_rq( + app.queues[intent.queue], + job=Job.fetch(intent.rq_job_id, connection=app.redis_connection), + ) + + +def test_a_dependent_run_waits_and_then_runs_once_its_upstream_run_succeeded( + fresh_db, app, make_automation +): + """The dependent run queues nothing while the upstream job runs, and its job is queued as soon as that job succeeded.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + + returns = _tick() + + assert returns[flows.id]["dispatch_state"] == "waiting" + flows_run = _run_of(fresh_db, flows) + assert flows_run.dispatch_state == "waiting" + assert flows_run.job_intents == [] + assert flows_run.claim_owner is None + snapper_run = _run_of(fresh_db, snapper) + assert flows_run.plan["upstream-runs"] == [ + { + "automation": snapper.id, + "allow-failure": False, + "due-at": SCHEDULED_AT.isoformat(), + "run": snapper_run.id, + "state": "pending", + "summary": f"Snap meter readings (automation {snapper.id}, run {snapper_run.id}) has yet to end.", + } + ] + + _work(app, snapper_run) + + flows_run = _run_of(fresh_db, flows) + assert flows_run.dispatch_state == "queued" + assert [intent.status for intent in flows_run.job_intents] == ["queued"] + assert [attempt.outcome for attempt in flows_run.attempts] == ["waiting", "queued"] + _work(app, flows_run) + assert _run_of(fresh_db, flows).execution_state == "succeeded" + + +def test_a_dependent_run_is_skipped_when_its_upstream_run_failed( + fresh_db, app, make_automation, failing_value +): + """Without allowing for failure, a failed upstream run ends the dependent run without queueing anything, and says why.""" + snapper = make_automation("Snap meter readings", value=-1) + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + + _work(app, _run_of(fresh_db, snapper)) + + flows_run = _run_of(fresh_db, flows) + assert flows_run.execution_state == "skipped" + assert flows_run.execution_completed_at is not None + assert flows_run.dispatch_completed_at is not None + assert flows_run.job_intents == [] + assert flows_run.last_error_type == "UpstreamRunFailed" + assert flows_run.last_error_message == ( + f"Skipped: Snap meter readings (automation {snapper.id}, run {_run_of(fresh_db, snapper).id}) failed." + ) + queued_for = { + Job.fetch(job_id, connection=app.redis_connection).meta["trigger"][ + "automation_id" + ] + for job_id in app.queues["ingestion"].job_ids + } + assert flows.id not in queued_for + + +def test_a_dependent_run_that_allows_for_failure_runs_anyway( + fresh_db, app, make_automation, failing_value +): + """Allowing for failure, the dependent run still waits for the upstream run, and then goes ahead although it failed.""" + snapper = make_automation("Snap meter readings", value=-1) + flows = make_automation("Compute flows", depends_on=[snapper], allow_failure=True) + assert _tick()[flows.id]["dispatch_state"] == "waiting" + + _work(app, _run_of(fresh_db, snapper)) + + flows_run = _run_of(fresh_db, flows) + assert flows_run.dispatch_state == "queued" + assert len(flows_run.job_intents) == 1 + + +def test_a_skip_cascades_down_a_chain(fresh_db, app, make_automation, failing_value): + """A run skipped for a failed upstream run did not deliver its own results, so what waits for it is skipped, too.""" + snapper = make_automation("Snap meter readings", value=-1) + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + returns = _tick() + assert returns[totals.id]["dispatch_state"] == "waiting" + + _work(app, _run_of(fresh_db, snapper)) + + assert _run_of(fresh_db, flows).execution_state == "skipped" + assert _run_of(fresh_db, totals).execution_state == "skipped" + + +def _kill_job_as_rq_finds_it(app, job: Job) -> None: + """Leave a job as RQ does when the work-horse running it was killed, so that it never got to report.""" + job.set_status(JobStatus.FAILED) + + +def _lose_job(app, job: Job) -> None: + """Have a job leave Redis without a trace, as when its hash expired.""" + job.delete() + + +def _abandon_job(app, job: Job) -> None: + """Leave a job as a worker does that disappeared altogether: started, with a heartbeat that lapsed long ago.""" + job.set_status(JobStatus.STARTED) + app.queues["ingestion"].remove(job.id) + app.redis_connection.zadd( + app.queues["ingestion"].started_job_registry.key, + {f"{job.id}:execution": 1}, + ) + + +@pytest.mark.parametrize( + "end_job, error_type", + [(_kill_job_as_rq_finds_it, None), (_lose_job, "JobLost"), (_abandon_job, None)], +) +@pytest.mark.parametrize("on_demand", [False, True]) +def test_a_killed_job_ends_its_run_and_what_waits_for_it( + fresh_db, app, make_automation, end_job, error_type, on_demand +): + """A job that gets killed cannot say so, so the runner finds out, and what waits for its run is not left waiting forever.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + if on_demand: + runs, returns, _ = run_automation_chain(snapper, "downstream", now=TICK) + dependent_runs = runs[1:] + snapper_run = fresh_db.session.get(AutomationRun, returns["run_id"]) + else: + _tick() + snapper_run = _run_of(fresh_db, snapper) + dependent_runs = [_run_of(fresh_db, flows), _run_of(fresh_db, totals)] + run_id, dependent_run_ids = snapper_run.id, [run.id for run in dependent_runs] + intent = snapper_run.job_intents[0] + # The worker got this far before it died. + automation_services.record_automation_job_started(run_id, intent.logical_job_key) + end_job(app, Job.fetch(intent.rq_job_id, connection=app.redis_connection)) + + # The next tick finds out, before any scheduled run is due again. + report = [] + get_dispatchable_automation_runs( + datetime(2026, 8, 5, 0, 30, tzinfo=timezone.utc) if on_demand else TICK, + report=report, + ) + + # Which it reports, so a runner's log does not say that nothing happened. + assert [(event["run"].id, event["event"]) for event in report] == [ + (run_id, "reconciled"), + *[(dependent_run_id, "released") for dependent_run_id in dependent_run_ids], + ] + assert report[1]["message"].startswith("Skipped: Snap meter readings") + fresh_db.session.expire_all() + snapper_run = fresh_db.session.get(AutomationRun, run_id) + assert snapper_run.execution_state == "failed" + assert snapper_run.execution_completed_at is not None + if error_type is not None: + assert snapper_run.last_error_type == error_type + for dependent_run_id in dependent_run_ids: + assert ( + fresh_db.session.get(AutomationRun, dependent_run_id).execution_state + == "skipped" + ) + + +def test_a_job_that_times_out_after_recording_its_success_still_succeeded( + fresh_db, app, make_automation, monkeypatch +): + """Releasing what waits for a run happens in the job that ended it, and failing there, as on a timeout, keeps the run's success.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + snapper_run = _run_of(fresh_db, snapper) + intent = snapper_run.job_intents[0] + with monkeypatch.context() as patch: + # The job's time runs out while it releases what waits for its run. + patch.setattr( + automation_services, + "release_automation_runs_waiting_for", + lambda run: None, + ) + automation_services.record_automation_job_succeeded( + snapper_run.id, intent.logical_job_key + ) + automation_services._record_automation_job_failure( + snapper_run.id, intent.logical_job_key, "JobTimeoutException", "Timed out." + ) + + assert _run_of(fresh_db, snapper).execution_state == "succeeded" + _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + assert _run_of(fresh_db, flows).dispatch_state == "queued" + + +def test_an_abandoned_run_on_demand_ends_and_so_does_what_waits_for_it( + fresh_db, app, make_automation +): + """A chain recorded on demand, whose first run nobody went on to queue, is ended by the runner once the claim on it expires.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + # Whoever asked for the chain went down between recording its runs and queueing the first one. + runs, _ = automation_services._start_on_demand_runs( + [snapper, flows], SCHEDULED_AT, "api-process", TICK + ) + snapper_run, flows_run = runs[snapper.id], runs[flows.id] + + report = [] + get_dispatchable_automation_runs(TICK + timedelta(minutes=5), report=report) + assert report == [] + assert _states(fresh_db, [snapper_run])[snapper.id] == ("claimed", "pending") + + get_dispatchable_automation_runs(TICK + timedelta(minutes=11), report=report) + + assert [(event["run"].id, event["event"]) for event in report] == [ + (snapper_run.id, "abandoned"), + (flows_run.id, "released"), + ] + assert _states(fresh_db, [snapper_run, flows_run]) == { + snapper.id: ("failed", "pending"), + flows.id: ("queued", "skipped"), + } + + +@pytest.mark.parametrize( + "status", [JobStatus.QUEUED, JobStatus.STARTED, JobStatus.DEFERRED] +) +def test_a_job_that_has_yet_to_end_is_left_alone( + fresh_db, app, make_automation, status +): + """However long a job takes, the runner does not call it over, and what waits for its run goes on waiting.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + snapper_run = _run_of(fresh_db, snapper) + job = Job.fetch( + snapper_run.job_intents[0].rq_job_id, connection=app.redis_connection + ) + job.set_status(status) + + _tick(datetime(2026, 8, 5, 6, 0, 30, tzinfo=timezone.utc)) + + fresh_db.session.expire_all() + assert _run_at(fresh_db, snapper, SCHEDULED_AT).execution_completed_at is None + assert _run_at(fresh_db, flows, SCHEDULED_AT).dispatch_state == "waiting" + + +def test_the_runner_looks_up_unfinished_jobs_on_each_tick( + fresh_db, app, make_automation +): + """Nothing else would notice that a job was killed, so a tick has to.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + Job.fetch( + _run_of(fresh_db, snapper).job_intents[0].rq_job_id, + connection=app.redis_connection, + ).set_status(JobStatus.FAILED) + + _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + + assert _run_of(fresh_db, snapper).execution_state == "failed" + assert _run_of(fresh_db, flows).execution_state == "skipped" + + +def test_a_job_that_finished_without_reporting_counts_as_succeeded( + fresh_db, app, make_automation +): + """A worker can go down between a job finishing and its report, which is no reason to skip what waits for it.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + snapper_run = _run_of(fresh_db, snapper) + Job.fetch( + snapper_run.job_intents[0].rq_job_id, connection=app.redis_connection + ).set_status(JobStatus.FINISHED) + + _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + + assert _run_of(fresh_db, snapper).execution_state == "succeeded" + assert _run_of(fresh_db, flows).dispatch_state == "queued" + + +def test_a_runner_goes_on_without_redis_when_looking_up_unfinished_jobs( + fresh_db, app, make_automation, monkeypatch +): + """Not being able to reach Redis says nothing about a job, so its run stays as it is, for the next tick to look at.""" + snapper = make_automation("Snap meter readings") + _tick() + + def unreachable(*args, **kwargs): + raise RedisError("Connection refused") + + monkeypatch.setattr(Job, "fetch_many", unreachable) + automation_services.reconcile_unfinished_automation_jobs() + + assert _run_of(fresh_db, snapper).execution_completed_at is None + + +def test_a_waiting_run_whose_release_was_missed_is_picked_up_by_the_next_tick( + fresh_db, app, make_automation, monkeypatch +): + """Should the upstream run end without releasing its dependents, the runner dispatches them on its next tick.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + with monkeypatch.context() as patch: + patch.setattr( + automation_services, + "release_automation_runs_waiting_for", + lambda run: None, + ) + _work(app, _run_of(fresh_db, snapper)) + assert _run_of(fresh_db, flows).dispatch_state == "waiting" + + returns = _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + + assert returns[flows.id]["dispatch_state"] == "queued" + assert _run_of(fresh_db, flows).dispatch_state == "queued" + + +def test_a_waiting_run_stays_waiting_on_a_tick_while_its_upstream_run_is_pending( + fresh_db, app, make_automation +): + """The runner leaves a waiting run alone for as long as the run it waits for has not ended.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + + assert _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) == {} + assert _run_of(fresh_db, flows).dispatch_state == "waiting" + + +def test_a_run_whose_upstream_automation_passed_over_that_time_is_skipped( + fresh_db, make_automation +): + """An upstream automation that moved past the scheduled time without a run there will never produce the input, so the run is skipped.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + # As if the upstream automation's missed run at 01:00 were left out by a bounded catch-up. + snapper.cursor = SCHEDULED_AT + fresh_db.session.commit() + + returns = _tick() + + assert returns[flows.id]["dispatch_state"] == "skipped" + assert _run_of(fresh_db, flows).plan["upstream-runs"][0]["state"] == "passed-over" + assert returns[flows.id]["message"] == ( + f"Skipped: Snap meter readings (automation {snapper.id}) did not run at 2026-08-05 01:00 UTC." + ) + + +@pytest.mark.parametrize("reset", ["reactivated", "new cron string"]) +def test_a_run_goes_ahead_when_its_upstream_automation_was_reset_after_that_time( + fresh_db, make_automation, freeze_server_now, reset +): + """A reset recurrence is not due at the times before the reset, so what waits for it goes ahead, rather than being skipped until its next run.""" + snapper = make_automation("Snap meter readings", cronstr="0 0 * * *") + flows = make_automation("Compute flows", depends_on=[snapper]) + if reset == "reactivated": + snapper.active = False + fresh_db.session.commit() + freeze_server_now(datetime(2026, 8, 5, 0, 30, tzinfo=timezone.utc)) + if reset == "reactivated": + update_automation(snapper, active=True) + else: + update_automation(snapper, cronstr="1 0 * * *") + fresh_db.session.commit() + + returns = _tick() + + assert returns[flows.id]["dispatch_state"] == "queued" + upstream = _run_of(fresh_db, flows).plan["upstream-runs"][0] + assert upstream["state"] == "not-scheduled" + due = "00:00" if reset == "reactivated" else "00:01" + assert upstream["summary"].endswith( + f"was not due at 2026-08-05 {due} UTC, as its recurrence was reset after that." + ) + + +@pytest.mark.parametrize("upstream_fails", [False, True]) +def test_a_run_waits_for_the_run_its_upstream_automation_caught_up_with( + fresh_db, app, make_automation, failing_value, upstream_fails +): + """After an outage, missed upstream runs are coalesced into the latest one, which a run waiting for an earlier one waits for instead.""" + puller = make_automation( + "Pull meter data", cronstr="*/15 * * * *", value=-1 if upstream_fails else 1 + ) + puller.cursor = datetime(2026, 8, 5, 0, 45, tzinfo=timezone.utc) + fresh_db.session.commit() + forecast = make_automation("Forecast", depends_on=[puller]) + + # The runner comes back at 01:16, so the puller's run at 01:00 is coalesced into the one at 01:15. + returns = _tick(datetime(2026, 8, 5, 1, 16, 30, tzinfo=timezone.utc)) + assert returns[forecast.id]["dispatch_state"] == "waiting" + catch_up_run = _run_at( + fresh_db, puller, datetime(2026, 8, 5, 1, 15, tzinfo=timezone.utc) + ) + _tick(datetime(2026, 8, 5, 1, 17, 30, tzinfo=timezone.utc)) + assert _run_at(fresh_db, forecast, SCHEDULED_AT).plan["upstream-runs"][0][ + "summary" + ] == ( + f"Pull meter data (automation {puller.id}, run {catch_up_run.id}), which caught up with its run due at 2026-08-05 01:00 UTC, has yet to end." + ) + + _work(app, catch_up_run) + + forecast_run = _run_at(fresh_db, forecast, SCHEDULED_AT) + if upstream_fails: + assert forecast_run.execution_state == "skipped" + else: + assert forecast_run.dispatch_state == "queued" + _work(app, forecast_run) + assert _run_at(fresh_db, forecast, SCHEDULED_AT).execution_state == ( + "succeeded" + ) + + +def test_a_run_goes_ahead_without_an_inactive_upstream_automation( + fresh_db, make_automation +): + """An upstream automation that was never going to run at that time is left out, and the record says so.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + snapper.active = False + fresh_db.session.commit() + + returns = _tick() + + assert returns[flows.id]["dispatch_state"] == "queued" + assert _run_of(fresh_db, flows).plan["upstream-runs"] == [ + { + "automation": snapper.id, + "allow-failure": False, + "due-at": SCHEDULED_AT.isoformat(), + "run": None, + "state": "not-scheduled", + "summary": f"Snap meter readings (automation {snapper.id}) is inactive, so it was not due at 2026-08-05 01:00 UTC.", + } + ] + + +def test_a_run_on_demand_does_not_wait(fresh_db, app, make_automation): + """Running an automation on demand means running it now, so it does not wait for the automations it depends on.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + + returns = run_automation(flows) + + assert returns["n_jobs"] == 1 + assert app.queues["ingestion"].count == 1 + + +def _run_at(db, automation: Automation, scheduled_at: datetime) -> AutomationRun: + db.session.expire_all() + return db.session.scalars( + select(AutomationRun).where( + AutomationRun.automation_id == automation.id, + AutomationRun.scheduled_at == scheduled_at, + ) + ).one() + + +def test_an_hourly_run_waits_for_the_upstream_run_due_at_its_own_time( + fresh_db, app, make_automation +): + """An hourly automation can wait for one that recurs every 15 minutes, and its run at 01:00 waits for the upstream run due at 01:00.""" + puller = make_automation("Pull meter data", cronstr="*/15 * * * *") + # Only the upstream run at 01:00 is due, rather than those at 00:15, 00:30 and 00:45, too. + puller.cursor = datetime(2026, 8, 5, 0, 45, tzinfo=timezone.utc) + fresh_db.session.commit() + forecast = make_automation("Forecast", depends_on=[puller]) + + returns = _tick() + + assert returns[forecast.id]["dispatch_state"] == "waiting" + puller_run = _run_at(fresh_db, puller, SCHEDULED_AT) + upstream = _run_at(fresh_db, forecast, SCHEDULED_AT).plan["upstream-runs"][0] + assert (upstream["due-at"], upstream["run"]) == ( + SCHEDULED_AT.isoformat(), + puller_run.id, + ) + + _work(app, puller_run) + + assert _run_at(fresh_db, forecast, SCHEDULED_AT).dispatch_state == "queued" + + +@pytest.mark.parametrize("upstream_fails", [False, True]) +def test_a_quarter_hourly_run_waits_for_the_latest_hourly_upstream_run( + fresh_db, app, make_automation, failing_value, upstream_fails +): + """A run every 15 minutes waits for the latest run of an hourly upstream automation. + + Its runs at 01:00 and 01:15 both wait for the upstream run due at 01:00, and both are released once that one ended. + """ + quarter_past = datetime(2026, 8, 5, 1, 15, tzinfo=timezone.utc) + puller = make_automation("Pull meter data", value=-1 if upstream_fails else 1) + forecast = make_automation("Forecast", depends_on=[puller], cronstr="*/15 * * * *") + forecast.cursor = datetime(2026, 8, 5, 0, 45, tzinfo=timezone.utc) + fresh_db.session.commit() + + _tick() + returns = _tick(datetime(2026, 8, 5, 1, 15, 30, tzinfo=timezone.utc)) + + assert returns[forecast.id]["dispatch_state"] == "waiting" + later_run = _run_at(fresh_db, forecast, quarter_past) + assert later_run.plan["upstream-runs"][0]["due-at"] == SCHEDULED_AT.isoformat() + + _work(app, _run_at(fresh_db, puller, SCHEDULED_AT)) + + expected = "skipped" if upstream_fails else "succeeded" + for scheduled_at in (SCHEDULED_AT, quarter_past): + run = _run_at(fresh_db, forecast, scheduled_at) + if not upstream_fails: + assert run.dispatch_state == "queued" + _work(app, run) + run = _run_at(fresh_db, forecast, scheduled_at) + assert run.execution_state == expected + + +@pytest.mark.parametrize("upstream_fails", [False, True]) +def test_running_a_chain_on_demand( + fresh_db, app, make_automation, failing_value, upstream_fails +): + """Run an automation on demand along with what waits for it: each run waits for the one before it, and a failure skips the rest.""" + snapper = make_automation("Snap meter readings", value=-1 if upstream_fails else 1) + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + # An inactive automation does not run, and neither does what only waits for it. + paused = make_automation("Paused", depends_on=[snapper]) + paused.active = False + make_automation("Waits for paused", depends_on=[paused]) + fresh_db.session.commit() + + runs, returns, excluded = run_automation_chain(snapper, "downstream", now=TICK) + dependent_runs = runs[1:] + + assert [(step.id, reason) for step, reason in excluded] == [ + (paused.id, "It is inactive.") + ] + assert [run.automation_id for run in dependent_runs] == [flows.id, totals.id] + assert [run.dispatch_state for run in dependent_runs] == ["waiting", "waiting"] + snapper_run = fresh_db.session.get(AutomationRun, returns["run_id"]) + assert snapper_run.trigger == "on-demand" + assert dependent_runs[1].plan["chain-upstream-runs"] == [ + {"automation": flows.id, "run": dependent_runs[0].id} + ] + # Their recurrence is left alone. + assert {snapper.cursor, flows.cursor, totals.cursor} == {CREATED_AT} + + _work(app, snapper_run) + if upstream_fails: + for run in dependent_runs: + assert _run_at(fresh_db, run.automation, SCHEDULED_AT).execution_state == ( + "skipped" + ) + return + flows_run = _run_at(fresh_db, flows, SCHEDULED_AT) + assert flows_run.dispatch_state == "queued" + assert _run_at(fresh_db, totals, SCHEDULED_AT).dispatch_state == "waiting" + _work(app, flows_run) + totals_run = _run_at(fresh_db, totals, SCHEDULED_AT) + assert totals_run.dispatch_state == "queued" + _work(app, totals_run) + assert _run_at(fresh_db, totals, SCHEDULED_AT).execution_state == "succeeded" + + +def _states(fresh_db, runs: list[AutomationRun]) -> dict[int, tuple[str, str]]: + fresh_db.session.expire_all() + return { + run.automation_id: (run.dispatch_state, run.execution_state) for run in runs + } + + +def test_running_with_upstream_works_its_way_down_and_stops_at_the_automation( + fresh_db, app, make_automation +): + """What an automation waits for runs first, what waits for it does not run, and neither does a branch next to it.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + make_automation("Sum further", depends_on=[totals]) + make_automation("Side branch", depends_on=[snapper]) + + runs, _, not_running = run_automation_chain(totals, "upstream", now=TICK) + + assert [run.automation_id for run in runs] == [snapper.id, flows.id, totals.id] + assert not_running == [] + assert [run.trigger for run in runs] == ["on-demand"] * 3 + assert _states(fresh_db, runs) == { + snapper.id: ("queued", "pending"), + flows.id: ("waiting", "pending"), + totals.id: ("waiting", "pending"), + } + assert {snapper.cursor, flows.cursor, totals.cursor} == {CREATED_AT} + _work(app, runs[0]) + assert _states(fresh_db, runs)[flows.id][0] == "queued" + assert _states(fresh_db, runs)[totals.id][0] == "waiting" + _work(app, runs[1]) + _work(app, runs[2]) + assert _states(fresh_db, runs)[totals.id] == ("queued", "succeeded") + + +def test_running_with_upstream_starts_every_branch_and_waits_for_all_of_them( + fresh_db, app, make_automation +): + """Several branches lead up to an automation, each starting right away, and it runs once the last of them ended.""" + prices = make_automation("Pull prices") + meters = make_automation("Pull meters") + forecast = make_automation("Forecast", depends_on=[prices, meters]) + + runs, _, _ = run_automation_chain(forecast, "upstream", now=TICK) + + assert _states(fresh_db, runs) == { + prices.id: ("queued", "pending"), + meters.id: ("queued", "pending"), + forecast.id: ("waiting", "pending"), + } + by_automation = {run.automation_id: run for run in runs} + _work(app, by_automation[prices.id]) + assert _states(fresh_db, runs)[forecast.id][0] == "waiting" + _work(app, by_automation[meters.id]) + assert _states(fresh_db, runs)[forecast.id][0] == "queued" + + +@pytest.mark.parametrize("allow_failure", [False, True]) +def test_a_failure_above_stops_an_automation_run_with_upstream( + fresh_db, app, make_automation, failing_value, allow_failure +): + """The automation is skipped if what it waits for failed, unless its dependency allows for that.""" + snapper = make_automation("Snap meter readings", value=-1) + flows = make_automation( + "Compute flows", depends_on=[snapper], allow_failure=allow_failure + ) + + runs, _, _ = run_automation_chain(flows, "upstream", now=TICK) + _work(app, runs[0]) + + assert _states(fresh_db, runs)[snapper.id][1] == "failed" + assert _states(fresh_db, runs)[flows.id] == ( + ("queued", "pending") if allow_failure else ("queued", "skipped") + ) + + +def test_a_branch_can_be_excluded_from_a_run_with_upstream( + fresh_db, app, make_automation +): + """Leaving out an automation takes along what only it needs, and what waits for it goes ahead without it.""" + tariffs = make_automation("Pull tariffs") + prices = make_automation("Compute prices", depends_on=[tariffs]) + meters = make_automation("Pull meters") + forecast = make_automation("Forecast", depends_on=[prices, meters]) + + runs, _, excluded = run_automation_chain( + forecast, "upstream", exclude=[prices.id], now=TICK + ) + + assert [run.automation_id for run in runs] == [meters.id, forecast.id] + assert [(step.id, reason) for step, reason in excluded] == [ + (prices.id, "You left it out.") + ] + _work(app, runs[0]) + assert _states(fresh_db, runs)[forecast.id][0] == "queued" + + +def test_what_may_not_run_is_reported_and_takes_along_what_is_above_it( + fresh_db, make_automation +): + """An automation that is inactive or not allowed does not run, and nothing would use the results of what is above it.""" + tariffs = make_automation("Pull tariffs") + public = make_automation("Public prices", depends_on=[tariffs]) + paused = make_automation("Paused") + paused.active = False + meters = make_automation("Pull meters") + forecast = make_automation("Forecast", depends_on=[public, paused, meters]) + + chain, excluded = get_chain_automations( + forecast, "upstream", may_run=lambda automation: automation.id != public.id + ) + + assert chain == [meters, forecast] + assert {(step.id, reason) for step, reason in excluded} == { + (public.id, "You may not run this automation."), + (paused.id, "It is inactive."), + } + + +@pytest.mark.parametrize("released_by", ["upstream job", "runner tick"]) +def test_an_inactive_automation_run_with_upstream_runs_once_they_ended( + fresh_db, app, make_automation, monkeypatch, released_by +): + """Inactive automations can be run on demand, which goes for one run with what is upstream of it, too.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + flows.active = False + fresh_db.session.commit() + + runs, _, _ = run_automation_chain(flows, "upstream", now=TICK) + with monkeypatch.context() as patch: + if released_by == "runner tick": + patch.setattr( + automation_services, + "release_automation_runs_waiting_for", + lambda run: None, + ) + _work(app, runs[0]) + if released_by == "runner tick": + assert _states(fresh_db, runs)[flows.id][0] == "waiting" + _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + + assert _states(fresh_db, runs)[flows.id] == ("queued", "pending") + + +def test_deactivating_an_automation_cancels_its_waiting_runs( + fresh_db, app, make_automation +): + """A waiting run of a deactivated automation would go ahead late once it is reactivated, so it is canceled, and what waits for it goes ahead.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + _tick() + assert _run_of(fresh_db, flows).dispatch_state == "waiting" + + update_automation(flows, active=False) + fresh_db.session.commit() + _work(app, _run_of(fresh_db, snapper)) + + flows_run = _run_of(fresh_db, flows) + assert (flows_run.dispatch_state, flows_run.execution_state) == ( + "queued", + "canceled", + ) + assert "deactivated" in flows_run.last_error_message + _tick(datetime(2026, 8, 5, 1, 1, 30, tzinfo=timezone.utc)) + totals_run = _run_of(fresh_db, totals) + assert totals_run.dispatch_state == "queued" + assert totals_run.plan["upstream-runs"][0]["state"] == "not-scheduled" + + +def test_an_automation_without_upstream_automations_just_runs( + fresh_db, app, make_automation +): + snapper = make_automation("Snap meter readings") + + runs, _, _ = run_automation_chain(snapper, "upstream", now=TICK) + + assert [run.automation_id for run in runs] == [snapper.id] + assert _states(fresh_db, runs)[snapper.id][0] == "queued" + + +def test_the_plan_of_a_run_says_what_will_and_what_will_not_run( + fresh_db, make_automation +): + """A plan lists the automations in the order they run, and says why one does not, giving nothing away about one that may not be read.""" + public = make_automation("Public prices") + meters = make_automation("Pull meters") + forecast = make_automation("Forecast", depends_on=[public, meters]) + schedule = make_automation("Schedule", depends_on=[forecast]) + + plan = plan_automation_chain( + forecast, + "upstream", + may_read=lambda automation: automation.id != public.id, + may_run=lambda automation: automation.id != public.id, + ) + + assert plan["excluded"] == [ + { + "automation": public.id, + "reason": "You may not run this automation.", + "waited-for-by": [forecast.id], + } + ] + assert [step["automation"] for step in plan["steps"]] == [meters.id, forecast.id] + assert plan["steps"][1]["waits-for"] == [meters.id] + assert plan["steps"][1]["name"] == "Forecast" + + downwards = plan_automation_chain( + forecast, + "downstream", + may_read=lambda automation: True, + may_run=lambda automation: automation.id != schedule.id, + ) + assert [step["automation"] for step in downwards["steps"]] == [forecast.id] + assert [step["automation"] for step in downwards["excluded"]] == [schedule.id] + # What the user may read says when it last succeeded, which it has not yet. + assert downwards["excluded"][0]["last-succeeded-at"] is None + assert downwards["excluded"][0]["waited-for-by"] == [] + + +def test_a_plan_says_when_an_excluded_automation_last_succeeded( + fresh_db, app, make_automation +): + """What waits for an excluded automation goes by the results it last produced, so the plan says when that was.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + _work(app, _run_of(fresh_db, snapper)) + snapper_run = _run_of(fresh_db, snapper) + assert snapper_run.execution_state == "succeeded" + + plan = plan_automation_chain(flows, "upstream", exclude=[snapper.id]) + + assert plan["excluded"][0]["last-succeeded-at"] == ( + snapper_run.execution_completed_at.isoformat() + ) + assert plan["excluded"][0]["waited-for-by"] == [flows.id] + + +def test_a_plan_lists_a_diamond_in_the_order_it_runs(fresh_db, make_automation): + """An automation comes after everything it waits for, also where it is reached sooner along another path.""" + prices = make_automation("Pull prices") + tariffs = make_automation("Compute tariffs", depends_on=[prices]) + # Forecast reaches prices directly and through tariffs, so prices and tariffs are equally near to it. + forecast = make_automation("Forecast", depends_on=[prices, tariffs]) + + plan = plan_automation_chain(forecast, "upstream") + assert [step["automation"] for step in plan["steps"]] == [ + prices.id, + tariffs.id, + forecast.id, + ] + plan = plan_automation_chain(prices, "downstream") + assert [step["automation"] for step in plan["steps"]] == [ + prices.id, + tariffs.id, + forecast.id, + ] + + +def test_a_chain_run_on_demand_and_a_scheduled_run_keep_apart( + fresh_db, app, make_automation +): + """Runs on demand can share a minute with the scheduled runs, and a scheduled run only ever waits for a scheduled upstream run.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + _tick() + + _, returns, _ = run_automation_chain(snapper, "downstream", now=TICK) + on_demand_snapper_run = fresh_db.session.get(AutomationRun, returns["run_id"]) + _work(app, on_demand_snapper_run) + + fresh_db.session.expire_all() + runs = fresh_db.session.scalars( + select(AutomationRun).where(AutomationRun.automation_id == flows.id) + ).all() + states = {run.trigger: run.dispatch_state for run in runs} + assert states == {"schedule": "waiting", "on-demand": "queued"} + + +def test_automations_waiting_for_each_other_can_change_their_schedules( + fresh_db, make_automation +): + """Runs wait for the latest upstream run due at or before their own time, so either automation can move to another schedule.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + + update_automation(snapper, cronstr="*/15 * * * *") + update_automation(flows, cronstr="5 * * * *") + fresh_db.session.commit() + + assert (snapper.cronstr, flows.cronstr) == ("*/15 * * * *", "5 * * * *") + assert _dependency_ids(fresh_db, flows) == [snapper.id] + + +def test_cycles_and_self_dependencies_are_refused(fresh_db, make_automation): + """An automation that ends up waiting for itself would never run.""" + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + totals = make_automation("Sum flows", depends_on=[flows]) + + with pytest.raises(ValidationError) as e: + update_automation(snapper, dependencies=[{"upstream": totals}]) + assert "waiting for itself" in str(e.value.messages["depends-on"]) + with pytest.raises(ValidationError): + update_automation(snapper, dependencies=[{"upstream": snapper}]) + fresh_db.session.rollback() + assert _dependency_ids(fresh_db, snapper) == [] + + +def test_deleting_an_upstream_automation_drops_the_dependency( + fresh_db, make_automation +): + snapper = make_automation("Snap meter readings") + flows = make_automation("Compute flows", depends_on=[snapper]) + + fresh_db.session.delete(snapper) + fresh_db.session.commit() + + assert _dependency_ids(fresh_db, flows) == [] + + +def _dependency_ids(db, automation: Automation) -> list[int]: + db.session.expire_all() + return [ + dependency.upstream_id + for dependency in db.session.get(Automation, automation.id).dependencies + ] diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 4ba933f8e3..1f3c9bd796 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -7,7 +7,7 @@ }, "termsOfService": null, "title": "FlexMeasures", - "version": "1.1.0" + "version": "1.1.1" }, "externalDocs": { "description": "FlexMeasures runs on the open source FlexMeasures technology. Read the docs here.", @@ -3538,7 +3538,7 @@ }, "get": { "summary": "Get details of one automation defined on an asset.", - "description": "In addition to the fields shown when listing automations, the response shows\nthe automation's parameters (forecast parameters or a schedule trigger message),\nthe data source it records under, as its `source` (null for schedule automations), including the configuration its data generator was set up with,\nthe sensors it reads from and writes to,\ndurable run status, and counts of recently created jobs, per job status.\nNote that jobs in Redis have a limited TTL, so not all past jobs will be counted, while durable run status records queueing attempts and outcomes even after those jobs expire.\nThe cursor is the time of the most recent run the automation committed to, in the automation's own timezone; runs at or before it are never queued again.\nIt advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded.\n", + "description": "In addition to the fields shown when listing automations, the response shows\nthe automation's parameters (forecast parameters or a schedule trigger message),\nthe data source it records under, as its `source` (null for schedule automations), including the configuration its data generator was set up with,\nthe sensors it reads from and writes to,\ndurable run status, and counts of recently created jobs, per job status.\nNote that jobs in Redis have a limited TTL, so not all past jobs will be counted, while durable run status records queueing attempts and outcomes even after those jobs expire.\nThe cursor is the time of the most recent run the automation committed to, in the automation's own timezone; runs at or before it are never queued again.\nIt advances just before queueing, so it does not indicate that queueing or the forecast itself succeeded.\n\nPass `include=waited-for-by` to also list the automations that wait for this one, such as before deleting it.\nThey are looked up across all automations, and one you may not read is listed as a bare ID.\n", "security": [ { "ApiKeyAuth": [] @@ -3562,6 +3562,21 @@ "schema": { "type": "integer" } + }, + { + "in": "query", + "name": "include", + "required": false, + "description": "Extra fields to include, namely `waited-for-by`.", + "schema": { + "type": "array", + "items": { + "type": "string", + "enum": [ + "waited-for-by" + ] + } + } } ], "responses": { @@ -3665,8 +3680,8 @@ ] }, "patch": { - "summary": "Update an automation's name, cron string, timezone or activation status.", - "description": "Any subset of the fields `name`, `cronstr`, `timezone` and `active` can be sent.\nChanging the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change,\nso runs from before it are not caught up on, while a run due in that very minute still is.\nOther automation fields cannot be updated; instead, create a new automation.\nRequires permission to add data under the asset.\n", + "summary": "Update an automation's name, cron string, timezone, activation status or dependencies.", + "description": "Any subset of the fields `name`, `cron`, `timezone`, `active` and `depends-on` can be sent.\nChanging the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change,\nso runs from before it are not caught up on, while a run due in that very minute still is.\nAn automation's runs wait for the latest runs of the automations in `depends-on` that are due at or before their own scheduled time,\nso either side of a dependency can change its recurrence or timezone.\nOther automation fields cannot be updated; instead, create a new automation.\nRequires permission to add data under the asset.\n", "security": [ { "ApiKeyAuth": [] @@ -3704,6 +3719,17 @@ "value": { "active": false } + }, + "wait_for_another_automation": { + "summary": "Wait for the runs of automation 4", + "description": "Each run waits for the latest run of automation 4 due at or before its own time, and is skipped if that one failed.", + "value": { + "depends-on": [ + { + "automation": 4 + } + ] + } } } } @@ -3734,6 +3760,118 @@ ] } }, + "/api/v3_0/assets/{id}/automations/{automation-id}/run-plan": { + "get": { + "summary": "See what running an automation along with what is upstream or downstream of it comes down to.", + "description": "Nothing is run.\nRuns always flow downstream; `include` says which part of the chain is included.\nThe automations that will run are listed under `steps`, in chain order, each saying what it waits for among these steps.\nThe automations that will not run are listed under `excluded`, each with the reason,\nwhich of the steps wait for it, and, if you may read it, when it last succeeded (null if it never did).\nThey still run on their own schedule, just not because of this request,\nand the steps that wait for them go by the results they last produced.\nAn automation you may not read stays a bare ID.\n", + "security": [ + { + "ApiKeyAuth": [] + } + ], + "parameters": [ + { + "in": "path", + "name": "id", + "required": true, + "description": "ID of the asset the automation is defined on.", + "schema": { + "type": "integer" + } + }, + { + "in": "path", + "name": "automation-id", + "required": true, + "description": "ID of the automation.", + "schema": { + "type": "integer" + } + }, + { + "in": "query", + "name": "include", + "required": true, + "description": "Which part of the chain to include, upstream or downstream of this automation.", + "schema": { + "type": "string", + "enum": [ + "upstream", + "downstream" + ] + } + }, + { + "in": "query", + "name": "exclude", + "required": false, + "description": "IDs of automations not to run.", + "schema": { + "type": "array", + "items": { + "type": "integer" + } + } + } + ], + "responses": { + "200": { + "description": "PROCESSED", + "content": { + "application/json": { + "examples": { + "run-plan": { + "summary": "One automation does not run", + "value": { + "steps": [ + { + "automation": 7, + "waits-for": [], + "name": "Forecast", + "asset": 3, + "asset-name": "Site" + } + ], + "excluded": [ + { + "automation": 5, + "reason": "You may not run this automation.", + "waited-for-by": [ + 7 + ], + "name": "Day-ahead prices", + "asset": 2, + "asset-name": "Market", + "last-succeeded-at": null + } + ] + } + } + } + } + } + }, + "401": { + "description": "UNAUTHORIZED" + }, + "403": { + "description": "INVALID_SENDER" + }, + "404": { + "description": "NOT_FOUND" + }, + "422": { + "description": "UNPROCESSABLE_ENTITY" + }, + "429": { + "description": "TOO_MANY_REQUESTS - You called the API more often than your rate limit allows. Wait for as long as the Retry-After header says, then try again." + } + }, + "tags": [ + "Assets" + ] + } + }, "/api/v3_0/assets/{id}/automations": { "get": { "summary": "Get all automations defined on an asset.", @@ -3818,7 +3956,7 @@ }, "post": { "summary": "Create an automation on an asset.", - "description": "Create a recurring forecasting, scheduling, reporting or plugin-defined task on the asset.\nThe parameters are validated by the schema matching the automation type:\nforecast parameters for type `forecasting`,\na schedule trigger message (without the asset id) for type `scheduling`,\nreport parameters for type `reporting`, or the registered plugin schema.\nRequires permission to add data under the asset.\n\nAn automation runs again and again, so its parameters cannot fix a moment in time:\na `start`, `end` or `prior` among them is refused.\nSay instead how the period each run covers relates to that run,\nwith two of `start-offset`, `end-offset` and `duration`.\nThe offsets are chains of comma-separated Pandas offsets, plus `DB` (day begin) and `HB` (hour begin),\napplied to the time the run was due, on the automation's own clock.\nLeave the timing out to start at the time of each run,\nor, for a report, to cover the period since the last successful report.\n\nThe automation can only involve sensors that you have access to yourself:\nread access to the sensors it reads data from,\nand permission to record data on the sensors it writes to.\n", + "description": "Create a recurring forecasting, scheduling, reporting or plugin-defined task on the asset.\nThe parameters are validated by the schema matching the automation type:\nforecast parameters for type `forecasting`,\na schedule trigger message (without the asset id) for type `scheduling`,\nreport parameters for type `reporting`, or the registered plugin schema.\nRequires permission to add data under the asset.\n\nAn automation runs again and again, so its parameters cannot fix a moment in time:\na `start`, `end` or `prior` among them is refused.\nSay instead how the period each run covers relates to that run,\nwith two of `start-offset`, `end-offset` and `duration`.\nThe offsets are chains of comma-separated Pandas offsets, plus `DB` (day begin) and `HB` (hour begin),\napplied to the time the run was due, on the automation's own clock.\nLeave the timing out to start at the time of each run,\nor, for a report, to cover the period since the last successful report.\n\nThe automation can only involve sensors that you have access to yourself:\nread access to the sensors it reads data from,\nand permission to record data on the sensors it writes to.\n\nAn automation can wait for other automations that it consumes the results of, listed in `depends-on`.\nEach of its runs then waits for their latest runs due at or before its own scheduled time, and queues its jobs once those ended,\nor is skipped if one of them failed, unless that dependency has `allow-failure` set.\nThe automations it waits for can be on any asset you can read, and may recur on another schedule.\n", "security": [ { "ApiKeyAuth": [] @@ -3907,6 +4045,68 @@ ] } }, + "/api/v3_0/assets/automations": { + "get": { + "summary": "List automations across assets.", + "description": "List the automations on public assets, or every automation you may read, such as to pick one to wait for.\nJob counts are left out, as those are looked up per asset.\nThe automations are listed by ID.\nWithout `page`, all of them are listed.\nPass `page`, and optionally `per-page` (default 10), to list one page of them, along with `num-records`, how many there are in all.\n", + "security": [ + { + "ApiKeyAuth": [] + } + ], + "parameters": [ + { + "in": "query", + "name": "scope", + "required": true, + "description": "`public` for the automations on public assets, or `all` for every automation you may read.", + "schema": { + "type": "string", + "enum": [ + "public", + "all" + ] + } + }, + { + "in": "query", + "name": "page", + "required": false, + "description": "The page to list, starting at 1.", + "schema": { + "type": "integer" + } + }, + { + "in": "query", + "name": "per-page", + "required": false, + "description": "How many automations to list per page.", + "schema": { + "type": "integer", + "default": 10 + } + } + ], + "responses": { + "200": { + "description": "PROCESSED" + }, + "401": { + "description": "UNAUTHORIZED" + }, + "422": { + "description": "UNPROCESSABLE_ENTITY" + }, + "429": { + "description": "TOO_MANY_REQUESTS - You called the API more often than your rate limit allows. Wait for as long as the Retry-After header says, then try again." + } + }, + "tags": [ + "Assets" + ] + } + }, "/api/v3_0/assets/{id}/chart": { "get": { "summary": "Download an embeddable chart with time series data", @@ -4919,7 +5119,7 @@ "/api/v3_0/assets/{id}/automations/{automation-id}/trigger": { "post": { "summary": "Trigger a single run of an automation.", - "description": "Run one automation now, once, in addition to its recurring runs.\nThis is useful to try out a new automation, to re-run one after fixing what made it fail,\nor to refresh its results after late input data arrived.\n\nThe automation runs with the parameters it was created with,\nand the jobs it queues are recorded as its jobs, just like the jobs of a recurring run.\n\nAn on-demand run does not affect the automation's recurrence: its cursor stays where it was,\nso the next recurring run still happens as scheduled, and a missed run is still caught up.\nInactive automations can be triggered, too, which is how you can try one out before activating it.\n", + "description": "Run one automation now, once, in addition to its recurring runs.\nThis is useful to try out a new automation, to re-run one after fixing what made it fail,\nor to refresh its results after late input data arrived.\n\nThe automation runs with the parameters it was created with,\nand the jobs it queues are recorded as its jobs, just like the jobs of a recurring run.\n\nAn on-demand run does not affect the automation's recurrence: its cursor stays where it was,\nso the next recurring run still happens as scheduled, and a missed run is still caught up.\nInactive automations can be triggered, too, which is how you can try one out before activating it.\n\nPass `with-upstream=true` to first run the active automations upstream of this one, which it waits for, directly or through others, and then this one,\nfor instance to bring it up to date with what its results are based on.\nPass `with-downstream=true` to also run the active automations downstream of this one, which wait for it, directly or through others,\nfor instance to re-run a chain of automations once the data at its start was backfilled.\nThe included automations run in chain order: each waits for the runs triggered along with it,\nand is skipped if one of those failed, unless its dependency allows for failure.\nAn automation that is inactive, or that you may not run, is excluded, along with what can only be reached through it.\nPass `exclude` to exclude automations yourself, such as one of several branches.\nSee the `run-plan` endpoint for what this would come down to.\nThe response lists the runs under `runs`, in chain order, and the excluded automations under `excluded`, each with the reason;\nits `job` is that of this automation's own run, if that started right away.\n", "security": [ { "ApiKeyAuth": [] @@ -4943,6 +5143,38 @@ "schema": { "type": "integer" } + }, + { + "in": "query", + "name": "with-upstream", + "required": false, + "description": "First run the active automations upstream of this one, which it waits for, directly or through others. Cannot be combined with `with-downstream`.", + "schema": { + "type": "boolean", + "default": false + } + }, + { + "in": "query", + "name": "with-downstream", + "required": false, + "description": "Also run the active automations downstream of this one, which wait for it, directly or through others.", + "schema": { + "type": "boolean", + "default": false + } + }, + { + "in": "query", + "name": "exclude", + "required": false, + "description": "With `with-upstream` or `with-downstream`, IDs of automations not to run.", + "schema": { + "type": "array", + "items": { + "type": "integer" + } + } } ], "responses": { @@ -4961,6 +5193,56 @@ "n-jobs": 2, "message": "Request has been accepted for processing." } + }, + "triggered-with-downstream": { + "summary": "Automation run accepted, along with the automations downstream of it", + "value": { + "status": "ACCEPTED", + "job": "364bfd06-c1fa-430b-8d25-8f5a547651fb", + "job-url": "/api/v3_0/jobs/364bfd06-c1fa-430b-8d25-8f5a547651fb", + "n-jobs": 1, + "message": "Started 1 run(s), and 1 run(s) wait for them.", + "runs": [ + { + "automation": 7, + "run": 411, + "dispatch-state": "queued" + }, + { + "automation": 8, + "run": 412, + "dispatch-state": "waiting" + } + ], + "excluded": [] + } + }, + "triggered-with-upstream": { + "summary": "Automation run accepted, after the automations upstream of it", + "value": { + "status": "ACCEPTED", + "job": null, + "n-jobs": 1, + "message": "Started 1 run(s), and 1 run(s) wait for them.", + "runs": [ + { + "automation": 7, + "run": 411, + "dispatch-state": "queued" + }, + { + "automation": 8, + "run": 412, + "dispatch-state": "waiting" + } + ], + "excluded": [ + { + "automation": 5, + "reason": "You may not run this automation." + } + ] + } } } } @@ -6819,6 +7101,26 @@ ], "additionalProperties": false }, + "AutomationDependency": { + "type": "object", + "properties": { + "automation": { + "type": "integer", + "description": "ID of the automation to wait for. It may recur on another schedule.", + "example": 4 + }, + "allow-failure": { + "type": "boolean", + "default": false, + "description": "Whether a run goes ahead when the upstream run failed or did not happen, rather than being skipped.", + "example": false + } + }, + "required": [ + "automation" + ], + "additionalProperties": false + }, "AutomationCreationSchema": { "type": "object", "properties": { @@ -6874,6 +7176,19 @@ ], "description": "ID of an existing data source to reuse, instead of naming a `data-generator` and its `config`.", "example": 6 + }, + "depends-on": { + "type": "array", + "description": "The automations whose runs each run of this automation waits for: their latest runs due at or before its own scheduled time. A run queues its jobs once these have ended, and is skipped if one of them failed, unless its dependency allows for failure. Upstream automations can be on any asset you can read, and may recur on another schedule. A run on demand does not wait.", + "example": [ + { + "automation": 4, + "allow-failure": false + } + ], + "items": { + "$ref": "#/components/schemas/AutomationDependency" + } } }, "required": [ @@ -6900,6 +7215,19 @@ }, "active": { "type": "boolean" + }, + "depends-on": { + "type": "array", + "description": "The automations whose runs each run of this automation waits for: their latest runs due at or before its own scheduled time. A run queues its jobs once these have ended, and is skipped if one of them failed, unless its dependency allows for failure. Upstream automations can be on any asset you can read, and may recur on another schedule. A run on demand does not wait. Replaces the automations it depended on; send an empty list to have its runs wait for none.", + "example": [ + { + "automation": 4, + "allow-failure": true + } + ], + "items": { + "$ref": "#/components/schemas/AutomationDependency" + } } }, "additionalProperties": false diff --git a/flexmeasures/ui/templates/assets/asset_automations.html b/flexmeasures/ui/templates/assets/asset_automations.html index 1a7c23a74f..f188cf147e 100644 --- a/flexmeasures/ui/templates/assets/asset_automations.html +++ b/flexmeasures/ui/templates/assets/asset_automations.html @@ -8,6 +8,36 @@ {% block breadcrumbs %} {{ super() }} {% endblock %}