diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst index ac97731fc4..ee5298c732 100644 --- a/documentation/api/change_log.rst +++ b/documentation/api/change_log.rst @@ -8,6 +8,7 @@ API change log v3.0-35 | November xx, 2026 """""""""""""""""""""""""""" +- ``POST /api/v3_0/assets//automations`` and ``PATCH /api/v3_0/assets//automations/`` now accept ``monitored``, which says whether ``flexmeasures monitor automations`` reports the automation's failed runs (``true`` by default), and the automation endpoints return it. Only the host's admins and the account-admins of the asset's organisation may create an unmonitored automation or switch monitoring off or on; others get a ``403``. - Renamed ``schedule-revision`` to ``revision`` in automation responses and in each run listed under ``run-stats``. The field counts the resets of an automation's recurrence, not anything about schedules; it was released under the old name in v3.0-34 (FlexMeasures v1.1.0), and is read-only, so only clients reading it need to switch. v3.0-34 | November 05, 2026 diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 9f0ccb3aa9..8baae9dfe6 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -11,6 +11,7 @@ v1.2.0 | December xx, 2026 New features ------------- +* Hosts can be emailed a digest of the automation runs that failed, got stuck or could not be queued, with ``flexmeasures monitor automations``, run from cron; each run is reported once, repeated failures are collapsed per automation, with the new ``FLEXMEASURES_BASE_URL`` setting each automation links to its *Automations* page, and admins can switch an automation's monitoring off [see `PR #2671 `_ and `PR #2702 `_] Infrastructure / Support ------------------------- diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst index 68cc8cf7ae..d72a77605c 100644 --- a/documentation/cli/change_log.rst +++ b/documentation/cli/change_log.rst @@ -4,6 +4,12 @@ FlexMeasures CLI Changelog ************************** +since v1.2.0 | December xx, 2026 +================================= + +* Add ``flexmeasures monitor automations`` to email a digest of the automation runs that failed since the last check: runs whose execution failed, which have been executing for too long (``--stuck-after-minutes``), or whose dispatch failed for good. Each run is reported once, and repeated failures are collapsed per automation. +* Add ``flexmeasures add automation --unmonitored`` and ``flexmeasures edit automation --monitored/--unmonitored`` to keep an automation's failed runs out of ``flexmeasures monitor automations``, or to report them again. ``flexmeasures show automations`` shows whether each automation is monitored. + since v1.1.0 | November 05, 2026 ================================= diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst index c37693ae3c..095c125dc0 100644 --- a/documentation/cli/commands.rst +++ b/documentation/cli/commands.rst @@ -107,6 +107,7 @@ of which some are referred to in this documentation. ================================================= ======================================= ``flexmeasures monitor latest-run`` Check if the given task's last successful execution happened less than the allowed time ago. ``flexmeasures monitor last-seen`` Check if given users last contact (via a request) happened less than the allowed time ago. +``flexmeasures monitor automations`` Email a digest of the automation runs that failed since the last check. ================================================= ======================================= diff --git a/documentation/configuration.rst b/documentation/configuration.rst index ebdbdd863f..82347bfcf7 100644 --- a/documentation/configuration.rst +++ b/documentation/configuration.rst @@ -878,11 +878,34 @@ Default: ``None`` FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run`` and ``flexmeasures monitor last-seen`` if no explicit user recipients are given. For example ``["fred@one.com", "wilma@two.com"]``. +E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run``, ``flexmeasures monitor last-seen`` and ``flexmeasures monitor automations`` if no explicit user recipients are given. For example ``["fred@one.com", "wilma@two.com"]``. Default: ``[]`` +.. _monitor_automations_stuck_after: + +FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +How long an automation run may execute before ``flexmeasures monitor automations`` reports it as stuck, for instance because the worker running its job died. +Its ``--stuck-after-minutes`` option overrides this setting. + +Default: ``timedelta(hours=6)`` + +.. _base_url: + +FLEXMEASURES_BASE_URL +^^^^^^^^^^^^^^^^^^^^^ + +Where this server is reached on the web, for instance ``"https://ems.example.com"``. +Emails sent from the CLI, outside of any request, use it to link to the web pages they are about. +For instance, the digest of ``flexmeasures monitor automations`` links each failing automation to the Automations page of its asset. +Without it, these emails carry no links. + +Default: ``None`` + + .. _redis-config: Redis diff --git a/documentation/features/automations.rst b/documentation/features/automations.rst index 03df6cf34a..009f950c32 100644 --- a/documentation/features/automations.rst +++ b/documentation/features/automations.rst @@ -287,6 +287,36 @@ This is what an operator needs to tell a run which failed before queueing anythi Editing an automation's cron string or timezone, or reactivating it, counts up its revision. Runs of the old and the new schedule therefore stay distinct, even when they fall on the same scheduled UTC time. +.. _monitoring_automations: + +Monitoring automation runs +-------------------------- + +As a host, you can have FlexMeasures email you a digest of the automation runs that failed, by running ``flexmeasures monitor automations`` regularly, for instance every hour: + +.. code-block:: bash + + 0 * * * * flexmeasures monitor automations + +A run is reported when its execution failed, when it has been executing for longer than six hours, or when its dispatch failed and will not be attempted again. +A forecast run therefore counts once its five dispatch attempts are used up, and a run of any other type after its first failed attempt. +The six hours are set by :ref:`monitor_automations_stuck_after`, and ``--stuck-after-minutes`` overrides that setting. + +Each run is reported once, so if one monitoring run is missed, the next one still reports what failed in the meantime. +Repeated failures of an automation are collapsed into one entry, saying how often it failed, when it first and last did, and with which error. +No email is sent when nothing failed. + +Every automation is monitored, unless it opts out, for instance an experiment whose failures nobody needs to hear about. +Untick *Monitored* on the Automations page, create it with ``flexmeasures add automation --unmonitored``, switch it with ``flexmeasures edit automation --monitored/--unmonitored``, or send ``"monitored": false`` to the API. +Its runs are then left out of every digest. +They are not marked as reported, though, so once the automation is monitored again, the next digest reports the runs that failed in the meantime, +saying how many of them failed while it was not monitored, and when monitoring was switched back on. +Only the host's admins and the account-admins of the asset's organisation may switch monitoring off or on, as it decides what the recipients of the digest hear about. +Switching it off only keeps the automation out of these digests; the automation runs as before. + +The digest goes to :ref:`default_monitoring_mail_recipients`, or to whoever you name with ``--recipient`` (a user ID or an email address, as often as needed). +The command stays out of ``flexmeasures jobs run-automations``, so that a problem with monitoring, such as an unreachable mail server, never holds up the automations themselves. + Running one automation on demand -------------------------------- diff --git a/documentation/host/error-monitoring.rst b/documentation/host/error-monitoring.rst index 1ff8d049cf..ac98a8ec1f 100644 --- a/documentation/host/error-monitoring.rst +++ b/documentation/host/error-monitoring.rst @@ -36,6 +36,17 @@ If you run distinct filters, such as separate checks per account, account group .. todo:: Adding roles and assigning them to accounts is not supported by the UI yet (user roles can be added in the UI). Account roles can be added with ``flexmeasures add account-role``. +Monitoring automations +----------------------- + +The CLI task ``flexmeasures monitor automations`` emails a digest of the automation runs that failed since its last check, with repeated failures collapsed per automation. +Run it from cron, for instance every hour; see :ref:`monitoring_automations` for what it reports. + +.. code-block:: bash + + $ flexmeasures monitor automations --recipient alerts@example.com + + Monitoring task runs --------------------- diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py index 39940cde22..bc910f670e 100644 --- a/flexmeasures/api/v3_0/assets.py +++ b/flexmeasures/api/v3_0/assets.py @@ -67,6 +67,7 @@ resolve_automation_sensors, run_automation, update_automation, + user_may_change_monitoring, ) from flexmeasures.data.models.charts import ASSET_CHART_BUILDERS from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType @@ -103,6 +104,7 @@ ) from flexmeasures.api.v3_0.utils import use_legacy_job_responses from flexmeasures.api.common.responses import ( + invalid_sender, unprocessable_entity, request_accepted_for_processing, ) @@ -505,6 +507,12 @@ def _eager_load_asset_relations_dumped_by( return query +# Switching monitoring off silences the digest for whoever receives it, so it is not up to everyone who manages automations. +MONITORING_IS_FOR_ADMINS = ( + "Only admins may switch the monitoring of an automation on or off." +) + + class AssetAPI(FlaskView): """ This API view exposes generic assets. @@ -1509,7 +1517,7 @@ def get_automations( description: | The response will be a list of automations: recurring forecasting, scheduling, reporting or plugin-defined tasks defined on the asset. Each entry shows the automation's ID, when it was created, - its type, name, activation status, and its recurrence, both as a cron string + its type, name, activation status, whether it is monitored, and its recurrence, both as a cron string and described in natural language. Each entry also shows the IANA timezone in which its cron expression is interpreted, and both its cursor and its next scheduled run as clock times in that same timezone (the next run is null while inactive). The next run excludes pending catch-up work. @@ -1556,6 +1564,7 @@ def get_automations( recurrence-description: "At 06:00" revision: 1 active: true + monitored: true job-stats: finished: 3 redis-connection-err: null @@ -1670,6 +1679,7 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset): recurrence-description: "At 06:00" revision: 1 active: true + monitored: true parameters: sensor: 2092 source: @@ -1876,6 +1886,10 @@ def post_automation(self, automation_data: dict, id: int, asset: GenericAsset): tags: - Assets """ + if automation_data.get("monitored") is False and not user_may_change_monitoring( + current_user, asset + ): + return invalid_sender(message=MONITORING_IS_FOR_ADMINS) try: automation, warnings = create_automation( asset, origin="API", check_permissions=True, **automation_data @@ -1912,13 +1926,13 @@ def patch_automation( self, automation_data: dict, id: int, automation_id: int, asset: GenericAsset ): """ - .. :quickref: Assets; Update an automation's name, cron string, timezone or activation status. + .. :quickref: Assets; Update an automation's name, cron string, timezone, activation status or monitoring. --- patch: - summary: Update an automation's name, cron string, timezone or activation status. + summary: Update an automation's name, cron string, timezone, activation status or monitoring. description: | - Any subset of the fields `name`, `cronstr`, `timezone` and `active` can be sent. + Any subset of the fields `name`, `cronstr`, `timezone`, `active` and `monitored` can be sent. Changing the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change, so runs from before it are not caught up on, while a run due in that very minute still is. Other automation fields cannot be updated; instead, create a new automation. @@ -1947,6 +1961,10 @@ def patch_automation( summary: Deactivate the automation value: active: false + stop-monitoring: + summary: Stop reporting the automation's failed runs + value: + monitored: false responses: 200: description: PROCESSED @@ -1966,6 +1984,13 @@ def patch_automation( return { "message": f"Asset {asset.id} has no automation with id {automation_id}." }, 404 + monitored = automation_data.get("monitored") + if ( + monitored is not None + and monitored != automation.monitored + and not user_may_change_monitoring(current_user, asset) + ): + return invalid_sender(message=MONITORING_IS_FOR_ADMINS) update_automation(automation, origin="API", **automation_data) db.session.commit() response = automation_schema.dump(automation) diff --git a/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py b/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py index 4fba308614..8d07f30dcd 100644 --- a/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py +++ b/flexmeasures/api/v3_0/tests/test_asset_copy_automations_fresh_db.py @@ -164,6 +164,8 @@ def test_the_output_scoped_automation_types_are_supported(): def test_copy_copies_direct_and_descendant_automations(fresh_db, automated_site): """Automations on the copied asset and on its descendants are copied, inactive and without run history.""" + # One original opted out of monitoring, so that the copies show they keep that choice rather than the default. + automated_site["meter_automation"].monitored = False asset_copy = copy_asset(automated_site["site"]) assert asset_copy.skipped_automations == [] @@ -187,6 +189,7 @@ def test_copy_copies_direct_and_descendant_automations(fresh_db, automated_site) assert copied.type == original.type assert copied.cronstr == original.cronstr assert copied.timezone == original.timezone + assert copied.monitored == original.monitored # A copy waits to be switched on, and starts without the original's run history. assert copied.active is False assert copied.cursor > OLD_CURSOR diff --git a/flexmeasures/api/v3_0/tests/test_automations_api.py b/flexmeasures/api/v3_0/tests/test_automations_api.py index fd3ef053e6..938c86449e 100644 --- a/flexmeasures/api/v3_0/tests/test_automations_api.py +++ b/flexmeasures/api/v3_0/tests/test_automations_api.py @@ -306,6 +306,7 @@ def test_post_automation( if expected_status_code == 201: assert response.json["name"] == "Posted schedules" assert response.json["active"] is True + assert response.json["monitored"] is True assert response.json["recurrence-description"] == "At 00:00" automation = db.session.get(Automation, response.json["id"]) assert automation.parameters == {"duration": "PT12H"} @@ -552,6 +553,151 @@ def test_post_and_patch_automation_timezone( assert response.status_code == 422 +@pytest.mark.parametrize( + "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True +) +def test_post_and_patch_automation_monitoring( + app, + db, + add_battery_assets, + requesting_user, +): + """An automation can opt out of monitoring on creation, and opt back in afterwards.""" + from flexmeasures.data.models.audit_log import AssetAuditLog + + battery = add_battery_assets["Test battery"] + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={ + "name": "Experimental schedules", + "cron": "0 0 * * *", + "type": "scheduling", + "parameters": {"duration": "PT12H"}, + "monitored": False, + }, + ) + assert response.status_code == 201, response.json + assert response.json["monitored"] is False + automation = db.session.get(Automation, response.json["id"]) + assert automation.monitored is False + + with app.test_client() as client: + response = client.patch( + url_for( + "AssetAPI:patch_automation", + id=battery.id, + automation_id=automation.id, + ), + json={"monitored": True}, + ) + assert response.status_code == 200, response.json + assert response.json["monitored"] is True + assert automation.monitored is True + assert db.session.scalars( + select(AssetAuditLog.event).where( + AssetAuditLog.affected_asset_id == battery.id, + AssetAuditLog.event.contains("monitoring switched on"), + ) + ).all() + + +@pytest.mark.parametrize( + "requesting_user, may_change_monitoring", + [ + ("test_admin_user@seita.nl", True), + ( + "test_prosumer_user_2@seita.nl", + True, + ), # account-admin of the asset's organisation + ( + "test_prosumer_user@seita.nl", + False, + ), # member, who may manage the automation otherwise + ( + "test_consultant@seita.nl", + False, + ), # once the asset's organisation is a client of the consultancy + ], + indirect=["requesting_user"], +) +def test_only_admins_may_switch_monitoring( + app, + db, + add_battery_assets, + requesting_user, + may_change_monitoring, +): + """Only the host's admins and the asset's account-admins may create an unmonitored automation, or switch monitoring on or off.""" + battery = add_battery_assets["Test battery"] + owner = battery.owner + original_consultancy_id = owner.consultancy_account_id + if requesting_user.has_role("consultant"): + # Make the consultant one of the organisation's own, so that they may manage its automations. + owner.consultancy_account_id = requesting_user.account_id + db.session.commit() + try: + _check_who_may_switch_monitoring(app, db, battery, may_change_monitoring) + finally: + owner.consultancy_account_id = original_consultancy_id + db.session.commit() + + +def _check_who_may_switch_monitoring(app, db, battery, may_change_monitoring: bool): + automation_data = { + "name": "Experimental schedules", + "cron": "0 0 * * *", + "type": "scheduling", + "parameters": {"duration": "PT12H"}, + } + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={**automation_data, "monitored": False}, + ) + if may_change_monitoring: + assert response.status_code == 201, response.json + else: + assert response.status_code == 403, response.json + assert "Only admins" in response.json["message"] + + # Anyone who may manage the automation may still create a monitored one, and edit it with the monitoring left as it is. + with app.test_client() as client: + response = client.post( + url_for("AssetAPI:post_automation", id=battery.id), + json={**automation_data, "name": "Monitored schedules", "monitored": True}, + ) + assert response.status_code == 201, response.json + automation = db.session.get(Automation, response.json["id"]) + with app.test_client() as client: + response = client.patch( + url_for( + "AssetAPI:patch_automation", + id=battery.id, + automation_id=automation.id, + ), + json={"name": "Renamed schedules", "monitored": True}, + ) + assert response.status_code == 200, response.json + + with app.test_client() as client: + response = client.patch( + url_for( + "AssetAPI:patch_automation", + id=battery.id, + automation_id=automation.id, + ), + json={"monitored": False}, + ) + if may_change_monitoring: + assert response.status_code == 200, response.json + assert automation.monitored is False + else: + assert response.status_code == 403, response.json + db.session.refresh(automation) + assert automation.monitored is True + + @pytest.mark.parametrize( "requesting_user", ["test_prosumer_user_2@seita.nl"], indirect=True ) diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py index fb8afad42f..b56599af35 100755 --- a/flexmeasures/cli/data_add.py +++ b/flexmeasures/cli/data_add.py @@ -1813,6 +1813,12 @@ def add_forecast( # noqa: C901 is_flag=True, help="Add this flag to create the automation in deactivated state.", ) +@click.option( + "--unmonitored", + "unmonitored", + is_flag=True, + help="Add this flag to keep the automation's failed runs out of `flexmeasures monitor automations`.", +) @click.option( "--data-generator", "--forecaster", @@ -1889,6 +1895,7 @@ def add_automation( # noqa: C901 timezone: str, automation_type: str, inactive: bool = False, + unmonitored: bool = False, generator_class: str | None = None, source: DataSource | None = None, config_file: TextIOBase | None = None, @@ -1987,6 +1994,7 @@ def add_automation( # noqa: C901 timezone=timezone, automation_type=automation_type, active=not inactive, + monitored=not unmonitored, parameters=parameters, generator_class=generator_class, config=config, diff --git a/flexmeasures/cli/data_edit.py b/flexmeasures/cli/data_edit.py index 5b33ad400c..568f215362 100644 --- a/flexmeasures/cli/data_edit.py +++ b/flexmeasures/cli/data_edit.py @@ -98,20 +98,28 @@ def fm_edit_data(): default=None, help="Activate or deactivate the automation.", ) +@click.option( + "--monitored/--unmonitored", + "monitored", + default=None, + help="Whether `flexmeasures monitor automations` reports the automation's failed runs.", +) def edit_automation( automation: Automation, name: str | None = None, cronstr: str | None = None, timezone: str | None = None, active: bool | None = None, + monitored: bool | None = None, ): - """Edit the name, recurrence, timezone or activation status of an automation.""" + """Edit the name, recurrence, timezone, activation status or monitoring of an automation.""" changes = update_automation( automation, name=name, cronstr=cronstr, timezone=timezone, active=active, + monitored=monitored, origin="CLI", ) if not changes: diff --git a/flexmeasures/cli/data_show.py b/flexmeasures/cli/data_show.py index cf1fd99add..b545e524ad 100644 --- a/flexmeasures/cli/data_show.py +++ b/flexmeasures/cli/data_show.py @@ -535,6 +535,7 @@ def list_automations( else f"{automation.type} (plugin unavailable)" ), "yes" if automation.active else "no", + "yes" if automation.monitored else "no", automation.cronstr, automation.timezone, ) @@ -546,6 +547,7 @@ def list_automations( "Name", "Type", "Active", + "Monitored", "Cron", "Timezone", ], @@ -567,6 +569,7 @@ def _show_automation(automation: Automation): ("Asset", f"{automation.asset.name} (ID: {automation.asset_id})"), ("Type", automation.type), ("Active", "yes" if automation.active else "no"), + ("Monitored", "yes" if automation.monitored else "no"), ( "Recurrence", f"{automation.cronstr} ({describe_cronstr(automation.cronstr)})", diff --git a/flexmeasures/cli/monitor.py b/flexmeasures/cli/monitor.py index 2c4edebbe0..2efb4cb5be 100644 --- a/flexmeasures/cli/monitor.py +++ b/flexmeasures/cli/monitor.py @@ -21,10 +21,15 @@ from flexmeasures.data import db from flexmeasures.data.models.task_runs import LatestTaskRun +from flexmeasures.data.services.automation_monitoring import ( + AutomationFailures, + get_unalerted_failed_automation_runs, + group_failed_runs_by_automation, +) from flexmeasures.data.models.user import Account, User from flexmeasures.data.schemas.account import AccountIdField from flexmeasures.api.common.schemas.users import UserIdField -from flexmeasures.utils.time_utils import server_now +from flexmeasures.utils.time_utils import exact_duration_str, server_now from flexmeasures.cli.utils import LoggedClickExceptionGroup, MsgStyle @@ -505,4 +510,139 @@ def monitor_last_seen( db.session.commit() +# Error messages can carry a whole traceback, which does not help a digest that lists many automations. +MAX_ERROR_LENGTH_IN_DIGEST = 500 + + +def _format_utc(moment: datetime) -> str: + return moment.astimezone(timezone.utc).strftime("%Y-%m-%d %H:%M UTC") + + +def _times(count: int) -> str: + return "once" if count == 1 else f"{count} times" + + +def automation_page_url(automation) -> str | None: + """Link to the Automations page of the automation's asset, opening that automation's info, if the server knows its own address.""" + base_url = app.config.get("FLEXMEASURES_BASE_URL") + if not base_url: + return None + return f"{base_url.rstrip('/')}/assets/{automation.asset_id}/automations?automation={automation.id}" + + +def describe_automation_failures(failures: AutomationFailures) -> str: + """Describe the failed runs of one automation in a few lines, for the monitoring digest.""" + automation = failures.automation + asset = automation.asset + if asset.owner is None: + where = f"public asset {asset.id} '{asset.name}'" + else: + where = f"asset {asset.id} '{asset.name}' of organisation '{asset.owner.name}'" + reasons = ", ".join( + f"{reason}: {count}" for reason, count in failures.reason_counts.items() + ) + last_run = failures.last_run + # With more than one failure, the first one tells how long the automation has been failing. + first_at = ( + f" first at {_format_utc(failures.first_run.scheduled_at)}," + if len(failures.failed_runs) > 1 + else "" + ) + lines = [ + f"Automation {automation.id} '{automation.name}' ({automation.type}) on {where}:", + f" failed {_times(len(failures.failed_runs))} ({reasons}),{first_at}" + f" last at {_format_utc(last_run.scheduled_at)} (run {last_run.id})", + ] + unmonitored_runs = failures.runs_failed_while_unmonitored + if unmonitored_runs: + lines.append( + f" {len(unmonitored_runs)} of these failed while the automation was not monitored," + f" before monitoring was switched back on at {_format_utc(automation.monitoring_resumed_at)}" + ) + last_error = failures.last_error + if last_error: + if len(last_error) > MAX_ERROR_LENGTH_IN_DIGEST: + last_error = last_error[:MAX_ERROR_LENGTH_IN_DIGEST] + " ..." + lines.append(f" last error: {last_error}") + url = automation_page_url(automation) + if url: + lines.append(f" runs and jobs: {url}") + return "\n".join(lines) + + +@fm_monitor.command("automations") +@with_appcontext +@click.option( + "--stuck-after-minutes", + type=click.IntRange(min=1), + default=None, + help="Report a run as stuck once it has been executing for this many minutes. Defaults to the FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER setting (6 hours).", +) +@click.option( + "--recipient", + type=str, + multiple=True, + help="User ID or email address to send the digest to. Use multiple times if needed. If not used, the default monitoring mail recipients are informed.", +) +def monitor_automations( + stuck_after_minutes: int | None = None, + recipient: tuple[str, ...] = (), +): + """ + Email a digest of the automation runs that failed since the last check. + + A run is reported when its execution failed, when it has been executing for too long (see --stuck-after-minutes), + or when its dispatch failed and will not be attempted again. + Each run is reported once, so a monitoring run that was missed is made up for by the next one. + Repeated failures are collapsed per automation, and no email is sent when nothing failed. + + Meant to be run regularly, for instance hourly from cron. + """ + email_recipients = get_monitoring_email_recipients(recipient) + if not email_recipients: + # Marking runs as reported without telling anyone would hide them from the next monitoring run. + raise click.UsageError( + "Nobody to send the digest to. Set FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS or use --recipient." + ) + if stuck_after_minutes is not None: + stuck_after = timedelta(minutes=stuck_after_minutes) + else: + stuck_after = app.config["FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER"] + + now = server_now() + failed_runs = get_unalerted_failed_automation_runs(stuck_after, now=now) + if not failed_runs: + click.secho( + "All good ― no automation runs failed since the last check.", + **MsgStyle.SUCCESS, + ) + return + failures = group_failed_runs_by_automation(failed_runs) + + subject = f"Failed runs of {len(failures)} automation(s)" + body = ( + f"These automations have runs that failed, or that have been executing for longer than {exact_duration_str(stuck_after)}, which were not reported before:\n\n" + + "\n\n".join(describe_automation_failures(f) for f in failures) + + ( + "\n\nWe suggest to check the logs." + if app.config.get("FLEXMEASURES_BASE_URL") + else "\n\nEach automation's runs are listed on the Automations page of its asset.\nWe suggest to check the logs." + ) + ) + capture_message_for_sentry(subject) + email = Message(subject=subject, bcc=email_recipients) + email.body = body + app.mail.send(email) + app.logger.error(body) + + # Only once the digest is out, so that a failure to send it leaves the runs to the next monitoring run. + for failed_run in failed_runs: + failed_run.run.alerted_at = now + db.session.commit() + click.secho( + f"Reported {len(failed_runs)} failed run(s) of {len(failures)} automation(s) to {', '.join(email_recipients)}.", + **MsgStyle.WARN, + ) + + app.cli.add_command(fm_monitor) diff --git a/flexmeasures/cli/tests/test_automations.py b/flexmeasures/cli/tests/test_automations.py index 5f0d44255a..492e56212a 100644 --- a/flexmeasures/cli/tests/test_automations.py +++ b/flexmeasures/cli/tests/test_automations.py @@ -1869,6 +1869,58 @@ def test_report_automation_refuses_a_sensor_nobody_checked( assert not app.redis_connection.get(f"automation-last-run:{automation.id}") +def test_a_failed_report_run_is_reported_by_monitoring( + app, + fresh_db, + setup_dummy_data_fresh_db, + clean_redis, + tmp_path, + freeze_server_now, + mocker, +): + """A report run whose job failed on the worker shows up in the host's monitoring digest, with the job's error.""" + from flexmeasures.cli.data_add import add_automation + from flexmeasures.cli.jobs import run_automations + from flexmeasures.cli.monitor import monitor_automations + from flexmeasures.data.models.reporting.pandas_reporter import PandasReporter + from flexmeasures.utils.job_utils import work_on_rq + + sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data_fresh_db + report_sensor = fresh_db.session.get(Sensor, report_sensor_id) + cli_input = _report_automation_cli_input( + tmp_path, + sensor1_id, + sensor2_id, + report_sensor_id, + parameters_extra={"start-offset": "DB"}, + asset_id=report_sensor.generic_asset_id, + ) + cli_input[cli_input.index("0 1 * * *")] = "* * * * *" # due every minute + cli_input += ["--timezone", "UTC"] + freeze_server_now(datetime(2023, 4, 10, 10, 0, 30, tzinfo=timezone.utc)) + runner = app.test_cli_runner() + result = runner.invoke(add_automation, cli_input) + assert "Successfully created" in result.output, result.output + automation = fresh_db.session.execute(select(Automation)).scalar_one() + result = runner.invoke(run_automations) + assert "queued 1 reporting job(s)" in result.output, result.output + + mocker.patch.object( + PandasReporter, "compute", side_effect=KeyError("df_agg is missing") + ) + work_on_rq(app.queues["reporting"]) + + with app.mail.record_messages() as outbox: + result = runner.invoke(monitor_automations, ["--recipient", "ops@example.test"]) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + body = outbox[0].body + assert f"Automation {automation.id} '{automation.name}' (reporting)" in body + assert "failed once (execution failed: 1)" in body + assert "last error: KeyError: 'df_agg is missing'" in body + + def _two_output_report_automation_cli_input( tmp_path, sensor1_id, @@ -2716,7 +2768,7 @@ def automation_row_cell(output: str, name: str, header: str) -> str: Substring checks on a whole row are not enough here, as a cell's value may also occur in the automation's own name. """ - headers = ["ID", "Asset", "Name", "Type", "Active", "Cron", "Timezone"] + headers = ["ID", "Asset", "Name", "Type", "Active", "Monitored", "Cron", "Timezone"] lines = output.splitlines() header_line = next(line for line in lines if all(h in line for h in headers)) starts = [header_line.index(h) for h in headers] @@ -2860,6 +2912,50 @@ def test_show_automations_includes_inactive_ones(app, setup_dummy_data_fresh_db) assert automation_row_cell(result.output, "Paused forecasts", "Active") == "no" +def test_add_edit_and_show_an_unmonitored_automation( + app, fresh_db, setup_dummy_data_fresh_db +): + """An automation can be created without monitoring and edited to be monitored, and both views say which it is.""" + from flexmeasures.cli.data_add import add_automation + from flexmeasures.cli.data_edit import edit_automation + from flexmeasures.cli.data_show import list_automations + + runner = app.test_cli_runner() + result = runner.invoke( + add_automation, + to_flags( + { + "asset": 1, + "name": "Experimental forecasts", + "sensor": setup_dummy_data_fresh_db[0], + } + ) + + ["--unmonitored"], + ) + assert "Successfully created" in result.output, result.output + automation = fresh_db.session.scalars(select(Automation)).one() + assert automation.monitored is False + + result = runner.invoke(list_automations, []) + assert result.exit_code == 0, result.output + assert ( + automation_row_cell(result.output, "Experimental forecasts", "Monitored") + == "no" + ) + result = runner.invoke(list_automations, ["--id", automation.id]) + assert result.exit_code == 0, result.output + assert re.search(r"^Monitored\s+no$", result.output, re.MULTILINE) + result = runner.invoke(list_automations, ["--id", automation.id, "--as-json"]) + assert result.exit_code == 0, result.output + assert json.loads(result.output)["monitored"] is False + + result = runner.invoke(edit_automation, ["--id", automation.id, "--monitored"]) + assert result.exit_code == 0, result.output + assert "monitoring switched on" in result.output + fresh_db.session.refresh(automation) + assert automation.monitored is True + + def test_show_automations_without_any_automations(app, setup_dummy_data_fresh_db): """Without automations, the command says so rather than printing an empty table.""" from flexmeasures.cli.data_show import list_automations diff --git a/flexmeasures/cli/tests/test_monitor.py b/flexmeasures/cli/tests/test_monitor.py index caa18fdd6f..ebcfc83da0 100644 --- a/flexmeasures/cli/tests/test_monitor.py +++ b/flexmeasures/cli/tests/test_monitor.py @@ -10,6 +10,7 @@ def test_monitor_help(app): from flexmeasures.cli.monitor import ( fm_monitor, + monitor_automations, monitor_last_seen, monitor_latest_run, ) @@ -19,7 +20,7 @@ def test_monitor_help(app): result = runner.invoke(fm_monitor, ["--help"]) assert result.exit_code == 0 - for command in (monitor_last_seen, monitor_latest_run): + for command in (monitor_last_seen, monitor_latest_run, monitor_automations): result = runner.invoke(command, ["--help"]) assert result.exit_code == 0 diff --git a/flexmeasures/cli/tests/test_monitor_automations_fresh_db.py b/flexmeasures/cli/tests/test_monitor_automations_fresh_db.py new file mode 100644 index 0000000000..bcc5953569 --- /dev/null +++ b/flexmeasures/cli/tests/test_monitor_automations_fresh_db.py @@ -0,0 +1,445 @@ +"""Tests for `flexmeasures monitor automations`, which emails a digest of failed automation runs.""" + +from __future__ import annotations + +from datetime import datetime, timedelta, timezone + +import pytest +from sqlalchemy import select + +from flexmeasures.cli.tests.utils import to_flags +from flexmeasures.data.models.automations import Automation, AutomationRun + +RECIPIENT = "ops@example.test" +DUE_AT = datetime(2026, 8, 5, 1, 0, tzinfo=timezone.utc) + + +@pytest.fixture(scope="function") +def clean_redis(app): + app.redis_connection.flushdb() + yield + app.redis_connection.flushdb() + + +@pytest.fixture() +def forecast_automation( + app, fresh_db, clean_redis, setup_fresh_test_forecast_data, freeze_server_now +): + """Create a forecast automation which is due at the frozen minute.""" + from flexmeasures.cli.data_add import add_automation + + freeze_server_now(DUE_AT - timedelta(minutes=2)) + sensor = setup_fresh_test_forecast_data["solar-sensor"] + result = app.test_cli_runner().invoke( + add_automation, + to_flags( + { + "asset": sensor.generic_asset_id, + "name": "Monitored forecasts", + "cron": "0 1 * * *", + "timezone": "UTC", + "sensor": sensor.id, + "duration": "PT2H", + "forecast-frequency": "PT1H", + "max-forecast-horizon": "PT2H", + "retrain-frequency": "PT1H", + } + ), + ) + assert result.exit_code == 0, result.output + freeze_server_now(DUE_AT) + return fresh_db.session.scalars(select(Automation)).one() + + +def run_automations(app, exit_code: int = 0): + from flexmeasures.cli.jobs import run_automations as run_automations_command + + result = app.test_cli_runner().invoke(run_automations_command) + assert result.exit_code == exit_code, result.output + + +def monitor(app, *args: str): + """Run the monitor, returning its result and the emails it sent.""" + from flexmeasures.cli.monitor import monitor_automations + + with app.mail.record_messages() as outbox: + result = app.test_cli_runner().invoke( + monitor_automations, ["--recipient", RECIPIENT, *args] + ) + return result, outbox + + +def perform_job(app, intent): + from rq import SimpleWorker + + queue = app.queues["forecasting"] + worker = SimpleWorker([queue], connection=queue.connection) + worker.perform_job(queue.fetch_job(intent.rq_job_id), queue) + + +def add_run( + db, automation: Automation, scheduled_at: datetime, **columns +) -> AutomationRun: + """Record a run of the automation in whatever state the test needs, dispatched by default.""" + columns = { + "automation_type": automation.type, + "dispatch_state": "queued", + "dispatch_completed_at": scheduled_at, + "execution_state": "pending", + "attempt_count": 1, + **columns, + } + run = AutomationRun( + automation=automation, + scheduled_at=scheduled_at, + schedule_revision=automation.schedule_revision, + generator_id=automation.generator_id, + parameters={}, + plan={}, + **columns, + ) + db.session.add(run) + db.session.commit() + return run + + +def test_a_failed_forecast_is_reported_once(app, fresh_db, forecast_automation): + """A forecast run whose job failed on the worker is emailed about, and only the first time the monitor runs.""" + run_automations(app) + run = fresh_db.session.scalars(select(AutomationRun)).one() + queue = app.queues["forecasting"] + wrap_up = next(i for i in run.job_intents if i.kind == "forecast-wrap-up") + cycle = next(i for i in run.job_intents if i.kind == "forecast-cycle") + # Make the wrap-up job fail by taking away one of the cycle jobs it reports on. + queue.fetch_job(cycle.rq_job_id).delete() + perform_job(app, wrap_up) + fresh_db.session.refresh(run) + assert run.execution_state == "failed" + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + assert outbox[0].bcc == [RECIPIENT] + assert outbox[0].subject == "Failed runs of 1 automation(s)" + body = outbox[0].body + assert ( + f"Automation {forecast_automation.id} 'Monitored forecasts' (forecasting)" + in body + ) + assert "failed once (execution failed: 1), last at 2026-08-05 01:00 UTC" in body + assert f"last error: {run.last_error_type}" in body + fresh_db.session.refresh(run) + assert run.alerted_at == DUE_AT + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert outbox == [] + assert "All good" in result.output + + +def test_a_running_forecast_is_reported_once_it_is_stuck( + app, fresh_db, forecast_automation, freeze_server_now +): + """A forecast run which started executing is reported once it has been at it for longer than the configured threshold.""" + run_automations(app) + run = fresh_db.session.scalars(select(AutomationRun)).one() + # The wrap-up job succeeds while the cycle jobs have not run, so the run keeps executing. + perform_job(app, next(i for i in run.job_intents if i.kind == "forecast-wrap-up")) + fresh_db.session.refresh(run) + assert run.execution_state == "running" + + freeze_server_now(DUE_AT + timedelta(hours=5, minutes=59)) + result, outbox = monitor(app) + assert result.exit_code == 0, result.output + assert outbox == [] + + freeze_server_now(DUE_AT + timedelta(hours=6)) + result, outbox = monitor(app) + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + assert "failed once (stuck: 1)" in outbox[0].body + assert "last error" not in outbox[0].body + + +def test_the_stuck_threshold_comes_from_the_config_or_the_command_line( + app, fresh_db, forecast_automation, freeze_server_now, monkeypatch +): + """The command-line option overrides the configured threshold, which overrides the default of six hours.""" + stuck_since = DUE_AT - timedelta(hours=2) + add_run( + fresh_db, + forecast_automation, + stuck_since, + execution_state="running", + execution_started_at=stuck_since, + ) + + result, outbox = monitor(app, "--stuck-after-minutes", "180") + assert result.exit_code == 0, result.output + assert outbox == [] + + monkeypatch.setitem( + app.config, "FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER", timedelta(hours=1) + ) + result, outbox = monitor(app, "--stuck-after-minutes", "180") + assert result.exit_code == 0, result.output + assert outbox == [] + + result, outbox = monitor(app) + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + assert "longer than 1 hour" in outbox[0].body + + +def test_a_forecast_dispatch_is_reported_once_its_attempts_are_used_up( + app, fresh_db, forecast_automation, freeze_server_now, mocker +): + """A forecast run whose dispatch failed is retried, so it is only reported once it will not be tried again.""" + from flexmeasures.data.services.automations import ( + AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS, + ) + + mocker.patch.object( + app.queues["forecasting"], + "enqueue_job", + side_effect=RuntimeError("redis unavailable"), + ) + run_automations(app, exit_code=1) + run = fresh_db.session.scalars(select(AutomationRun)).one() + assert run.dispatch_state == "failed" + + result, outbox = monitor(app) + assert result.exit_code == 0, result.output + assert outbox == [] + + # Each attempt waits longer than the last, so stepping an hour ahead each time takes them all. + for attempt_no in range(2, AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS + 1): + freeze_server_now(DUE_AT + timedelta(hours=attempt_no)) + result, outbox = monitor(app) + assert outbox == [], f"reported before attempt {attempt_no}" + run_automations(app, exit_code=1) + fresh_db.session.refresh(run) + assert run.attempt_count == AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS + + result, outbox = monitor(app) + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + assert "failed once (dispatch failed: 1)" in outbox[0].body + assert "last error: RuntimeError: redis unavailable" in outbox[0].body + + +def test_which_runs_count_as_failed(fresh_db, forecast_automation): + """Failures are judged the same way for every automation type, except that only a forecast run is dispatched again.""" + from flexmeasures.data.services.automation_monitoring import ( + get_unalerted_failed_automation_runs, + ) + + def add(minute: int, **columns) -> AutomationRun: + return add_run( + fresh_db, forecast_automation, DUE_AT - timedelta(minutes=minute), **columns + ) + + expected = { + add(1, execution_state="failed").id: "execution failed", + add( + 2, + execution_state="running", + execution_started_at=DUE_AT - timedelta(hours=7), + ).id: "stuck", + # A schedule run is never dispatched again, so its first failed attempt is its last. + add( + 3, + automation_type="scheduling", + dispatch_state="failed", + dispatch_completed_at=None, + claim_expires_at=DUE_AT + timedelta(minutes=1), + ).id: "dispatch failed", + # A runner which died mid-dispatch leaves its claim behind, until its lease runs out. + add( + 4, + automation_type="reporting", + dispatch_state="claimed", + dispatch_completed_at=None, + claim_owner="runner:dead", + claim_expires_at=DUE_AT - timedelta(minutes=1), + ).id: "dispatch failed", + } + not_failed = [ + add(5, execution_state="succeeded"), + add( + 6, + execution_state="running", + execution_started_at=DUE_AT - timedelta(hours=5), + ), + # A forecast run with attempts to spare will be dispatched again. + add(7, dispatch_state="failed", dispatch_completed_at=None), + # A runner is still dispatching this one. + add( + 8, + automation_type="scheduling", + dispatch_state="claimed", + dispatch_completed_at=None, + claim_owner="runner:alive", + claim_expires_at=DUE_AT + timedelta(minutes=5), + ), + add(9, execution_state="failed", alerted_at=DUE_AT - timedelta(hours=1)), + ] + + failed_runs = get_unalerted_failed_automation_runs(timedelta(hours=6), now=DUE_AT) + + assert {f.run.id: f.reason for f in failed_runs} == expected + assert not {run.id for run in not_failed} & {f.run.id for f in failed_runs} + + +def test_repeated_failures_are_collapsed_per_automation( + app, fresh_db, forecast_automation +): + """An automation which failed several times takes up one entry, naming its latest failure and error.""" + for hours_ago, error in ((3, "first"), (1, "latest"), (2, "middle")): + add_run( + fresh_db, + forecast_automation, + DUE_AT - timedelta(hours=hours_ago), + execution_state="failed", + last_error_type="ValueError", + last_error_message=error, + ) + add_run( + fresh_db, + forecast_automation, + DUE_AT - timedelta(minutes=30), + automation_type="scheduling", + dispatch_state="failed", + dispatch_completed_at=None, + ) + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + body = outbox[0].body + assert body.count(f"Automation {forecast_automation.id} ") == 1 + assert ( + "failed 4 times (execution failed: 3, dispatch failed: 1), first at 2026-08-04 22:00 UTC, last at 2026-08-05 00:30 UTC" + in body + ) + # The dispatch failure recorded no error, so the latest error is that of the latest failed execution. + assert "last error: ValueError: latest" in body + assert all( + run.alerted_at is not None + for run in fresh_db.session.scalars(select(AutomationRun)) + ) + + +def test_without_recipients_nothing_is_marked_as_reported( + app, fresh_db, forecast_automation, monkeypatch +): + """Without anyone to email, the monitor refuses to run, rather than marking runs as reported that nobody heard of.""" + from flexmeasures.cli.monitor import monitor_automations + + monkeypatch.setitem( + app.config, "FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS", [] + ) + monkeypatch.setitem(app.config, "FLEXMEASURES_MONITORING_MAIL_RECIPIENTS", []) + run = add_run(fresh_db, forecast_automation, DUE_AT, execution_state="failed") + + result = app.test_cli_runner().invoke(monitor_automations) + + assert result.exit_code == 2 + assert "Nobody to send the digest to" in result.output + fresh_db.session.refresh(run) + assert run.alerted_at is None + + +def test_an_unmonitored_automation_is_reported_once_it_is_monitored_again( + app, fresh_db, forecast_automation +): + """The failed runs of an automation that opted out of monitoring are left out, until it opts back in.""" + forecast_automation.monitored = False + fresh_db.session.commit() + run = add_run(fresh_db, forecast_automation, DUE_AT, execution_state="failed") + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert outbox == [] + fresh_db.session.refresh(run) + assert run.alerted_at is None + + forecast_automation.monitored = True + fresh_db.session.commit() + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + assert f"Automation {forecast_automation.id} " in outbox[0].body + fresh_db.session.refresh(run) + assert run.alerted_at == DUE_AT + + +def test_the_digest_says_which_runs_failed_while_unmonitored( + app, fresh_db, forecast_automation, freeze_server_now +): + """Once monitoring is switched back on, the digest names the first failure and says how many runs failed while it was off.""" + from flexmeasures.data.services.automations import update_automation + + update_automation(forecast_automation, monitored=False, origin="test") + fresh_db.session.commit() + for hours_ago in (72, 48, 24): + add_run( + fresh_db, + forecast_automation, + DUE_AT - timedelta(hours=hours_ago), + execution_state="failed", + ) + switched_on_at = DUE_AT - timedelta(hours=2) + freeze_server_now(switched_on_at) + update_automation(forecast_automation, monitored=True, origin="test") + fresh_db.session.commit() + freeze_server_now(DUE_AT) + # One more failure once it is monitored again. + add_run(fresh_db, forecast_automation, DUE_AT, execution_state="failed") + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + body = outbox[0].body + assert ( + "failed 4 times (execution failed: 4), first at 2026-08-02 01:00 UTC, last at 2026-08-05 01:00 UTC" + in body + ) + assert ( + " 3 of these failed while the automation was not monitored, before monitoring was switched back on at 2026-08-04 23:00 UTC" + in body + ) + + +def test_the_digest_links_to_the_automation_once_the_base_url_is_set( + app, fresh_db, forecast_automation, monkeypatch +): + """With the server's own address configured, each automation in the digest links to its info on the Automations page.""" + add_run(fresh_db, forecast_automation, DUE_AT, execution_state="failed") + link = f"/assets/{forecast_automation.asset_id}/automations?automation={forecast_automation.id}" + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert link not in outbox[0].body + + add_run( + fresh_db, + forecast_automation, + DUE_AT - timedelta(hours=1), + execution_state="failed", + ) + monkeypatch.setitem( + app.config, "FLEXMEASURES_BASE_URL", "https://ems.example.test/" + ) + + result, outbox = monitor(app) + + assert result.exit_code == 0, result.output + assert f" runs and jobs: https://ems.example.test{link}" in outbox[0].body diff --git a/flexmeasures/data/migrations/versions/d4b7e9a1c362_add_alerted_at_to_automation_runs.py b/flexmeasures/data/migrations/versions/d4b7e9a1c362_add_alerted_at_to_automation_runs.py new file mode 100644 index 0000000000..74230d5642 --- /dev/null +++ b/flexmeasures/data/migrations/versions/d4b7e9a1c362_add_alerted_at_to_automation_runs.py @@ -0,0 +1,57 @@ +"""Record when an automation run was reported by `flexmeasures monitor automations` + +Revision ID: d4b7e9a1c362 +Revises: bf22b21d0990 +Create Date: 2026-10-07 10:00:00.000000 + +The monitor reports each failed or stuck run once, and marks it in ``alerted_at``. +The upgrade marks the runs that had already failed for good as reported, so that the first monitoring run does not report every failure on record. +It judges them by the monitor's rules as they stand at this revision, with the default stuck threshold of 6 hours, +and leaves out any run that may still be retried or finish. + +The partial index holds only the runs the monitor has to look at: +those not reported yet which failed, are still running, or never finished their dispatch. +Every other run, which is nearly all of them, stays out of it. +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "d4b7e9a1c362" +down_revision = "bf22b21d0990" +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column( + "automation_run", + sa.Column("alerted_at", sa.DateTime(timezone=True), nullable=True), + ) + op.create_index( + "automation_run_unalerted_idx", + "automation_run", + ["automation_id"], + postgresql_where=sa.text( + "alerted_at IS NULL" + " AND (execution_state = 'failed' OR execution_state = 'running' OR dispatch_completed_at IS NULL)" + ), + ) + # Copied from the monitor's query rather than imported, so this migration keeps doing what it did when the query changes. + op.execute(""" + UPDATE automation_run + SET alerted_at = now() + WHERE execution_state = 'failed' + OR (execution_state = 'running' AND execution_started_at <= now() - interval '6 hours') + OR ( + dispatch_completed_at IS NULL + AND (claim_owner IS NULL OR claim_expires_at IS NULL OR claim_expires_at <= now()) + AND NOT (automation_type = 'forecasting' AND attempt_count < 5) + ) + """) + + +def downgrade(): + op.drop_index("automation_run_unalerted_idx", table_name="automation_run") + op.drop_column("automation_run", "alerted_at") diff --git a/flexmeasures/data/migrations/versions/e8a3f6c2b917_add_monitored_to_automations.py b/flexmeasures/data/migrations/versions/e8a3f6c2b917_add_monitored_to_automations.py new file mode 100644 index 0000000000..ff2f153321 --- /dev/null +++ b/flexmeasures/data/migrations/versions/e8a3f6c2b917_add_monitored_to_automations.py @@ -0,0 +1,38 @@ +"""Let an automation opt out of monitoring + +Revision ID: e8a3f6c2b917 +Revises: d4b7e9a1c362 +Create Date: 2026-10-12 10:00:00.000000 + +`flexmeasures monitor automations` skips the runs of an automation whose ``monitored`` is false. +Every existing automation stays monitored. +``monitoring_resumed_at`` records when monitoring was last switched back on, +so that the digest can tell which of the runs it reports failed while the automation was not monitored. +""" + +from alembic import op +import sqlalchemy as sa + +# revision identifiers, used by Alembic. +revision = "e8a3f6c2b917" +down_revision = "d4b7e9a1c362" +branch_labels = None +depends_on = None + + +def upgrade(): + op.add_column( + "automation", + sa.Column( + "monitored", sa.Boolean(), nullable=False, server_default=sa.text("true") + ), + ) + op.add_column( + "automation", + sa.Column("monitoring_resumed_at", sa.DateTime(timezone=True), nullable=True), + ) + + +def downgrade(): + op.drop_column("automation", "monitoring_resumed_at") + op.drop_column("automation", "monitored") diff --git a/flexmeasures/data/models/automations.py b/flexmeasures/data/models/automations.py index cbba34f811..cc344501a9 100644 --- a/flexmeasures/data/models/automations.py +++ b/flexmeasures/data/models/automations.py @@ -95,6 +95,12 @@ class Automation(db.Model, AuthModelMixin): ) schedule_revision = db.Column(db.Integer, nullable=False, default=1) active = db.Column(db.Boolean, nullable=False, default=True) + # Whether `flexmeasures monitor automations` reports this automation's failed runs. + monitored = db.Column( + db.Boolean, nullable=False, default=True, server_default="true" + ) + # When monitoring was last switched back on, which tells the digest which runs failed while it was off. + monitoring_resumed_at = db.Column(db.DateTime(timezone=True), nullable=True) generator_id = db.Column( db.Integer, db.ForeignKey("data_source.id"), nullable=False ) @@ -245,6 +251,8 @@ class AutomationRun(db.Model): execution_completed_at = db.Column(db.DateTime(timezone=True), nullable=True) last_error_type = db.Column(db.String(160), nullable=True) last_error_message = db.Column(db.Text, nullable=True) + # When `flexmeasures monitor automations` reported this run as failed or stuck, so that it is reported once. + alerted_at = db.Column(db.DateTime(timezone=True), nullable=True) parameters = db.Column(MutableDict.as_mutable(JSONB), nullable=False, default=dict) plan = db.Column(MutableDict.as_mutable(JSONB), nullable=False, default=dict) @@ -274,6 +282,7 @@ class AutomationRun(db.Model): "dispatch_completed_at", "execution_started_at", "execution_completed_at", + "alerted_at", ) def validate_datetime_is_aware( self, key: str, value: datetime | None diff --git a/flexmeasures/data/schemas/automations.py b/flexmeasures/data/schemas/automations.py index b272d83a80..69a0a781f6 100644 --- a/flexmeasures/data/schemas/automations.py +++ b/flexmeasures/data/schemas/automations.py @@ -92,6 +92,13 @@ class AutomationCreationSchema(Schema): }, ) active = fields.Bool(load_default=True) + monitored = fields.Bool( + load_default=True, + metadata={ + "description": "Whether `flexmeasures monitor automations` reports this automation's failed runs. Runs that failed while it was not monitored are reported once it is monitored again. Only the host's admins and the account-admins of the asset's organisation may create an unmonitored automation.", + "example": True, + }, + ) parameters = fields.Dict(keys=fields.Str(), load_default=dict) generator_class = fields.Str( data_key="data-generator", @@ -151,7 +158,7 @@ def validate_generator_is_named_once(self, data: dict, **kwargs): class AutomationUpdateSchema(Schema): - """Request schema for updating an automation's name, recurrence, timezone and/or activation status. + """Request schema for updating an automation's name, recurrence, timezone, activation status and/or monitoring. The parameters cannot be updated, so the sensors an automation involves stay the ones its creator was checked against. """ @@ -165,6 +172,12 @@ class AutomationUpdateSchema(Schema): } ) active = fields.Bool() + monitored = fields.Bool( + metadata={ + "description": "Whether `flexmeasures monitor automations` reports this automation's failed runs. Runs that failed while it was not monitored are reported once it is monitored again. Only the host's admins and the account-admins of the asset's organisation may change it.", + "example": True, + }, + ) class AutomationSchema(ma.SQLAlchemySchema): @@ -211,6 +224,12 @@ class Meta: }, ) active = ma.auto_field() + monitored = ma.auto_field( + metadata={ + "description": "Whether `flexmeasures monitor automations` reports this automation's failed runs. Runs that failed while it was not monitored are reported once it is monitored again.", + "example": True, + }, + ) @staticmethod def _in_automation_timezone( diff --git a/flexmeasures/data/services/automation_copies.py b/flexmeasures/data/services/automation_copies.py index 2d0eba027a..6a9cc63cd8 100644 --- a/flexmeasures/data/services/automation_copies.py +++ b/flexmeasures/data/services/automation_copies.py @@ -197,6 +197,7 @@ def _copy_automation( timezone=automation.timezone, # A copy starts inactive, so that it can be inspected and tested before it runs. active=False, + monitored=automation.monitored, generator_id=generator.id, parameters=parameters, # A fresh cursor, so that the copy does not inherit the original's run history. diff --git a/flexmeasures/data/services/automation_monitoring.py b/flexmeasures/data/services/automation_monitoring.py new file mode 100644 index 0000000000..10d06b7811 --- /dev/null +++ b/flexmeasures/data/services/automation_monitoring.py @@ -0,0 +1,176 @@ +"""Find the automation runs that failed, so that monitoring can report them.""" + +from __future__ import annotations + +from collections import Counter +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone + +from sqlalchemy import and_, not_, or_, select +from sqlalchemy.orm import selectinload + +from flexmeasures.data import db +from flexmeasures.data.models.automations import Automation, AutomationRun +from flexmeasures.data.models.generic_assets import GenericAsset +from flexmeasures.data.services.automations import AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS +from flexmeasures.utils.time_utils import server_now + +EXECUTION_FAILED = "execution failed" +STUCK = "stuck" +DISPATCH_FAILED = "dispatch failed" + + +@dataclass(frozen=True) +class FailedAutomationRun: + """An automation run that failed, with the way it failed.""" + + run: AutomationRun + reason: str + + +@dataclass +class AutomationFailures: + """The failed runs of one automation, collapsed into what a digest says about them.""" + + automation: Automation + failed_runs: list[FailedAutomationRun] = field(default_factory=list) + + @property + def reason_counts(self) -> Counter: + """How many runs failed in each way.""" + return Counter(failed_run.reason for failed_run in self.failed_runs) + + @property + def first_run(self) -> AutomationRun: + """The failed run that was due first.""" + return min( + (failed_run.run for failed_run in self.failed_runs), + key=lambda run: (run.scheduled_at, run.id), + ) + + @property + def runs_failed_while_unmonitored(self) -> list[AutomationRun]: + """The failed runs that were due before monitoring was last switched back on. + + Had the automation been monitored then, an earlier digest would have reported them. + """ + resumed_at = self.automation.monitoring_resumed_at + if resumed_at is None: + return [] + return [ + failed_run.run + for failed_run in self.failed_runs + if failed_run.run.scheduled_at < resumed_at + ] + + @property + def last_run(self) -> AutomationRun: + """The failed run that was due most recently.""" + return max( + (failed_run.run for failed_run in self.failed_runs), + key=lambda run: (run.scheduled_at, run.id), + ) + + @property + def last_error(self) -> str | None: + """The error of the most recent failed run that recorded one, if any did.""" + runs_with_an_error = [ + failed_run.run + for failed_run in self.failed_runs + if failed_run.run.last_error_type or failed_run.run.last_error_message + ] + if not runs_with_an_error: + return None + run = max(runs_with_an_error, key=lambda run: (run.scheduled_at, run.id)) + return ": ".join( + part for part in (run.last_error_type, run.last_error_message) if part + ) + + +def _dispatch_failed_for_good(now: datetime): + """Return the criterion for a run whose dispatch did not finish and will not be attempted again. + + No runner may hold a live claim on it, or it is still being dispatched. + Only a forecast run is dispatched again, until its attempts are used up (see `get_dispatchable_automation_runs`), + so a run of any other type has failed for good after its first attempt. + """ + return and_( + AutomationRun.dispatch_completed_at.is_(None), + or_( + AutomationRun.claim_owner.is_(None), + AutomationRun.claim_expires_at.is_(None), + AutomationRun.claim_expires_at <= now, + ), + not_( + and_( + AutomationRun.automation_type == "forecasting", + AutomationRun.attempt_count < AUTOMATION_RUN_MAX_DISPATCH_ATTEMPTS, + ) + ), + ) + + +def get_unalerted_failed_automation_runs( + stuck_after: timedelta, + now: datetime | None = None, +) -> list[FailedAutomationRun]: + """Return the automation runs that failed and were not reported yet, oldest first. + + A run failed when its execution failed, when its execution has been running for longer than ``stuck_after``, + or when its dispatch failed and will not be attempted again. + This holds for every automation type, although only the types that record their execution can fail on it. + Runs of an automation that opted out of monitoring are left out, and stay unreported until it opts back in. + + :param stuck_after: How long a run may execute before it counts as stuck. + :param now: The moment to judge by, which defaults to the server's current time. + """ + if now is None: + now = server_now() + now = now.astimezone(timezone.utc) + execution_failed = AutomationRun.execution_state == "failed" + stuck = and_( + AutomationRun.execution_state == "running", + AutomationRun.execution_started_at <= now - stuck_after, + ) + runs = db.session.scalars( + select(AutomationRun) + .join(AutomationRun.automation) + .where( + Automation.monitored.is_(True), + AutomationRun.alerted_at.is_(None), + or_(execution_failed, stuck, _dispatch_failed_for_good(now)), + ) + .options( + selectinload(AutomationRun.automation) + .joinedload(Automation.asset) + .joinedload(GenericAsset.owner) + ) + .order_by( + AutomationRun.automation_id, AutomationRun.scheduled_at, AutomationRun.id + ) + ).all() + failed_runs = [] + for run in runs: + if run.execution_state == "failed": + reason = EXECUTION_FAILED + elif ( + run.execution_state == "running" + and run.execution_started_at <= now - stuck_after + ): + reason = STUCK + else: + reason = DISPATCH_FAILED + failed_runs.append(FailedAutomationRun(run=run, reason=reason)) + return failed_runs + + +def group_failed_runs_by_automation( + failed_runs: list[FailedAutomationRun], +) -> list[AutomationFailures]: + """Collapse failed runs per automation, keeping the order in which the automations first appear.""" + failures: dict[int, AutomationFailures] = {} + for failed_run in failed_runs: + automation = failed_run.run.automation + failures.setdefault(automation.id, AutomationFailures(automation=automation)) + failures[automation.id].failed_runs.append(failed_run) + return list(failures.values()) diff --git a/flexmeasures/data/services/automations.py b/flexmeasures/data/services/automations.py index 5ef282abf5..e02b1db58f 100644 --- a/flexmeasures/data/services/automations.py +++ b/flexmeasures/data/services/automations.py @@ -2024,6 +2024,7 @@ def create_automation( source: DataSource | None = None, origin: str = "API", check_permissions: bool = False, + monitored: bool = True, ) -> tuple[Automation, list[str]]: """Create an automation through its registered handler, without committing.""" from flask_security import current_user @@ -2047,6 +2048,8 @@ def create_automation( origin=origin, check_permissions=check_permissions, ) + # Set here rather than by the handler, so that no handler, a plugin's included, needs to know about monitoring. + automation.monitored = monitored if check_permissions: if current_user.is_anonymous: from werkzeug.exceptions import Unauthorized @@ -2235,15 +2238,34 @@ def _create_builtin_automation( return automation, warnings +def user_may_change_monitoring(user, asset) -> bool: + """Tell whether the user may switch the monitoring of the asset's automations on or off. + + Switching it off silences the digest for whoever receives it, so it is up to admins: + the host's admins, and the account-admins of the organisation owning the asset. + Members and consultants may not, although they may manage the automations otherwise. + """ + from flexmeasures.auth.policy import ACCOUNT_ADMIN_ROLE, user_has_admin_access + + if user_has_admin_access(user, "update"): + return True + return ( + asset.account_id is not None + and user.account_id == asset.account_id + and user.has_role(ACCOUNT_ADMIN_ROLE) + ) + + def update_automation( automation: Automation, name: str | None = None, cronstr: str | None = None, timezone: str | None = None, active: bool | None = None, + monitored: bool | None = None, origin: str = "API", ) -> list[str]: - """Update an automation's name, cron string, timezone and/or activation status (not committed yet). + """Update an automation's name, cron string, timezone, activation status and/or monitoring (not committed yet). Anything that changes which runs are due, namely the recurrence, the timezone and reactivation, also resets the cursor to just before the minute of the change, so that runs from before it are not caught up on, @@ -2272,6 +2294,13 @@ def update_automation( if active: reset_cursor = True automation.active = active + if monitored is not None and monitored != automation.monitored: + changes.append( + "monitoring switched on" if monitored else "monitoring switched off" + ) + automation.monitored = monitored + if monitored: + automation.monitoring_resumed_at = server_now() if reset_cursor: # A new revision keeps the durable runs of the old and the new schedule apart, # even where they fall on the same scheduled UTC time. diff --git a/flexmeasures/data/tests/test_automations_fresh_db.py b/flexmeasures/data/tests/test_automations_fresh_db.py index ac36ef3ad4..ef61434dde 100644 --- a/flexmeasures/data/tests/test_automations_fresh_db.py +++ b/flexmeasures/data/tests/test_automations_fresh_db.py @@ -333,6 +333,50 @@ def refuse_to_save(*args, **kwargs): assert run.execution_completed_at is not None +def test_a_failed_schedule_run_is_reported_by_monitoring( + fresh_db, + app, + add_battery_assets_fresh_db, + add_market_prices_fresh_db, + clean_scheduling_redis, + monkeypatch, +): + """A schedule run whose job failed on the worker shows up in the host's monitoring digest, with the job's error.""" + from flexmeasures.cli.monitor import monitor_automations + from flexmeasures.data.services import scheduling + from flexmeasures.data.services.automations import ( + mark_automation_run_dispatch_queued, + ) + from flexmeasures.utils.job_utils import work_on_rq + + automation, run = _planned_schedule_run( + fresh_db, add_battery_assets_fresh_db["Test battery"], sequential=False + ) + run_automation(automation, automation_run=run) + # Its dispatch went through, as a runner would record, so only its execution can fail. + mark_automation_run_dispatch_queued(run.id) + + def refuse_to_save(*args, **kwargs): + raise ValueError("The database is read-only today.") + + monkeypatch.setattr(scheduling, "save_generator_results", refuse_to_save) + work_on_rq( + app.queues["scheduling"], exc_handler=scheduling.handle_scheduling_exception + ) + + with app.mail.record_messages() as outbox: + result = app.test_cli_runner().invoke( + monitor_automations, ["--recipient", "ops@example.test"] + ) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + body = outbox[0].body + assert f"Automation {automation.id} 'Nightly schedules' (scheduling)" in body + assert "failed once (execution failed: 1)" in body + assert "last error: ValueError: The database is read-only today." in body + + def test_a_schedule_job_queued_outside_a_run_is_not_recorded( fresh_db, app, diff --git a/flexmeasures/data/tests/test_plugin_automations_fresh_db.py b/flexmeasures/data/tests/test_plugin_automations_fresh_db.py index 14f546c647..478f8ee0c9 100644 --- a/flexmeasures/data/tests/test_plugin_automations_fresh_db.py +++ b/flexmeasures/data/tests/test_plugin_automations_fresh_db.py @@ -535,6 +535,44 @@ def test_a_plugin_run_records_how_its_job_ended( assert run.execution_state == "succeeded" +def test_a_failed_plugin_run_is_reported_by_monitoring( + fresh_db, app, ingestion_plugin, ingestion_assets, monkeypatch +): + """A plugin run whose job failed on the worker shows up in the host's monitoring digest, with the job's error.""" + from flexmeasures.cli.monitor import monitor_automations + from flexmeasures.data.services.automations import ( + mark_automation_run_dispatch_queued, + ) + + root, sensors = ingestion_assets + automation = make_automation(fresh_db, root, sensors[0], parameters={"value": 3}) + run = _claimed_run(fresh_db, automation) + + def fail_to_ingest(generator, value): + raise ConnectionError("The meter portal is down.") + + monkeypatch.setattr(MockIngestionGenerator, "_compute", fail_to_ingest) + result = run_automation(automation, automation_run=run) + # Its dispatch went through, as a runner would record, so only its execution can fail. + mark_automation_run_dispatch_queued(run.id) + work_on_rq( + app.queues["ingestion"], + job=Job.fetch(result["job_id"], connection=app.redis_connection), + ) + + with app.mail.record_messages() as outbox: + result = app.test_cli_runner().invoke( + monitor_automations, ["--recipient", "ops@example.test"] + ) + + assert result.exit_code == 0, result.output + assert len(outbox) == 1 + body = outbox[0].body + assert f"Automation {automation.id} '{automation.name}' ({automation.type})" in body + assert "failed once (execution failed: 1)" in body + assert "last error: ConnectionError: The meter portal is down." in body + + def test_a_job_that_ended_before_its_run_recorded_it_is_caught_up_on( fresh_db, app, ingestion_plugin, ingestion_assets, monkeypatch ): diff --git a/flexmeasures/ui/static/openapi-specs.json b/flexmeasures/ui/static/openapi-specs.json index 4ba933f8e3..ffead0f7a8 100644 --- a/flexmeasures/ui/static/openapi-specs.json +++ b/flexmeasures/ui/static/openapi-specs.json @@ -3585,6 +3585,7 @@ "recurrence-description": "At 06:00", "revision": 1, "active": true, + "monitored": true, "parameters": { "sensor": 2092 }, @@ -3665,8 +3666,8 @@ ] }, "patch": { - "summary": "Update an automation's name, cron string, timezone or activation status.", - "description": "Any subset of the fields `name`, `cronstr`, `timezone` and `active` can be sent.\nChanging the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change,\nso runs from before it are not caught up on, while a run due in that very minute still is.\nOther automation fields cannot be updated; instead, create a new automation.\nRequires permission to add data under the asset.\n", + "summary": "Update an automation's name, cron string, timezone, activation status or monitoring.", + "description": "Any subset of the fields `name`, `cronstr`, `timezone`, `active` and `monitored` can be sent.\nChanging the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change,\nso runs from before it are not caught up on, while a run due in that very minute still is.\nOther automation fields cannot be updated; instead, create a new automation.\nRequires permission to add data under the asset.\n", "security": [ { "ApiKeyAuth": [] @@ -3704,6 +3705,12 @@ "value": { "active": false } + }, + "stop-monitoring": { + "summary": "Stop reporting the automation's failed runs", + "value": { + "monitored": false + } } } } @@ -3737,7 +3744,7 @@ "/api/v3_0/assets/{id}/automations": { "get": { "summary": "Get all automations defined on an asset.", - "description": "The response will be a list of automations: recurring forecasting, scheduling, reporting or plugin-defined tasks\ndefined on the asset. Each entry shows the automation's ID, when it was created,\nits type, name, activation status, and its recurrence, both as a cron string\nand described in natural language. Each entry also shows the IANA timezone in which its cron expression is interpreted,\nand both its cursor and its next scheduled run as clock times in that same timezone (the next run is null while inactive).\nThe next run excludes pending catch-up work.\n\nBy default, the automations of the assets below it are included as well, at any depth, so that a site asset reports everything that runs below it.\nPass `include-child-assets=false` to list only the automations defined on the asset itself.\nOnly the assets below it which you may read are included.\nEach entry names the asset it is defined on, in `asset` and `asset-name`.\n", + "description": "The response will be a list of automations: recurring forecasting, scheduling, reporting or plugin-defined tasks\ndefined on the asset. Each entry shows the automation's ID, when it was created,\nits type, name, activation status, whether it is monitored, and its recurrence, both as a cron string\nand described in natural language. Each entry also shows the IANA timezone in which its cron expression is interpreted,\nand both its cursor and its next scheduled run as clock times in that same timezone (the next run is null while inactive).\nThe next run excludes pending catch-up work.\n\nBy default, the automations of the assets below it are included as well, at any depth, so that a site asset reports everything that runs below it.\nPass `include-child-assets=false` to list only the automations defined on the asset itself.\nOnly the assets below it which you may read are included.\nEach entry names the asset it is defined on, in `asset` and `asset-name`.\n", "security": [ { "ApiKeyAuth": [] @@ -3787,6 +3794,7 @@ "recurrence-description": "At 06:00", "revision": 1, "active": true, + "monitored": true, "job-stats": { "finished": 3 } @@ -6848,6 +6856,12 @@ "type": "boolean", "default": true }, + "monitored": { + "type": "boolean", + "default": true, + "description": "Whether `flexmeasures monitor automations` reports this automation's failed runs. Runs that failed while it was not monitored are reported once it is monitored again. Only the host's admins and the account-admins of the asset's organisation may create an unmonitored automation.", + "example": true + }, "parameters": { "type": "object", "additionalProperties": {} @@ -6900,6 +6914,11 @@ }, "active": { "type": "boolean" + }, + "monitored": { + "type": "boolean", + "description": "Whether `flexmeasures monitor automations` reports this automation's failed runs. Runs that failed while it was not monitored are reported once it is monitored again. Only the host's admins and the account-admins of the asset's organisation may change it.", + "example": true } }, "additionalProperties": false diff --git a/flexmeasures/ui/templates/assets/asset_automations.html b/flexmeasures/ui/templates/assets/asset_automations.html index 1a7c23a74f..28364112e7 100644 --- a/flexmeasures/ui/templates/assets/asset_automations.html +++ b/flexmeasures/ui/templates/assets/asset_automations.html @@ -143,6 +143,15 @@

{% endif %} +
+ + +
+ {% if user_can_manage_automations %}
@@ -243,6 +252,16 @@
Configuration
+
+ + + {% if not user_can_change_monitoring %} +
Only admins can switch monitoring off or on.
+ {% endif %} +
+
+ + + {% if not user_can_change_monitoring %} +
Only admins can switch monitoring off or on.
+ {% endif %} +