Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
6c7798a
data/services/automations: record how schedule, report and plugin run…
BelhsanHmida Oct 7, 2026
196ccc1
data/tests: cover how schedule, report and plugin runs end
BelhsanHmida Oct 7, 2026
da14723
docs: every automation run records how its jobs ended
BelhsanHmida Oct 7, 2026
2441424
changelog: append PR #2670 to the durable automation runs entry
BelhsanHmida Oct 7, 2026
56e5f83
data/models: record when an automation run was reported by monitoring
BelhsanHmida Oct 7, 2026
39ca4a0
utils/config: add FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER
BelhsanHmida Oct 7, 2026
ddfbe7a
data/services: find the automation runs that failed and were not repo…
BelhsanHmida Oct 7, 2026
3cf9ef6
cli: add flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
47e200c
cli/tests: test flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
379a8b8
docs: document flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
4569653
data/models: let an automation opt out of monitoring
BelhsanHmida Oct 12, 2026
57458d1
data/services: leave out the runs of automations that are not monitored
BelhsanHmida Oct 12, 2026
956d413
cli/tests: test that monitoring skips unmonitored automations until t…
BelhsanHmida Oct 12, 2026
c0f2aba
data/services: set whether an automation is monitored when creating, …
BelhsanHmida Oct 12, 2026
8f9f6cb
api: accept and report whether an automation is monitored
BelhsanHmida Oct 12, 2026
6095634
api/tests: test opting an automation out of monitoring and back in, a…
BelhsanHmida Oct 12, 2026
f829681
cli: set and show whether an automation is monitored
BelhsanHmida Oct 12, 2026
3235823
cli/tests: test adding, editing and showing an unmonitored automation
BelhsanHmida Oct 12, 2026
72a5e11
ui: set whether an automation is monitored on the Automations page
BelhsanHmida Oct 12, 2026
39585bd
ui/tests: test that a copy keeps whether its original is monitored
BelhsanHmida Oct 12, 2026
038be34
docs: document opting an automation out of monitoring
BelhsanHmida Oct 12, 2026
21fc37d
Merge origin/main into feat/automation-run-outcomes
BelhsanHmida Oct 12, 2026
185502b
Merge feat/automation-run-outcomes (#2670) into feat/2519-monitor-aut…
BelhsanHmida Oct 12, 2026
f58aaa6
data/migrations: revise main's latest migration with the alerted_at m…
BelhsanHmida Oct 12, 2026
d37502e
tests: report failed schedule, report and plugin runs in the monitori…
BelhsanHmida Oct 12, 2026
3734a4e
docs: drop the note that only forecast runs are monitored for their e…
BelhsanHmida Oct 12, 2026
d910da4
changelog: add PR #2671 to the automations, in detail list
BelhsanHmida Oct 12, 2026
ed81fd7
Merge feat/2519-monitor-automations (#2671) into feat/2519-monitor-au…
BelhsanHmida Oct 12, 2026
00815c4
changelog: append PR #2702 to the monitor automations entry
BelhsanHmida Oct 12, 2026
5f37108
docs/api: run-stats records execution state for every automation type
BelhsanHmida Oct 12, 2026
627c635
Merge origin/main into feat/automation-run-outcomes
BelhsanHmida Oct 14, 2026
94e4a52
Merge main, where #2657 records a schedule run's jobs before queueing…
Flix6x Oct 20, 2026
cc5295a
Record the error of a job that failed before its run recorded it by i…
Flix6x Oct 20, 2026
0c85cd1
Merge feat/automation-run-outcomes (#2670's final tip) into feat/2519…
BelhsanHmida Oct 29, 2026
84d04b0
Merge #2670's squash commit into feat/2519-monitor-automations
BelhsanHmida Oct 29, 2026
576fa2a
Merge origin/main into feat/2519-monitor-automations
BelhsanHmida Oct 29, 2026
e83eb29
data/migrations: revise main's latest migration with the alerted_at m…
BelhsanHmida Oct 29, 2026
b57c036
utils/time_utils: move the exact duration formatting out of the monit…
BelhsanHmida Nov 15, 2026
ff3ec20
data/migrations: count runs that failed before the upgrade as reported
BelhsanHmida Nov 15, 2026
644b604
cli/monitor: link each automation in the digest to its info on the Au…
BelhsanHmida Nov 15, 2026
d27047c
Merge branch 'feat/2519-monitor-automations' into feat/2519-monitor-a…
BelhsanHmida Nov 15, 2026
bc49707
api/v3_0: let only admins switch an automation's monitoring off or on
BelhsanHmida Nov 15, 2026
dbca642
cli/monitor: say which runs failed while the automation was not monit…
BelhsanHmida Nov 15, 2026
67f40a0
ui/tests: check that only admins can switch monitoring in the automat…
BelhsanHmida Nov 15, 2026
54789ab
ui/assets: filter the Automations page on monitoring
BelhsanHmida Nov 15, 2026
90d76ef
documentation: add the monitored field and options to the API and CLI…
BelhsanHmida Nov 15, 2026
8ed0a06
Merge remote-tracking branch 'origin/main' into feat/2519-monitor-aut…
BelhsanHmida Nov 15, 2026
40a58eb
Merge branch 'feat/2519-monitor-automations' into feat/2519-monitor-a…
BelhsanHmida Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/api/change_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ API change log
v3.0-35 | November xx, 2026
""""""""""""""""""""""""""""

- ``POST /api/v3_0/assets/<id>/automations`` and ``PATCH /api/v3_0/assets/<id>/automations/<automation_id>`` now accept ``monitored``, which says whether ``flexmeasures monitor automations`` reports the automation's failed runs (``true`` by default), and the automation endpoints return it. Only the host's admins and the account-admins of the asset's organisation may create an unmonitored automation or switch monitoring off or on; others get a ``403``.
- Renamed ``schedule-revision`` to ``revision`` in automation responses and in each run listed under ``run-stats``. The field counts the resets of an automation's recurrence, not anything about schedules; it was released under the old name in v3.0-34 (FlexMeasures v1.1.0), and is read-only, so only clients reading it need to switch.

v3.0-34 | November 05, 2026
Expand Down
1 change: 1 addition & 0 deletions documentation/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ v1.2.0 | December xx, 2026

New features
-------------
* Hosts can be emailed a digest of the automation runs that failed, got stuck or could not be queued, with ``flexmeasures monitor automations``, run from cron; each run is reported once, repeated failures are collapsed per automation, with the new ``FLEXMEASURES_BASE_URL`` setting each automation links to its *Automations* page, and admins can switch an automation's monitoring off [see `PR #2671 <https://www.github.com/FlexMeasures/flexmeasures/pull/2671>`_ and `PR #2702 <https://www.github.com/FlexMeasures/flexmeasures/pull/2702>`_]

Infrastructure / Support
-------------------------
Expand Down
6 changes: 6 additions & 0 deletions documentation/cli/change_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@
FlexMeasures CLI Changelog
**************************

since v1.2.0 | December xx, 2026
=================================

* Add ``flexmeasures monitor automations`` to email a digest of the automation runs that failed since the last check: runs whose execution failed, which have been executing for too long (``--stuck-after-minutes``), or whose dispatch failed for good. Each run is reported once, and repeated failures are collapsed per automation.
* Add ``flexmeasures add automation --unmonitored`` and ``flexmeasures edit automation --monitored/--unmonitored`` to keep an automation's failed runs out of ``flexmeasures monitor automations``, or to report them again. ``flexmeasures show automations`` shows whether each automation is monitored.

since v1.1.0 | November 05, 2026
=================================

Expand Down
1 change: 1 addition & 0 deletions documentation/cli/commands.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ of which some are referred to in this documentation.
================================================= =======================================
``flexmeasures monitor latest-run`` Check if the given task's last successful execution happened less than the allowed time ago.
``flexmeasures monitor last-seen`` Check if given users last contact (via a request) happened less than the allowed time ago.
``flexmeasures monitor automations`` Email a digest of the automation runs that failed since the last check.
================================================= =======================================


Expand Down
25 changes: 24 additions & 1 deletion documentation/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -878,11 +878,34 @@ Default: ``None``
FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run`` and ``flexmeasures monitor last-seen`` if no explicit user recipients are given. For example ``["[email protected]", "[email protected]"]``.
E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run``, ``flexmeasures monitor last-seen`` and ``flexmeasures monitor automations`` if no explicit user recipients are given. For example ``["[email protected]", "[email protected]"]``.

Default: ``[]``


.. _monitor_automations_stuck_after:

FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

How long an automation run may execute before ``flexmeasures monitor automations`` reports it as stuck, for instance because the worker running its job died.
Its ``--stuck-after-minutes`` option overrides this setting.

Default: ``timedelta(hours=6)``

.. _base_url:

FLEXMEASURES_BASE_URL
^^^^^^^^^^^^^^^^^^^^^

Where this server is reached on the web, for instance ``"https://ems.example.com"``.
Emails sent from the CLI, outside of any request, use it to link to the web pages they are about.
For instance, the digest of ``flexmeasures monitor automations`` links each failing automation to the Automations page of its asset.
Without it, these emails carry no links.

Default: ``None``


.. _redis-config:

Redis
Expand Down
30 changes: 30 additions & 0 deletions documentation/features/automations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,36 @@ This is what an operator needs to tell a run which failed before queueing anythi
Editing an automation's cron string or timezone, or reactivating it, counts up its revision.
Runs of the old and the new schedule therefore stay distinct, even when they fall on the same scheduled UTC time.

.. _monitoring_automations:

Monitoring automation runs
--------------------------

As a host, you can have FlexMeasures email you a digest of the automation runs that failed, by running ``flexmeasures monitor automations`` regularly, for instance every hour:

.. code-block:: bash

0 * * * * flexmeasures monitor automations

A run is reported when its execution failed, when it has been executing for longer than six hours, or when its dispatch failed and will not be attempted again.
A forecast run therefore counts once its five dispatch attempts are used up, and a run of any other type after its first failed attempt.
The six hours are set by :ref:`monitor_automations_stuck_after`, and ``--stuck-after-minutes`` overrides that setting.

Each run is reported once, so if one monitoring run is missed, the next one still reports what failed in the meantime.
Repeated failures of an automation are collapsed into one entry, saying how often it failed, when it first and last did, and with which error.
No email is sent when nothing failed.

Every automation is monitored, unless it opts out, for instance an experiment whose failures nobody needs to hear about.
Untick *Monitored* on the Automations page, create it with ``flexmeasures add automation --unmonitored``, switch it with ``flexmeasures edit automation --monitored/--unmonitored``, or send ``"monitored": false`` to the API.
Its runs are then left out of every digest.
They are not marked as reported, though, so once the automation is monitored again, the next digest reports the runs that failed in the meantime,
saying how many of them failed while it was not monitored, and when monitoring was switched back on.
Only the host's admins and the account-admins of the asset's organisation may switch monitoring off or on, as it decides what the recipients of the digest hear about.
Switching it off only keeps the automation out of these digests; the automation runs as before.

The digest goes to :ref:`default_monitoring_mail_recipients`, or to whoever you name with ``--recipient`` (a user ID or an email address, as often as needed).
The command stays out of ``flexmeasures jobs run-automations``, so that a problem with monitoring, such as an unreachable mail server, never holds up the automations themselves.

Running one automation on demand
--------------------------------

Expand Down
11 changes: 11 additions & 0 deletions documentation/host/error-monitoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,17 @@ If you run distinct filters, such as separate checks per account, account group
.. todo:: Adding roles and assigning them to accounts is not supported by the UI yet (user roles can be added in the UI). Account roles can be added with ``flexmeasures add account-role``.


Monitoring automations
-----------------------

The CLI task ``flexmeasures monitor automations`` emails a digest of the automation runs that failed since its last check, with repeated failures collapsed per automation.
Run it from cron, for instance every hour; see :ref:`monitoring_automations` for what it reports.

.. code-block:: bash

$ flexmeasures monitor automations --recipient [email protected]


Monitoring task runs
---------------------

Expand Down
33 changes: 29 additions & 4 deletions flexmeasures/api/v3_0/assets.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@
resolve_automation_sensors,
run_automation,
update_automation,
user_may_change_monitoring,
)
from flexmeasures.data.models.charts import ASSET_CHART_BUILDERS
from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
Expand Down Expand Up @@ -103,6 +104,7 @@
)
from flexmeasures.api.v3_0.utils import use_legacy_job_responses
from flexmeasures.api.common.responses import (
invalid_sender,
unprocessable_entity,
request_accepted_for_processing,
)
Expand Down Expand Up @@ -505,6 +507,12 @@ def _eager_load_asset_relations_dumped_by(
return query


# Switching monitoring off silences the digest for whoever receives it, so it is not up to everyone who manages automations.
MONITORING_IS_FOR_ADMINS = (
"Only admins may switch the monitoring of an automation on or off."
)


class AssetAPI(FlaskView):
"""
This API view exposes generic assets.
Expand Down Expand Up @@ -1509,7 +1517,7 @@ def get_automations(
description: |
The response will be a list of automations: recurring forecasting, scheduling, reporting or plugin-defined tasks
defined on the asset. Each entry shows the automation's ID, when it was created,
its type, name, activation status, and its recurrence, both as a cron string
its type, name, activation status, whether it is monitored, and its recurrence, both as a cron string
and described in natural language. Each entry also shows the IANA timezone in which its cron expression is interpreted,
and both its cursor and its next scheduled run as clock times in that same timezone (the next run is null while inactive).
The next run excludes pending catch-up work.
Expand Down Expand Up @@ -1556,6 +1564,7 @@ def get_automations(
recurrence-description: "At 06:00"
revision: 1
active: true
monitored: true
job-stats:
finished: 3
redis-connection-err: null
Expand Down Expand Up @@ -1670,6 +1679,7 @@ def get_automation(self, id: int, automation_id: int, asset: GenericAsset):
recurrence-description: "At 06:00"
revision: 1
active: true
monitored: true
parameters:
sensor: 2092
source:
Expand Down Expand Up @@ -1876,6 +1886,10 @@ def post_automation(self, automation_data: dict, id: int, asset: GenericAsset):
tags:
- Assets
"""
if automation_data.get("monitored") is False and not user_may_change_monitoring(
current_user, asset
):
return invalid_sender(message=MONITORING_IS_FOR_ADMINS)
try:
automation, warnings = create_automation(
asset, origin="API", check_permissions=True, **automation_data
Expand Down Expand Up @@ -1912,13 +1926,13 @@ def patch_automation(
self, automation_data: dict, id: int, automation_id: int, asset: GenericAsset
):
"""
.. :quickref: Assets; Update an automation's name, cron string, timezone or activation status.
.. :quickref: Assets; Update an automation's name, cron string, timezone, activation status or monitoring.

---
patch:
summary: Update an automation's name, cron string, timezone or activation status.
summary: Update an automation's name, cron string, timezone, activation status or monitoring.
description: |
Any subset of the fields `name`, `cronstr`, `timezone` and `active` can be sent.
Any subset of the fields `name`, `cronstr`, `timezone`, `active` and `monitored` can be sent.
Changing the recurrence or the timezone, or reactivating the automation, resets its cursor to just before the minute of the change,
so runs from before it are not caught up on, while a run due in that very minute still is.
Other automation fields cannot be updated; instead, create a new automation.
Expand Down Expand Up @@ -1947,6 +1961,10 @@ def patch_automation(
summary: Deactivate the automation
value:
active: false
stop-monitoring:
summary: Stop reporting the automation's failed runs
value:
monitored: false
responses:
200:
description: PROCESSED
Expand All @@ -1966,6 +1984,13 @@ def patch_automation(
return {
"message": f"Asset {asset.id} has no automation with id {automation_id}."
}, 404
monitored = automation_data.get("monitored")
if (
monitored is not None
and monitored != automation.monitored
and not user_may_change_monitoring(current_user, asset)
):
return invalid_sender(message=MONITORING_IS_FOR_ADMINS)
update_automation(automation, origin="API", **automation_data)
db.session.commit()
response = automation_schema.dump(automation)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,8 @@ def test_the_output_scoped_automation_types_are_supported():

def test_copy_copies_direct_and_descendant_automations(fresh_db, automated_site):
"""Automations on the copied asset and on its descendants are copied, inactive and without run history."""
# One original opted out of monitoring, so that the copies show they keep that choice rather than the default.
automated_site["meter_automation"].monitored = False
asset_copy = copy_asset(automated_site["site"])

assert asset_copy.skipped_automations == []
Expand All @@ -187,6 +189,7 @@ def test_copy_copies_direct_and_descendant_automations(fresh_db, automated_site)
assert copied.type == original.type
assert copied.cronstr == original.cronstr
assert copied.timezone == original.timezone
assert copied.monitored == original.monitored
# A copy waits to be switched on, and starts without the original's run history.
assert copied.active is False
assert copied.cursor > OLD_CURSOR
Expand Down
Loading
Loading