Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6c7798a
data/services/automations: record how schedule, report and plugin run…
BelhsanHmida Oct 7, 2026
196ccc1
data/tests: cover how schedule, report and plugin runs end
BelhsanHmida Oct 7, 2026
da14723
docs: every automation run records how its jobs ended
BelhsanHmida Oct 7, 2026
2441424
changelog: append PR #2670 to the durable automation runs entry
BelhsanHmida Oct 7, 2026
56e5f83
data/models: record when an automation run was reported by monitoring
BelhsanHmida Oct 7, 2026
39ca4a0
utils/config: add FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER
BelhsanHmida Oct 7, 2026
ddfbe7a
data/services: find the automation runs that failed and were not repo…
BelhsanHmida Oct 7, 2026
3cf9ef6
cli: add flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
47e200c
cli/tests: test flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
379a8b8
docs: document flexmeasures monitor automations
BelhsanHmida Oct 7, 2026
21fc37d
Merge origin/main into feat/automation-run-outcomes
BelhsanHmida Oct 12, 2026
185502b
Merge feat/automation-run-outcomes (#2670) into feat/2519-monitor-aut…
BelhsanHmida Oct 12, 2026
f58aaa6
data/migrations: revise main's latest migration with the alerted_at m…
BelhsanHmida Oct 12, 2026
d37502e
tests: report failed schedule, report and plugin runs in the monitori…
BelhsanHmida Oct 12, 2026
3734a4e
docs: drop the note that only forecast runs are monitored for their e…
BelhsanHmida Oct 12, 2026
d910da4
changelog: add PR #2671 to the automations, in detail list
BelhsanHmida Oct 12, 2026
5f37108
docs/api: run-stats records execution state for every automation type
BelhsanHmida Oct 12, 2026
627c635
Merge origin/main into feat/automation-run-outcomes
BelhsanHmida Oct 14, 2026
94e4a52
Merge main, where #2657 records a schedule run's jobs before queueing…
Flix6x Oct 20, 2026
cc5295a
Record the error of a job that failed before its run recorded it by i…
Flix6x Oct 20, 2026
0c85cd1
Merge feat/automation-run-outcomes (#2670's final tip) into feat/2519…
BelhsanHmida Oct 29, 2026
84d04b0
Merge #2670's squash commit into feat/2519-monitor-automations
BelhsanHmida Oct 29, 2026
576fa2a
Merge origin/main into feat/2519-monitor-automations
BelhsanHmida Oct 29, 2026
e83eb29
data/migrations: revise main's latest migration with the alerted_at m…
BelhsanHmida Oct 29, 2026
b57c036
utils/time_utils: move the exact duration formatting out of the monit…
BelhsanHmida Nov 15, 2026
ff3ec20
data/migrations: count runs that failed before the upgrade as reported
BelhsanHmida Nov 15, 2026
644b604
cli/monitor: link each automation in the digest to its info on the Au…
BelhsanHmida Nov 15, 2026
8ed0a06
Merge remote-tracking branch 'origin/main' into feat/2519-monitor-aut…
BelhsanHmida Nov 15, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ v1.2.0 | December xx, 2026

New features
-------------
* Hosts can be emailed a digest of the automation runs that failed, got stuck or could not be queued, with ``flexmeasures monitor automations``, run from cron; each run is reported once, repeated failures are collapsed per automation, and with the new ``FLEXMEASURES_BASE_URL`` setting each automation links to its *Automations* page [see `PR #2671 <https://www.github.com/FlexMeasures/flexmeasures/pull/2671>`_]

Infrastructure / Support
-------------------------
Expand Down
5 changes: 5 additions & 0 deletions documentation/cli/change_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
FlexMeasures CLI Changelog
**************************

since v1.2.0 | December xx, 2026
=================================

* Add ``flexmeasures monitor automations`` to email a digest of the automation runs that failed since the last check: runs whose execution failed, which have been executing for too long (``--stuck-after-minutes``), or whose dispatch failed for good. Each run is reported once, and repeated failures are collapsed per automation.

since v1.1.0 | November 05, 2026
=================================

Expand Down
1 change: 1 addition & 0 deletions documentation/cli/commands.rst
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ of which some are referred to in this documentation.
================================================= =======================================
``flexmeasures monitor latest-run`` Check if the given task's last successful execution happened less than the allowed time ago.
``flexmeasures monitor last-seen`` Check if given users last contact (via a request) happened less than the allowed time ago.
``flexmeasures monitor automations`` Email a digest of the automation runs that failed since the last check.
================================================= =======================================


Expand Down
25 changes: 24 additions & 1 deletion documentation/configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -878,11 +878,34 @@ Default: ``None``
FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run`` and ``flexmeasures monitor last-seen`` if no explicit user recipients are given. For example ``["[email protected]", "[email protected]"]``.
E-mail addresses to send monitoring alerts to from the CLI tasks ``flexmeasures monitor latest-run``, ``flexmeasures monitor last-seen`` and ``flexmeasures monitor automations`` if no explicit user recipients are given. For example ``["[email protected]", "[email protected]"]``.

Default: ``[]``


.. _monitor_automations_stuck_after:

FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

How long an automation run may execute before ``flexmeasures monitor automations`` reports it as stuck, for instance because the worker running its job died.
Its ``--stuck-after-minutes`` option overrides this setting.

Default: ``timedelta(hours=6)``

.. _base_url:

FLEXMEASURES_BASE_URL
^^^^^^^^^^^^^^^^^^^^^

Where this server is reached on the web, for instance ``"https://ems.example.com"``.
Emails sent from the CLI, outside of any request, use it to link to the web pages they are about.
For instance, the digest of ``flexmeasures monitor automations`` links each failing automation to the Automations page of its asset.
Without it, these emails carry no links.

Default: ``None``


.. _redis-config:

Redis
Expand Down
22 changes: 22 additions & 0 deletions documentation/features/automations.rst
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,28 @@ This is what an operator needs to tell a run which failed before queueing anythi
Editing an automation's cron string or timezone, or reactivating it, counts up its revision.
Runs of the old and the new schedule therefore stay distinct, even when they fall on the same scheduled UTC time.

.. _monitoring_automations:

Monitoring automation runs
--------------------------

As a host, you can have FlexMeasures email you a digest of the automation runs that failed, by running ``flexmeasures monitor automations`` regularly, for instance every hour:

.. code-block:: bash

0 * * * * flexmeasures monitor automations

A run is reported when its execution failed, when it has been executing for longer than six hours, or when its dispatch failed and will not be attempted again.
A forecast run therefore counts once its five dispatch attempts are used up, and a run of any other type after its first failed attempt.
The six hours are set by :ref:`monitor_automations_stuck_after`, and ``--stuck-after-minutes`` overrides that setting.

Each run is reported once, so if one monitoring run is missed, the next one still reports what failed in the meantime.
Repeated failures of an automation are collapsed into one entry, saying how often it failed, when it last did and with which error.
No email is sent when nothing failed.

The digest goes to :ref:`default_monitoring_mail_recipients`, or to whoever you name with ``--recipient`` (a user ID or an email address, as often as needed).
The command stays out of ``flexmeasures jobs run-automations``, so that a problem with monitoring, such as an unreachable mail server, never holds up the automations themselves.

Running one automation on demand
--------------------------------

Expand Down
11 changes: 11 additions & 0 deletions documentation/host/error-monitoring.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,17 @@ If you run distinct filters, such as separate checks per account, account group
.. todo:: Adding roles and assigning them to accounts is not supported by the UI yet (user roles can be added in the UI). Account roles can be added with ``flexmeasures add account-role``.


Monitoring automations
-----------------------

The CLI task ``flexmeasures monitor automations`` emails a digest of the automation runs that failed since its last check, with repeated failures collapsed per automation.
Run it from cron, for instance every hour; see :ref:`monitoring_automations` for what it reports.

.. code-block:: bash

$ flexmeasures monitor automations --recipient [email protected]


Monitoring task runs
---------------------

Expand Down
130 changes: 129 additions & 1 deletion flexmeasures/cli/monitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,15 @@

from flexmeasures.data import db
from flexmeasures.data.models.task_runs import LatestTaskRun
from flexmeasures.data.services.automation_monitoring import (
AutomationFailures,
get_unalerted_failed_automation_runs,
group_failed_runs_by_automation,
)
from flexmeasures.data.models.user import Account, User
from flexmeasures.data.schemas.account import AccountIdField
from flexmeasures.api.common.schemas.users import UserIdField
from flexmeasures.utils.time_utils import server_now
from flexmeasures.utils.time_utils import exact_duration_str, server_now
from flexmeasures.cli.utils import LoggedClickExceptionGroup, MsgStyle


Expand Down Expand Up @@ -505,4 +510,127 @@ def monitor_last_seen(
db.session.commit()


# Error messages can carry a whole traceback, which does not help a digest that lists many automations.
MAX_ERROR_LENGTH_IN_DIGEST = 500


def _format_utc(moment: datetime) -> str:
return moment.astimezone(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")


def _times(count: int) -> str:
return "once" if count == 1 else f"{count} times"


def automation_page_url(automation) -> str | None:
"""Link to the Automations page of the automation's asset, opening that automation's info, if the server knows its own address."""
base_url = app.config.get("FLEXMEASURES_BASE_URL")
if not base_url:
return None
return f"{base_url.rstrip('/')}/assets/{automation.asset_id}/automations?automation={automation.id}"


def describe_automation_failures(failures: AutomationFailures) -> str:
"""Describe the failed runs of one automation in a few lines, for the monitoring digest."""
automation = failures.automation
asset = automation.asset
if asset.owner is None:
where = f"public asset {asset.id} '{asset.name}'"
else:
where = f"asset {asset.id} '{asset.name}' of organisation '{asset.owner.name}'"
reasons = ", ".join(
f"{reason}: {count}" for reason, count in failures.reason_counts.items()
)
last_run = failures.last_run
lines = [
f"Automation {automation.id} '{automation.name}' ({automation.type}) on {where}:",
f" failed {_times(len(failures.failed_runs))} ({reasons}),"
f" last at {_format_utc(last_run.scheduled_at)} (run {last_run.id})",
]
last_error = failures.last_error
if last_error:
if len(last_error) > MAX_ERROR_LENGTH_IN_DIGEST:
last_error = last_error[:MAX_ERROR_LENGTH_IN_DIGEST] + " ..."
lines.append(f" last error: {last_error}")
url = automation_page_url(automation)
if url:
lines.append(f" runs and jobs: {url}")
return "\n".join(lines)


@fm_monitor.command("automations")
@with_appcontext
@click.option(
"--stuck-after-minutes",
type=click.IntRange(min=1),
default=None,
help="Report a run as stuck once it has been executing for this many minutes. Defaults to the FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER setting (6 hours).",
)
@click.option(
"--recipient",
type=str,
multiple=True,
help="User ID or email address to send the digest to. Use multiple times if needed. If not used, the default monitoring mail recipients are informed.",
)
def monitor_automations(
stuck_after_minutes: int | None = None,
recipient: tuple[str, ...] = (),
):
"""
Email a digest of the automation runs that failed since the last check.

A run is reported when its execution failed, when it has been executing for too long (see --stuck-after-minutes),
or when its dispatch failed and will not be attempted again.
Each run is reported once, so a monitoring run that was missed is made up for by the next one.
Repeated failures are collapsed per automation, and no email is sent when nothing failed.

Meant to be run regularly, for instance hourly from cron.
"""
email_recipients = get_monitoring_email_recipients(recipient)
if not email_recipients:
# Marking runs as reported without telling anyone would hide them from the next monitoring run.
raise click.UsageError(
"Nobody to send the digest to. Set FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS or use --recipient."
)
if stuck_after_minutes is not None:
stuck_after = timedelta(minutes=stuck_after_minutes)
else:
stuck_after = app.config["FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER"]

now = server_now()
failed_runs = get_unalerted_failed_automation_runs(stuck_after, now=now)
if not failed_runs:
click.secho(
"All good ― no automation runs failed since the last check.",
**MsgStyle.SUCCESS,
)
return
failures = group_failed_runs_by_automation(failed_runs)

subject = f"Failed runs of {len(failures)} automation(s)"
body = (
f"These automations have runs that failed, or that have been executing for longer than {exact_duration_str(stuck_after)}, which were not reported before:\n\n"
+ "\n\n".join(describe_automation_failures(f) for f in failures)
+ (
"\n\nWe suggest to check the logs."
if app.config.get("FLEXMEASURES_BASE_URL")
else "\n\nEach automation's runs are listed on the Automations page of its asset.\nWe suggest to check the logs."
)
)
capture_message_for_sentry(subject)
email = Message(subject=subject, bcc=email_recipients)
email.body = body
app.mail.send(email)
app.logger.error(body)

# Only once the digest is out, so that a failure to send it leaves the runs to the next monitoring run.
for failed_run in failed_runs:
failed_run.run.alerted_at = now
db.session.commit()
click.secho(
f"Reported {len(failed_runs)} failed run(s) of {len(failures)} automation(s) to {', '.join(email_recipients)}.",
**MsgStyle.WARN,
)


app.cli.add_command(fm_monitor)
52 changes: 52 additions & 0 deletions flexmeasures/cli/tests/test_automations.py
Original file line number Diff line number Diff line change
Expand Up @@ -1869,6 +1869,58 @@ def test_report_automation_refuses_a_sensor_nobody_checked(
assert not app.redis_connection.get(f"automation-last-run:{automation.id}")


def test_a_failed_report_run_is_reported_by_monitoring(
app,
fresh_db,
setup_dummy_data_fresh_db,
clean_redis,
tmp_path,
freeze_server_now,
mocker,
):
"""A report run whose job failed on the worker shows up in the host's monitoring digest, with the job's error."""
from flexmeasures.cli.data_add import add_automation
from flexmeasures.cli.jobs import run_automations
from flexmeasures.cli.monitor import monitor_automations
from flexmeasures.data.models.reporting.pandas_reporter import PandasReporter
from flexmeasures.utils.job_utils import work_on_rq

sensor1_id, sensor2_id, report_sensor_id, _ = setup_dummy_data_fresh_db
report_sensor = fresh_db.session.get(Sensor, report_sensor_id)
cli_input = _report_automation_cli_input(
tmp_path,
sensor1_id,
sensor2_id,
report_sensor_id,
parameters_extra={"start-offset": "DB"},
asset_id=report_sensor.generic_asset_id,
)
cli_input[cli_input.index("0 1 * * *")] = "* * * * *" # due every minute
cli_input += ["--timezone", "UTC"]
freeze_server_now(datetime(2023, 4, 10, 10, 0, 30, tzinfo=timezone.utc))
runner = app.test_cli_runner()
result = runner.invoke(add_automation, cli_input)
assert "Successfully created" in result.output, result.output
automation = fresh_db.session.execute(select(Automation)).scalar_one()
result = runner.invoke(run_automations)
assert "queued 1 reporting job(s)" in result.output, result.output

mocker.patch.object(
PandasReporter, "compute", side_effect=KeyError("df_agg is missing")
)
work_on_rq(app.queues["reporting"])

with app.mail.record_messages() as outbox:
result = runner.invoke(monitor_automations, ["--recipient", "[email protected]"])

assert result.exit_code == 0, result.output
assert len(outbox) == 1
body = outbox[0].body
assert f"Automation {automation.id} '{automation.name}' (reporting)" in body
assert "failed once (execution failed: 1)" in body
assert "last error: KeyError: 'df_agg is missing'" in body


def _two_output_report_automation_cli_input(
tmp_path,
sensor1_id,
Expand Down
3 changes: 2 additions & 1 deletion flexmeasures/cli/tests/test_monitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
def test_monitor_help(app):
from flexmeasures.cli.monitor import (
fm_monitor,
monitor_automations,
monitor_last_seen,
monitor_latest_run,
)
Expand All @@ -19,7 +20,7 @@ def test_monitor_help(app):
result = runner.invoke(fm_monitor, ["--help"])
assert result.exit_code == 0

for command in (monitor_last_seen, monitor_latest_run):
for command in (monitor_last_seen, monitor_latest_run, monitor_automations):
result = runner.invoke(command, ["--help"])

assert result.exit_code == 0
Expand Down
Loading
Loading