Skip to content

Feat: monitor automations user digests - #2703

Open
BelhsanHmida wants to merge 40 commits into
mainfrom
feat/2519-monitor-automations-user-digests
Open

BelhsanHmida wants to merge 40 commits into
mainfrom
feat/2519-monitor-automations-user-digests

Conversation

@BelhsanHmida

@BelhsanHmida BelhsanHmida commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

Extends flexmeasures monitor automations (#2671) to also email the people responsible for the assets, so they hear about their failed automations without the host forwarding them.

Stacked on #2671: this PR is opened against feat/2519-monitor-automations and will move to main once #2671 is merged.

  • Besides the host digest, each user who should hear about a failed run gets one digest per monitoring run:
    • users with the account-admin role in the asset's organisation;
    • users with the consultant role in its consultancy organisation (account.consultancy_account_id), as the ACL already has it.
  • Each user's digest lists only the automations they may read, checked with user_has_scoped_permission (Add scoped named permissions and restricted user roles #2607) against automation.__acl__()["read"], and collapses repeated failures the way the host digest does.
  • Inactive users and users without an email address are skipped.
  • The host keeps the full digest, which is the only one that covers public assets (they have no organisation).
  • User digests say "on assets you look after" and leave out the host's "check the logs" line.
  • New setting FLEXMEASURES_MONITOR_AUTOMATIONS_NOTIFY_USERS, off by default, so hosts decide when their users start receiving these emails.
  • Added changelog item in documentation/changelog.rst (this PR's number, appended to Feat: flexmeasures monitor automations #2671's line) and documentation/cli/change_log.rst.

Decisions worth a look

  • A run counts as reported once the host's digest is sent. alerted_at is committed first, then the user digests go out, and a user digest that cannot be sent is logged and not retried. The alternative, waiting until every user is told, would resend the host digest and all other user digests after a single bounce, unless we track alerts per recipient.
  • With the setting off, runs are marked exactly as in Feat: flexmeasures monitor automations #2671.
  • The recipient rule is a placeholder until Notifications from hosts to users #2039 (notifications from hosts to users) lets users choose.
  • --recipient only replaces the host's recipients; user digests are still sent when the setting is on.

Look & Feel

One monitoring run with the setting on, where Acme (consulted by GreenConsult) and Beta each have failed automations, and a public asset has a stuck one:

Inbox Automations listed
the host all four, the public one included
Acme's account admin Acme's two
GreenConsult's consultant Acme's two
Beta's account admin Beta's one

Acme's plain user, its inactive admin and its admin without an email address get nothing, and neither does GreenConsult's user without the consultant role.

What Acme's account admin receives:

Subject: Failed runs of 2 automation(s)

These automations on assets you look after have runs that failed, or that have been executing for longer than 6 hours, which were not reported before:

Automation 1 'Solar forecasts' (forecasting) on asset 4 'Rooftop PV' of organisation 'Acme':
  failed 3 times (execution failed: 3), last at 2026-10-12 07:44 UTC (run 3)
  last error: ValueError: not enough data (2 h ago)

Automation 2 'Heat pump schedule' (scheduling) on asset 5 'House' of organisation 'Acme':
  failed once (dispatch failed: 1), last at 2026-10-12 08:44 UTC (run 4)

Each automation's runs are listed on the Automations page of its asset.

How to test

  • flexmeasures/cli/tests/test_monitor_automations_fresh_db.py:
    • automations on a prosumer, a consultancy client and a public asset: who gets a digest, and with which automations;
    • user digests are off by default;
    • a user digest that bounces is not sent again, while the others still go out.
  • Each check was broken in turn to see a test fail: the ACL filter, excluding public assets, the active and email filters, the consultant and account-admin clauses, the setting, the user wording and the mail error handling.
  • Manually, with MailHog (docker run -p 1025:1025 -p 8025:8025 mailhog/mailhog) catching the emails:
    1. Set MAIL_SERVER = "127.0.0.1", MAIL_PORT = 1025 and FLEXMEASURES_DEFAULT_MONITORING_MAIL_RECIPIENTS = ["[email protected]"], and create organisations, users and failed runs like those in the table above.
    2. Run flexmeasures monitor automations: only the host is emailed.
    3. Reset with UPDATE automation_run SET alerted_at = NULL;, set FLEXMEASURES_MONITOR_AUTOMATIONS_NOTIFY_USERS = True, and run it again: the inboxes match the table.
    4. Run it once more: "All good", and no emails.
    5. Stop MailHog, reset and run it: the host digest fails, no run is marked, and no user is emailed.

Further Improvements

  • Consider switching the setting on by default after one release.
  • We might add notifications soon, now planned for hosts to users, and exopand those to include this. So users can subscribe to automation alerts. This means we drop the choice which user roles exactlz get them.

Related Items

Closes #2668, a sub-issue of #2519. Builds on #2671.


Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

…s end

Context:
- Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs.

Change:
- record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded.
- The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job.
- Plugin jobs carry automation_run_id in their trigger.
- A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended.

Change:
- Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded.
- A plugin run records its job's outcome, also when the job finished before the run recorded it.
- Report run tests check the run's outcome: succeeded, nothing to do, and refused.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, as forecast runs did.

Change:
- Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, not only forecast runs.

Change:
- Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a monitor command will email a digest of failed automation runs,
  and has to report each failure once, also when a monitoring run is missed.

Change:
- Add a nullable alerted_at column to AutomationRun, with a migration.
- Add a partial index on the runs the monitor looks at
  (not reported yet, and failed, running or not fully dispatched),
  so that its hourly query does not scan the whole run history.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: a run whose worker died stays "running" for good, so monitoring needs a threshold after which it counts as stuck.

Change:
- Add the setting, defaulting to six hours, well above any job timeout.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…rted yet

Context:
- #2519: monitoring reports runs whose execution failed, which are stuck, or whose dispatch failed for good.

Change:
- Add get_unalerted_failed_automation_runs, with one criterion for every automation type:
  only a forecast run with dispatch attempts to spare is left out, as it will be dispatched again,
  and so is a run that a runner still holds a live claim on.
- Add group_failed_runs_by_automation, collapsing failures per automation with their counts, latest run and latest error.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: automations fail silently unless someone watches their asset's Automations page.

Change:
- Email one digest of the failed runs to the monitoring recipients (or --recipient),
  collapsed per automation, and mark the runs as reported once it is sent.
- Send nothing when nothing failed, and refuse to run without recipients,
  rather than marking runs as reported that nobody heard of.
- --stuck-after-minutes overrides FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER.
- Kept out of flexmeasures jobs run-automations, so a mail problem never holds up dispatching.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: cover the new monitor command.

Change:
- Drive forecast runs to a failed execution, a stuck execution and an exhausted dispatch through the real runner and worker,
  and check each is reported once, and not before it counts.
- Check which runs count as failed for other types, the threshold's config and CLI override,
  collapsing per automation, and refusing to run without recipients.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- #2519: new monitor command and setting.

Change:
- Describe what the command reports in the automations feature docs, and point to it from the host's error monitoring page.
- List the command, document the stuck threshold setting, and add a CLI changelog entry.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Off by default, so that hosts decide when their users start receiving digests of failed automation runs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The account admins of each asset's organisation and the consultants of its consultancy organisation, if active and with an email address, each with the failed automations they may read.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
With FLEXMEASURES_MONITOR_AUTOMATIONS_NOTIFY_USERS on, flexmeasures monitor automations also emails each account admin and consultant a digest of the failed automations they may read.
Runs are marked as reported once the host's digest is sent, so a user digest that cannot be sent is logged rather than retried.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor.

Change:
- Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py.
- Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor.
- The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type.
- The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Builds the host digest on top of the run outcomes of schedule, report and plugin runs.
Brings in current main as well.
The only conflict was the CLI changelog, where both sides' entries are kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained migrations since this branch started, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise f7c2e8b1a409, main's head, rather than c5e1a7b94d20.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ng digest

Context:
- #2670 has schedule, report and plugin runs record how their jobs ended,
  so their failed executions can now reach the host digest.

Change:
- Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error.
- The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…xecution

Context:
- Since #2670, every automation type records how its run's jobs ended.

Change:
- Remove the note from the monitoring section of the automations docs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- PR #2671 adds flexmeasures monitor automations.

Change:
- Add one bullet for it to the per-PR automations list.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida BelhsanHmida self-assigned this Oct 12, 2026
@BelhsanHmida
BelhsanHmida marked this pull request as ready for review October 12, 2026 01:45
@BelhsanHmida
BelhsanHmida changed the base branch from main to feat/2519-monitor-automations October 12, 2026 01:45
…tomations-user-digests

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/cli/change_log.rst
#	documentation/features/automations.rst
…eir scoped permissions

Since #2607, a role grant and an ACL principal must match together, which user_has_scoped_permission checks and user_matches_principals does not.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@read-the-docs-community

read-the-docs-community Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

BelhsanHmida and others added 10 commits October 12, 2026 02:52
Context:
- The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue.

Change:
- Say execution state and job records are kept for every type of automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py.

Change:
- Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
… them

A schedule run now names its jobs after itself and records them before they are queued,
so this branch no longer records them once they are queued, which would record each job twice.
Its jobs report how they ended through the same decorator as before, as they are looked up by their RQ ID.
A sequential schedule's device jobs are keyed by the sensor they schedule, rather than by their position.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…ts stored summary

Every queue's exception handler now stores the type and message of a job's error in its meta (#2698),
so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line.
The meta is read afresh, as the run holds the job as it was when queued.
A job without such a summary is still described by its traceback, without a type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…-monitor-automations

Prepares for merging #2670's squash commit, whose tree is identical to this tip.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Its content is already here from the branch's own lineage, so conflicts take this branch's side.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
The only conflict was the CLI changelog, where main had moved the `show automations` entry up; this branch's `monitor automations` entry is kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…igration

Context:
- Main gained a merge migration since the last sync, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise bf22b21d0990, main's head, rather than f7c2e8b1a409.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…utomations-user-digests

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida
BelhsanHmida requested a review from nhoening October 29, 2026 11:29
@nhoening nhoening mentioned this pull request Nov 12, 2026
2 tasks done
…or CLI

Requested in review, so other code can reuse it.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Otherwise a host's first digest lists every run that ever failed.
The upgrade marks the runs that the monitor would report now, by its rules at this revision, and leaves out runs that may still be retried or finish.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…tomations page

The digest is sent from the CLI, where FlexMeasures does not know its own address, so the new FLEXMEASURES_BASE_URL setting gives it.
Without the setting, the digest carries no links, as before.
The Automations page opens the tab and info panel of the automation named in ?automation=<id>, which show its latest run, its error and its jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…utomations-user-digests

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	flexmeasures/cli/monitor.py
#	flexmeasures/cli/tests/test_monitor_automations_fresh_db.py
#	flexmeasures/utils/config_defaults.py
…riable name

Fixes flake8's E741 from the merge of #2671.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…omations

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
…utomations-user-digests

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
Base automatically changed from feat/2519-monitor-automations to main November 17, 2026 22:27

@nhoening nhoening left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this successfully, but I have one remaining request.


.. _monitor_automations_notify_users:

FLEXMEASURES_MONITOR_AUTOMATIONS_NOTIFY_USERS

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this would be easier to deal with if it was a parameter flag to the CLI command.

Changing a config setting means we have to re-deploy the app. Changing the call to a cron job command is much easier.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Email account admins and consultants about their own failed automation runs

3 participants