Skip to content

Feat: automation run outcomes - #2670

Merged
Flix6x merged 9 commits into
mainfrom
feat/automation-run-outcomes
Oct 20, 2026
Merged

Flix6x merged 9 commits into
mainfrom
feat/automation-run-outcomes

Conversation

@BelhsanHmida

@BelhsanHmida BelhsanHmida commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Schedule, report and plugin runs now record how their jobs ended, as forecast runs already did, so every run's execution_state moves to succeeded or failed instead of staying pending.

  • A schedule run's jobs are recorded before they are queued, under IDs derived from the run, by Dispatch a schedule automation run again, queueing only the jobs it still owes #2657 (each device job and the wrap-up job, for a sequential schedule). (Edited: this PR first recorded them after queueing, too; that part was dropped when merging main.)
  • record_automation_run_jobs records the jobs a report or plugin run queued, then takes from Redis the outcome of any job a worker already finished before the record existed, with the error type and message the queue's exception handler stored (One failure summary in a job's meta for every queue #2698). A run that queued no jobs has succeeded.
  • The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID. It wraps make_schedule, cb_done_sequential_scheduling_job, run_report_job and execute_automation_job, so plugin types get it for free. Jobs queued outside a run, or never recorded by one, are left alone.
  • Plugin jobs now carry automation_run_id in their trigger, as schedule and report jobs already did.
  • Docs: the paragraph on a run's execution state in features/automations.rst now covers every type.
  • Added changelog item in documentation/changelog.rst (PR number appended to the Feat/2393 durable automation runs #2457 entry)

Answers to the issue's questions

  • Hook point: a wrapper in the worker. The decorator leaves the RQ on_success/on_failure callbacks to the fallback scheduler, which already uses them, and leaves the exception handlers untouched.
  • A schedule that fell back: a failed run, with the error that made it fall back. The schedule the automation asked for could not be computed, so the run reports a failure, whatever the fallback scheduler then produced.
  • A job that died with its worker: left to the host digest's "stuck run" rule (Feat: flexmeasures monitor automations #2671). The job never reports back, so its run stays running, and the digest flags a run still running after the stuck threshold.

The issue proposed recording the intents before the jobs are enqueued. Schedule runs now do so, since #2657. Report and plugin runs get their job IDs from RQ, so their intents are recorded right after enqueueing, and the run catches up from Redis on any job that already ended. Giving those run-derived IDs as well is a possible follow-up.

How to test

On a toy battery account (flexmeasures add toy-account --kind battery) with today's prices on sensor 7, and a sensor on asset 5 for the report (here sensor 10):

printf 'duration: "PT12H"\nresolution: "PT15M"\nflex-model:\n  - sensor: 8\n' > schedule.yml
printf 'duration: "PT12H"\nresolution: "PT15M"\nsequential: true\nflex-model:\n  - sensor: 8\n' > sequential.yml
printf 'duration: "PT12H"\nresolution: "PT15M"\nflex-model:\n  - sensor: 8\n    soc-min: "400 kWh"\n    power-capacity: "1 kW"\n' > infeasible.yml
printf 'required_input:\n  - name: prices\nrequired_output:\n  - name: doubled\ntransformations:\n  - df_input: prices\n    method: multiply\n    args: [2]\n    df_output: doubled\n' > report-config.yml
printf 'input:\n  - name: prices\n    sensor: 7\noutput:\n  - name: doubled\n    sensor: 10\n' > report.yml

flexmeasures add automation --asset 5 --cron "* * * * *" --name "ok" --type scheduling --parameters schedule.yml
flexmeasures add automation --asset 5 --cron "* * * * *" --name "sequential" --type scheduling --parameters sequential.yml
flexmeasures add automation --asset 5 --cron "* * * * *" --name "no prices" --type scheduling --start-offset 5D --parameters schedule.yml
flexmeasures add automation --asset 5 --cron "* * * * *" --name "infeasible" --type scheduling --parameters infeasible.yml
flexmeasures add automation --asset 5 --cron "* * * * *" --name "report" --type reporting --reporter PandasReporter --config report-config.yml --parameters report.yml --start-offset DB
flexmeasures add automation --asset 5 --cron "* * * * *" --name "report up to midnight" --type reporting --reporter PandasReporter --config report-config.yml --parameters report.yml --end-offset DB

flexmeasures jobs run-automations   # once the next minute has started
flexmeasures jobs run-worker --queue scheduling
flexmeasures jobs run-worker --queue reporting

Then open each automation's Info panel on the Automations page (or GET /api/v3_0/assets/5/automations/<id>, under run-stats):

Automation Run ends
ok succeeded
sequential succeeded, with both its jobs (device-8, wrap-up)
no prices failed, UnknownPricesException
infeasible failed, InfeasibleProblemException
report succeeded
report up to midnight, on its second run succeeded without queueing a job, as there is nothing new to report

Before this PR, all of these stayed pending.
flexmeasures jobs run-automation --automation <id> still runs a job as before, and records no run.

Automated:

  • flexmeasures/data/tests/test_automations_fresh_db.py: test_a_schedule_run_records_how_its_jobs_ended, test_a_schedule_run_records_a_failed_job (single and sequential), test_a_schedule_job_queued_outside_a_run_is_not_recorded
  • flexmeasures/data/tests/test_plugin_automations_fresh_db.py: test_a_plugin_run_records_how_its_job_ended, test_a_job_that_ended_before_its_run_recorded_it_is_caught_up_on, test_a_job_that_failed_before_its_run_recorded_it_is_caught_up_on_with_its_error
  • flexmeasures/cli/tests/test_automations.py: the report run tests now also check the run's outcome (succeeded, nothing to do, refused).

Further Improvements

  • A run whose job never started stays pending, so a stuck rule timed from the start of execution misses it. This happens when a worker dies before picking the job up, or in a narrow race: a job finishes before its run recorded it, and RQ has not yet marked it finished when the run checks Redis. Feat: flexmeasures monitor automations #2671 is to also count a run as stuck when it finished dispatching longer ago than the threshold and is still pending or running.
  • The digest only alerts: a stuck run stays running in the run history. If that bothers people, the dispatcher could fail such runs, after asking Redis what became of their jobs.
  • Existing schedule, report and plugin runs stay pending; no backfill.

Related Items

Closes #2666.


Sign-off

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

…s end

Context:
- Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs.

Change:
- record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded.
- The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job.
- Plugin jobs carry automation_run_id in their trigger.
- A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended.

Change:
- Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded.
- A plugin run records its job's outcome, also when the job finished before the run recorded it.
- Report run tests check the run's outcome: succeeded, nothing to do, and refused.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, as forecast runs did.

Change:
- Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- Schedule, report and plugin runs now record how their jobs ended, not only forecast runs.

Change:
- Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor.

Change:
- Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py.
- Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor.
- The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type.
- The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 12, 2026
…omations

Builds the host digest on top of the run outcomes of schedule, report and plugin runs.
Brings in current main as well.
The only conflict was the CLI changelog, where both sides' entries are kept.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 12, 2026
…ng digest

Context:
- #2670 has schedule, report and plugin runs record how their jobs ended,
  so their failed executions can now reach the host digest.

Change:
- Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error.
- The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 12, 2026
…xecution

Context:
- Since #2670, every automation type records how its run's jobs ended.

Change:
- Remove the note from the monitoring section of the automations docs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida
BelhsanHmida requested a balanced review from Copilot October 12, 2026 01:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Context:
- The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue.

Change:
- Say execution state and job records are kept for every type of automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context:
- main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py.

Change:
- Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
@BelhsanHmida BelhsanHmida mentioned this pull request Oct 14, 2026
22 tasks done
Flix6x and others added 2 commits October 20, 2026 12:01
… them

A schedule run now names its jobs after itself and records them before they are queued,
so this branch no longer records them once they are queued, which would record each job twice.
Its jobs report how they ended through the same decorator as before, as they are looked up by their RQ ID.
A sequential schedule's device jobs are keyed by the sensor they schedule, rather than by their position.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
…ts stored summary

Every queue's exception handler now stores the type and message of a job's error in its meta (#2698),
so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line.
The meta is read afresh, as the run holds the job as it was when queued.
A job without such a summary is still described by its traceback, without a type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>
@Flix6x

Flix6x commented Oct 20, 2026

Copy link
Copy Markdown
Member

@BelhsanHmida, I took this PR over to get it past #2657, which recorded schedule runs differently. Could you review my changes?

  1. Merged main (94e4a52). Since Dispatch a schedule automation run again, queueing only the jobs it still owes #2657, a schedule run records its jobs before queueing them, under IDs derived from the run (automation-run-{id}-device-{sensor id}, -wrap-up, -schedule). Recording them again after queueing broke both unique constraints on automation_run_job. So I kept main's _run_schedule_automation and dropped this PR's recording of schedule jobs. Your decorator still records how they end, because it finds a job by its RQ ID. test_a_schedule_run_records_how_its_jobs_ended now expects device-<sensor id> instead of device-0. record_automation_run_jobs now only serves report and plugin runs, and its docstrings say so.
  2. The error of a job that failed before its run recorded it (cc5295a). Every queue's exception handler now stores the error's type and message in the job's meta (One failure summary in a job's meta for every queue #2698). So the catch-up path records that type, instead of an AutomationJobFailed built from the last traceback line. Writing the test showed one more thing: the run holds the Job as it was when queued, so the meta has to be fetched again first. New test: test_a_job_that_failed_before_its_run_recorded_it_is_caught_up_on_with_its_error. It fails without either part of the change.

I updated the description to match, marking the edited bullet. Full suite: 3182 passed, 4 xfailed.

A possible follow-up: give report and plugin runs IDs derived from the run as well, so they too can record their jobs before queueing them (see #2694).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The change hinges on subtle cross-queue worker/transaction semantics and dispatch-time-vs-worker-time race handling (including Redis catch-up and session rollbacks) that warrant a human maintainer's final judgment despite no concrete defect being found.

No open issues 🎉

Review effort: Balanced

@BelhsanHmida

Copy link
Copy Markdown
Contributor Author

@Flix6x Reviewed both, looks good. Keeping #2657's recording and finding schedule jobs by their RQ ID is the right fix. I also checked that RQ runs the exception handler before it marks a job failed, so the catch-up always finds the stored summary.

@Flix6x
Flix6x merged commit d4a0c2b into main Oct 20, 2026
14 checks passed
@Flix6x
Flix6x deleted the feat/automation-run-outcomes branch October 20, 2026 21:10
@Flix6x Flix6x added this to the 1.1.0 milestone Oct 20, 2026
BelhsanHmida added a commit that referenced this pull request Oct 29, 2026
…-monitor-automations

Prepares for merging #2670's squash commit, whose tree is identical to this tip.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 29, 2026
Its content is already here from the branch's own lineage, so conflicts take this branch's side.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
BelhsanHmida added a commit that referenced this pull request Oct 30, 2026
Resolve conflicts with the squash-merged #2670 and with #2657:
take main's naming of schedule jobs and its job-failure path,
keep the release of waiting runs where a run ends,
and use AUTOMATION_RUN_RETRIED_TYPES, which now includes scheduling.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
nhoening pushed a commit that referenced this pull request Nov 17, 2026
* data/services/automations: record how schedule, report and plugin runs end

Context:
- Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs.

Change:
- record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded.
- The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job.
- Plugin jobs carry automation_run_id in their trigger.
- A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* data/tests: cover how schedule, report and plugin runs end

Context:
- Schedule, report and plugin runs now record how their jobs ended.

Change:
- Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded.
- A plugin run records its job's outcome, also when the job finished before the run recorded it.
- Report run tests check the run's outcome: succeeded, nothing to do, and refused.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* docs: every automation run records how its jobs ended

Context:
- Schedule, report and plugin runs now record how their jobs ended, as forecast runs did.

Change:
- Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* changelog: append PR #2670 to the durable automation runs entry

Context:
- Schedule, report and plugin runs now record how their jobs ended, not only forecast runs.

Change:
- Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* data/models: record when an automation run was reported by monitoring

Context:
- #2519: a monitor command will email a digest of failed automation runs,
  and has to report each failure once, also when a monitoring run is missed.

Change:
- Add a nullable alerted_at column to AutomationRun, with a migration.
- Add a partial index on the runs the monitor looks at
  (not reported yet, and failed, running or not fully dispatched),
  so that its hourly query does not scan the whole run history.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* utils/config: add FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER

Context:
- #2519: a run whose worker died stays "running" for good, so monitoring needs a threshold after which it counts as stuck.

Change:
- Add the setting, defaulting to six hours, well above any job timeout.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* data/services: find the automation runs that failed and were not reported yet

Context:
- #2519: monitoring reports runs whose execution failed, which are stuck, or whose dispatch failed for good.

Change:
- Add get_unalerted_failed_automation_runs, with one criterion for every automation type:
  only a forecast run with dispatch attempts to spare is left out, as it will be dispatched again,
  and so is a run that a runner still holds a live claim on.
- Add group_failed_runs_by_automation, collapsing failures per automation with their counts, latest run and latest error.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* cli: add flexmeasures monitor automations

Context:
- #2519: automations fail silently unless someone watches their asset's Automations page.

Change:
- Email one digest of the failed runs to the monitoring recipients (or --recipient),
  collapsed per automation, and mark the runs as reported once it is sent.
- Send nothing when nothing failed, and refuse to run without recipients,
  rather than marking runs as reported that nobody heard of.
- --stuck-after-minutes overrides FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER.
- Kept out of flexmeasures jobs run-automations, so a mail problem never holds up dispatching.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* cli/tests: test flexmeasures monitor automations

Context:
- #2519: cover the new monitor command.

Change:
- Drive forecast runs to a failed execution, a stuck execution and an exhausted dispatch through the real runner and worker,
  and check each is reported once, and not before it counts.
- Check which runs count as failed for other types, the threshold's config and CLI override,
  collapsing per automation, and refusing to run without recipients.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* docs: document flexmeasures monitor automations

Context:
- #2519: new monitor command and setting.

Change:
- Describe what the command reports in the automations feature docs, and point to it from the host's error monitoring page.
- List the command, document the stuck threshold setting, and add a CLI changelog entry.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* data/migrations: revise main's latest migration with the alerted_at migration

Context:
- Main gained migrations since this branch started, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise f7c2e8b1a409, main's head, rather than c5e1a7b94d20.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* tests: report failed schedule, report and plugin runs in the monitoring digest

Context:
- #2670 has schedule, report and plugin runs record how their jobs ended,
  so their failed executions can now reach the host digest.

Change:
- Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error.
- The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* docs: drop the note that only forecast runs are monitored for their execution

Context:
- Since #2670, every automation type records how its run's jobs ended.

Change:
- Remove the note from the monitoring section of the automations docs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* changelog: add PR #2671 to the automations, in detail list

Context:
- PR #2671 adds flexmeasures monitor automations.

Change:
- Add one bullet for it to the per-PR automations list.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* docs/api: run-stats records execution state for every automation type

Context:
- The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue.

Change:
- Say execution state and job records are kept for every type of automation.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* Record the error of a job that failed before its run recorded it by its stored summary

Every queue's exception handler now stores the type and message of a job's error in its meta (#2698),
so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line.
The meta is read afresh, as the run holds the job as it was when queued.
A job without such a summary is still described by its traceback, without a type.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC
Signed-off-by: F.N. Claessen <[email protected]>

* data/migrations: revise main's latest migration with the alerted_at migration

Context:
- Main gained a merge migration since the last sync, which left two Alembic heads.

Change:
- Make d4b7e9a1c362 revise bf22b21d0990, main's head, rather than f7c2e8b1a409.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* utils/time_utils: move the exact duration formatting out of the monitor CLI

Requested in review, so other code can reuse it.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* data/migrations: count runs that failed before the upgrade as reported

Otherwise a host's first digest lists every run that ever failed.
The upgrade marks the runs that the monitor would report now, by its rules at this revision, and leaves out runs that may still be retried or finish.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

* cli/monitor: link each automation in the digest to its info on the Automations page

The digest is sent from the CLI, where FlexMeasures does not know its own address, so the new FLEXMEASURES_BASE_URL setting gives it.
Without the setting, the digest carries no links, as before.
The Automations page opens the tab and info panel of the automation named in ?automation=<id>, which show its latest run, its error and its jobs.

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>

---------

Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Signed-off-by: F.N. Claessen <[email protected]>
Co-authored-by: F.N. Claessen <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record how scheduling reporting and plugin automation run ends.

3 participants