Repository navigation
Feat: automation run outcomes - #2670
Conversation
…s end Context: - Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs. Change: - record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded. - The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job. - Plugin jobs carry automation_run_id in their trigger. - A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended. Change: - Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded. - A plugin run records its job's outcome, also when the job finished before the run recorded it. - Report run tests check the run's outcome: succeeded, nothing to do, and refused. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended, as forecast runs did. Change: - Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - Schedule, report and plugin runs now record how their jobs ended, not only forecast runs. Change: - Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - main moved 11 commits on, including #2689, which saves what a data generator returns through generator_results and renames the unchecked-sensor errors to GeneratorWritesUncheckedSensor. Change: - Keep the record_outcome_on_automation_run imports next to main's new ones in data/automations.py and services/reporting.py. - Keep the decorator on make_schedule where main dropped ScheduleWritesUncheckedSensor. - The report refusal test expects GeneratorWritesUncheckedSensor as the run's error type. - The failed schedule test breaks the save through save_generator_results, which make_schedule now calls instead of save_to_db. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Documentation build overview
38 files changed ·
|
…omations Builds the host digest on top of the run outcomes of schedule, report and plugin runs. Brings in current main as well. The only conflict was the CLI changelog, where both sides' entries are kept. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…ng digest Context: - #2670 has schedule, report and plugin runs record how their jobs ended, so their failed executions can now reach the host digest. Change: - Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error. - The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
…xecution Context: - Since #2670, every automation type records how its run's jobs ended. Change: - Remove the note from the monitoring section of the automations docs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue. Change: - Say execution state and job records are kept for every type of automation. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Context: - main moved 4 commits on, and #2647 changed the rq.job import in test_automations_fresh_db.py. Change: - Keep main's `from rq.job import Job, JobStatus` and this branch's `from sqlalchemy import select`. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
… them A schedule run now names its jobs after itself and records them before they are queued, so this branch no longer records them once they are queued, which would record each job twice. Its jobs report how they ended through the same decorator as before, as they are looked up by their RQ ID. A sequential schedule's device jobs are keyed by the sensor they schedule, rather than by their position. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <[email protected]>
…ts stored summary Every queue's exception handler now stores the type and message of a job's error in its meta (#2698), so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line. The meta is read afresh, as the run holds the job as it was when queued. A job without such a summary is still described by its traceback, without a type. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <[email protected]>
|
@BelhsanHmida, I took this PR over to get it past #2657, which recorded schedule runs differently. Could you review my changes?
I updated the description to match, marking the edited bullet. Full suite: 3182 passed, 4 xfailed. A possible follow-up: give report and plugin runs IDs derived from the run as well, so they too can record their jobs before queueing them (see #2694). |
There was a problem hiding this comment.
🔵 Needs a closer look
The change hinges on subtle cross-queue worker/transaction semantics and dispatch-time-vs-worker-time race handling (including Redis catch-up and session rollbacks) that warrant a human maintainer's final judgment despite no concrete defect being found.
No open issues 🎉
Review effort: Balanced
…-monitor-automations Prepares for merging #2670's squash commit, whose tree is identical to this tip. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Its content is already here from the branch's own lineage, so conflicts take this branch's side. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
Resolve conflicts with the squash-merged #2670 and with #2657: take main's naming of schedule jobs and its job-failure path, keep the release of waiting runs where a run ends, and use AUTOMATION_RUN_RETRIED_TYPES, which now includes scheduling. Signed-off-by: Mohamed Belhsan Hmida <[email protected]>
* data/services/automations: record how schedule, report and plugin runs end Context: - Only forecast runs followed their jobs, so schedule, report and plugin runs stayed at execution state 'pending' whatever became of their jobs. Change: - record_automation_run_jobs records the jobs such a run queued, then takes the outcome of any job that already finished or failed from Redis; a run with no jobs has succeeded. - The record_outcome_on_automation_run decorator records a job's start, success or failure on its run, found through the job's RQ ID, and wraps make_schedule, the sequential wrap-up job, run_report_job and execute_automation_job. - Plugin jobs carry automation_run_id in their trigger. - A schedule that fell back counts as a failed run, as the fallback job is not one of the run's recorded jobs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * data/tests: cover how schedule, report and plugin runs end Context: - Schedule, report and plugin runs now record how their jobs ended. Change: - Schedule runs, single and sequential, succeed once their jobs did and fail with the job's error; a job queued outside a run is not recorded. - A plugin run records its job's outcome, also when the job finished before the run recorded it. - Report run tests check the run's outcome: succeeded, nothing to do, and refused. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * docs: every automation run records how its jobs ended Context: - Schedule, report and plugin runs now record how their jobs ended, as forecast runs did. Change: - Say which jobs each type of run records, that a run without jobs has succeeded, and that a schedule which fell back counts as failed. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * changelog: append PR #2670 to the durable automation runs entry Context: - Schedule, report and plugin runs now record how their jobs ended, not only forecast runs. Change: - Append PR #2670 to the #2457 entry, and drop "for its forecast jobs", which no longer holds. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * data/models: record when an automation run was reported by monitoring Context: - #2519: a monitor command will email a digest of failed automation runs, and has to report each failure once, also when a monitoring run is missed. Change: - Add a nullable alerted_at column to AutomationRun, with a migration. - Add a partial index on the runs the monitor looks at (not reported yet, and failed, running or not fully dispatched), so that its hourly query does not scan the whole run history. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * utils/config: add FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER Context: - #2519: a run whose worker died stays "running" for good, so monitoring needs a threshold after which it counts as stuck. Change: - Add the setting, defaulting to six hours, well above any job timeout. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * data/services: find the automation runs that failed and were not reported yet Context: - #2519: monitoring reports runs whose execution failed, which are stuck, or whose dispatch failed for good. Change: - Add get_unalerted_failed_automation_runs, with one criterion for every automation type: only a forecast run with dispatch attempts to spare is left out, as it will be dispatched again, and so is a run that a runner still holds a live claim on. - Add group_failed_runs_by_automation, collapsing failures per automation with their counts, latest run and latest error. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * cli: add flexmeasures monitor automations Context: - #2519: automations fail silently unless someone watches their asset's Automations page. Change: - Email one digest of the failed runs to the monitoring recipients (or --recipient), collapsed per automation, and mark the runs as reported once it is sent. - Send nothing when nothing failed, and refuse to run without recipients, rather than marking runs as reported that nobody heard of. - --stuck-after-minutes overrides FLEXMEASURES_MONITOR_AUTOMATIONS_STUCK_AFTER. - Kept out of flexmeasures jobs run-automations, so a mail problem never holds up dispatching. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * cli/tests: test flexmeasures monitor automations Context: - #2519: cover the new monitor command. Change: - Drive forecast runs to a failed execution, a stuck execution and an exhausted dispatch through the real runner and worker, and check each is reported once, and not before it counts. - Check which runs count as failed for other types, the threshold's config and CLI override, collapsing per automation, and refusing to run without recipients. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * docs: document flexmeasures monitor automations Context: - #2519: new monitor command and setting. Change: - Describe what the command reports in the automations feature docs, and point to it from the host's error monitoring page. - List the command, document the stuck threshold setting, and add a CLI changelog entry. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * data/migrations: revise main's latest migration with the alerted_at migration Context: - Main gained migrations since this branch started, which left two Alembic heads. Change: - Make d4b7e9a1c362 revise f7c2e8b1a409, main's head, rather than c5e1a7b94d20. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * tests: report failed schedule, report and plugin runs in the monitoring digest Context: - #2670 has schedule, report and plugin runs record how their jobs ended, so their failed executions can now reach the host digest. Change: - Fail a schedule, a report and a plugin job on the worker, and check the digest names the automation, the failed execution and the job's error. - The schedule and plugin runs are marked dispatched, as a runner would, so that only their execution can make them count. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * docs: drop the note that only forecast runs are monitored for their execution Context: - Since #2670, every automation type records how its run's jobs ended. Change: - Remove the note from the monitoring section of the automations docs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * changelog: add PR #2671 to the automations, in detail list Context: - PR #2671 adds flexmeasures monitor automations. Change: - Add one bullet for it to the per-PR automations list. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * docs/api: run-stats records execution state for every automation type Context: - The v3.0-33 entry on run-stats said only forecast runs record their execution state, which PR #2670 makes untrue. Change: - Say execution state and job records are kept for every type of automation. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * Record the error of a job that failed before its run recorded it by its stored summary Every queue's exception handler now stores the type and message of a job's error in its meta (#2698), so a run catching up on a job that already failed records that type, rather than a stand-in type around the last traceback line. The meta is read afresh, as the run holds the job as it was when queued. A job without such a summary is still described by its traceback, without a type. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_0129WrXeJ5gia2pctFH93BqC Signed-off-by: F.N. Claessen <[email protected]> * data/migrations: revise main's latest migration with the alerted_at migration Context: - Main gained a merge migration since the last sync, which left two Alembic heads. Change: - Make d4b7e9a1c362 revise bf22b21d0990, main's head, rather than f7c2e8b1a409. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * utils/time_utils: move the exact duration formatting out of the monitor CLI Requested in review, so other code can reuse it. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * data/migrations: count runs that failed before the upgrade as reported Otherwise a host's first digest lists every run that ever failed. The upgrade marks the runs that the monitor would report now, by its rules at this revision, and leaves out runs that may still be retried or finish. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> * cli/monitor: link each automation in the digest to its info on the Automations page The digest is sent from the CLI, where FlexMeasures does not know its own address, so the new FLEXMEASURES_BASE_URL setting gives it. Without the setting, the digest carries no links, as before. The Automations page opens the tab and info panel of the automation named in ?automation=<id>, which show its latest run, its error and its jobs. Signed-off-by: Mohamed Belhsan Hmida <[email protected]> --------- Signed-off-by: Mohamed Belhsan Hmida <[email protected]> Signed-off-by: F.N. Claessen <[email protected]> Co-authored-by: F.N. Claessen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
Description
Schedule, report and plugin runs now record how their jobs ended, as forecast runs already did, so every run's
execution_statemoves tosucceededorfailedinstead of stayingpending.record_automation_run_jobsrecords the jobs a report or plugin run queued, then takes from Redis the outcome of any job a worker already finished before the record existed, with the error type and message the queue's exception handler stored (One failure summary in a job's meta for every queue #2698). A run that queued no jobs has succeeded.record_outcome_on_automation_rundecorator records a job's start, success or failure on its run, found through the job's RQ ID. It wrapsmake_schedule,cb_done_sequential_scheduling_job,run_report_jobandexecute_automation_job, so plugin types get it for free. Jobs queued outside a run, or never recorded by one, are left alone.automation_run_idin their trigger, as schedule and report jobs already did.features/automations.rstnow covers every type.documentation/changelog.rst(PR number appended to the Feat/2393 durable automation runs #2457 entry)Answers to the issue's questions
on_success/on_failurecallbacks to the fallback scheduler, which already uses them, and leaves the exception handlers untouched.running, and the digest flags a run still running after the stuck threshold.The issue proposed recording the intents before the jobs are enqueued. Schedule runs now do so, since #2657. Report and plugin runs get their job IDs from RQ, so their intents are recorded right after enqueueing, and the run catches up from Redis on any job that already ended. Giving those run-derived IDs as well is a possible follow-up.
How to test
On a toy battery account (
flexmeasures add toy-account --kind battery) with today's prices on sensor 7, and a sensor on asset 5 for the report (here sensor 10):Then open each automation's Info panel on the Automations page (or
GET /api/v3_0/assets/5/automations/<id>, underrun-stats):succeededsucceeded, with both its jobs (device-8,wrap-up)failed,UnknownPricesExceptionfailed,InfeasibleProblemExceptionsucceededsucceededwithout queueing a job, as there is nothing new to reportBefore this PR, all of these stayed
pending.flexmeasures jobs run-automation --automation <id>still runs a job as before, and records no run.Automated:
flexmeasures/data/tests/test_automations_fresh_db.py:test_a_schedule_run_records_how_its_jobs_ended,test_a_schedule_run_records_a_failed_job(single and sequential),test_a_schedule_job_queued_outside_a_run_is_not_recordedflexmeasures/data/tests/test_plugin_automations_fresh_db.py:test_a_plugin_run_records_how_its_job_ended,test_a_job_that_ended_before_its_run_recorded_it_is_caught_up_on,test_a_job_that_failed_before_its_run_recorded_it_is_caught_up_on_with_its_errorflexmeasures/cli/tests/test_automations.py: the report run tests now also check the run's outcome (succeeded, nothing to do, refused).Further Improvements
pending, so a stuck rule timed from the start of execution misses it. This happens when a worker dies before picking the job up, or in a narrow race: a job finishes before its run recorded it, and RQ has not yet marked it finished when the run checks Redis. Feat: flexmeasures monitor automations #2671 is to also count a run as stuck when it finished dispatching longer ago than the threshold and is stillpendingorrunning.runningin the run history. If that bothers people, the dispatcher could fail such runs, after asking Redis what became of their jobs.pending; no backfill.Related Items
Closes #2666.
Sign-off