Skip to content

Filter status page jobs by type and re-run failed jobs #2833

Open

Description

@BelhsanHmida

Context

The Jobs tab of an asset's status page lists every job of the asset, and by default of the assets below it.
Two things make it hard to use once a site has many jobs:

  • Forecasts are hard to find among the report jobs.
    The table can be sorted by queue, but not narrowed down to one.
  • After spotting a failed job, there is no way to run it again from this page.
    Admins can requeue it in /tasks, but nobody else can reach that page.

Proposal

1. Filter jobs by type

  • Add an optional queue parameter to GET /api/v3_0/assets/<id>/jobs, accepting forecasting, scheduling or reporting.
    Without it, the response stays as it is.
  • num-records stays the full count, and filtered-records becomes the filtered count.
  • Add a dropdown to the Jobs tab, next to the sub-assets switch, which reloads the table from page 1.

This can ship on its own.

2. Re-run a failed job

  • Add POST /api/v3_0/jobs/<uuid>/rerun, which calls job.requeue().
    It returns 202 when the job is queued again, and 409 when the job is not failed.
  • It uses @limit_triggers() and writes an audit log entry.
  • Add can_rerun to each listed job, so that the Re-run button only shows when it would work.
  • Permissions follow the type of job:
    trigger-schedules for schedules, trigger-forecasts on the sensor for forecasts, and trigger-reports on the asset for reports.
    Reports also repeat the input and output checks of trigger_report.
    A job without a linked asset or sensor is refused.

job.requeue() only works on failed jobs.
It keeps the job ID, so the job stays indexed under its assets and sensors, and its dependents stay attached.

Open questions

  1. Only failed jobs, or finished ones too?
    I'd start with failed only.
    For a finished job we can't just requeue, we'd have to create a new job with a new ID and add it to the job map for each sensor.
    We'd also have to drop its automation_run_id, otherwise it changes the record of the original run.
    Finished forecasts have another problem: the belief time is pinned, so if the values differ we'd probably get a UniqueViolation.
  2. **For schedules **
    The problem is that a re-run keeps the old start, but if belief_time isn't set it becomes now.
    So if a schedule fails at 10:00 and you re-run it at 16:00, it becomes the newest belief and can override the 16:00 plan. if that's okay we can add it
  3. What happens to a failed automation run when the re-run works?
    We need to decide the same thing for feat: Retry an automation's failed jobs before its next run #2755, so let's pick one answer for both.
    I think it should become succeeded and the old error should go away.
    Right now it does become succeeded, but execution_completed_at and last_error_* stay as they were.
    You can already get this today by requeueing from /tasks.
    Feat: flexmeasures monitor automations #2671 and Feat: automation dependencies #2706 depend on this too.

Out of scope

Plugin automation jobs are indexed as <queue>:asset:<id>, while the status page only reads scheduling:asset.
So plugin jobs on other queues do not show up on this page at all, and neither the filter nor Re-run reaches them.
That deserves its own fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions