Skip to content

feat: let a Project declare a longer health-check timeout for its unit #533

Open

Description

@ExtraToast

Problem

A Project's unit renders wait: true and no spec.timeout, so Flux waits for health at most until the unit's interval (10m, chapter 55's pin-source table) and then marks the revision HealthCheckFailed. A Process whose first start after an upgrade is legitimately slow fails its delivery even though the rollout lands.

It happened on 2026-10-21. jellyfin 10.11.11's revision ended HealthCheckFailed at 09:21:43Z after 9m30.6s, waiting on Deployment/jellyfin-system/jellyfin. The likely cause is a slow first start for Jellyfin 10.11's database migration, but that is unverified because the pod is gone. The next reconcile at 09:31:43Z passed in 169ms, so the version landed, but the delivery recorded a failure.

The node was also mid-recovery that morning (the jellyfin pod could not start until the GPU node's CDI spec was restored), so this occurrence may not be a pure slow start. The gap holds either way: nothing lets a Project say that its first start takes longer than the interval.

What to build

Let a Project declare a longer health-check timeout, which deploy-kit renders into its unit's spec.timeout. When nothing is declared, the default stays the interval, as today.

The shape is open:

  • Per Application or per Process. A unit is one Project, and Flux has one timeout per Kustomization, so the rendered value would be the maximum over the unit's Processes.
  • A bound. Should the Platform document cap it, so a typo cannot hold a unit for hours?

Acceptance criteria

  • Chapter 55 (and the authoring chapter that gains the field) states the field, its default and how it reaches spec.timeout.
  • Both implementations render it, with the oracle files updated and a worked example that declares it.
  • A unit that declares nothing renders exactly as today.

Requested by the estate-dashboard redesign session (workspace-ca).

Activity

  1. added
    type: featureNew user-facing or operator-facing capability.
    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.
    status: triageNeeds initial review and classification.
    on Oct 21, 2026
  2. ExtraToast commented on Oct 27, 2026

    @ExtraToast
    ContributorAuthor

    Resume here

    • State: a decision for the maintainer: per Application or per Process, and whether the Platform document caps it.
    • Next step: the maintainer picks the shape; then a spec change in deploy-kit.
    • Blocked by: the maintainer.
    • Decisions: never pin a stateless Process for locality.
    • Gotchas: this is also the fix for estate-dashboard's single readiness failures. Measured 2026-10-26 from Prometheus (prober_probe_total, 24 h): its pod on enschede-pi-2 failed readiness 116 times, on pi-3 4 times, and on frankfurt (beside the Postgres) none in 1.5 h. That is latency, not a fault.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.status: triageNeeds initial review and classification.type: featureNew user-facing or operator-facing capability.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions