Repository navigation
A failed estate Job stays failed until what names it changes #461
Description
What happened
The Vault policy job and the catalog method's Job are named by a digest of what they run with (#454, #459). On 2026-10-16 the first postgres-catalog-7b66ab65da64 failed, because the Vault policy job had not yet written its role. Once a fixed policy job had run, the catalog Job's inputs were unchanged, so it kept its name and stayed Failed. Flux's health check kept estate-vso-secrets not Ready, which held apps-delivery, apps-tribelt and apps-estate-dashboard, until the Job was deleted by hand. The hourly CronJob runs the same pod and succeeded, but the unit's health check reads the hash-named Job.
What should happen
A Job that failed for a reason outside its own inputs runs again without a person deleting it. Options: the unit does not health-check the one-shot Job and relies on the CronJob, the Job carries a backoffLimit and deadline long enough to outlast the policy job writing its role, or the catalog Job is ordered after the policy job.
Acceptance criteria
- A catalog Job that fails because its Vault role does not exist yet succeeds on its own once the role exists, with no manual delete.
- Both implementations and the oracle trees agree.
Activity
ExtraToast commentedon Oct 29, 2026
#473 (merged 2026-10-17, for #472) already fixes this, so I opened no PR. I recommend closing this issue as completed. I could not close it from here. #473 took the issue's second option:
restartPolicy: OnFailure, and the deadline moved onto the pod asactiveDeadlineSeconds, so it limits one attempt instead of the whole Job.backoffLimit: 2147483647and no deadline of its own. A catalog Job whose Vault role does not exist yet keeps retrying, using the Job controller's backoff, which tops out at 6 minutes. The first attempt after the policy job writes the role succeeds. The Job never reachesFailed, so its hash name can no longer leave it stuck.backoffLimit.Where it lives on
main:src/adapters/kubernetes/catalog.tsandsrc/adapters/vault-policy/job.ts, both usingUNTIL_SUCCEEDEDfromsrc/adapters/shared/placement.ts;spec/v1/examples/knowledge-platform/rendered/estate/vso-secrets/postgres-catalog.yamlpinsbackoffLimit,restartPolicyand the per-attempt deadline byte for byte;#applying-the-catalogand chapter 30'sjob.yamlrow;emf/matched when #473 merged and has been frozen atmde-task-3since then, so the second criterion stands as of that merge.The live proof is still open: a catalog Job that fails, then succeeds, with no manual delete. That needs someone watching the cluster.