feat(nightly): refuse to run the harness against a stale sei-chain image - #502
Conversation
The suites read SEID_IMAGE, SEID_IMAGE_MOCK and SEID_IMAGE_CHAOS from env, and Flux's image automation advances those values in git. Once Flux pushes its bump branch it reports Ready, so every stage after that — the bump PR opening, its checks, the merge — can fail while the automation still looks healthy. The manifests keep the old tag, the suites pass, and nothing else in the fleet distinguishes that from a good run. TestMain now parses the date out of each nightly tag before any suite starts and exits non-zero if one is more than 36h behind. The date is structurally guaranteed by the ImagePolicy that selects the tag, so this needs no registry call, no cluster read and no added RBAC, and it holds whichever upstream stage broke — the image build or the bump. Failing here rather than mid-run means the existing NightlyHarnessSuiteFailed alert covers the whole class, and the two-and-a-half-hour run is not spent on a build nobody advanced. Scope: the check skips any tag that is not nightly-shaped, so the upgrade suites' pinned commit images are untouched, and it is inert without SEI_NODE_CLUSTER so local runs are unaffected. SEID_IMAGE_MAX_AGE_HOURS overrides the budget. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR SummaryLow Risk Overview For cluster runs (
Reviewed by Cursor Bugbot for commit b5777de. Bugbot is set up for automated code reviews on this repo. Configure here. |
Closes the detection gap held open on sei-protocol/platform#1489: once Flux pushes its image-bump branch the
ImageUpdateAutomationreportsReady, so every stage after that — the bump PR opening, its checks, the merge — can fail while the automation still looks healthy. The manifests keep the old tag, the suites pass, and nothing else in the fleet distinguishes that from a good run.Concretely, all of these currently leave
mainstale and the 08:00 harness green:gh pr createdenied by the Actions PR-creation settingmergeStateStatusBLOCKED/BEHIND/DIRTYThe check
TestMainparses the date out of each nightly tag before any suite starts and exits non-zero if one is more than 36h behind.The date is structurally guaranteed by the ImagePolicy that selects the tag, so this needs no registry call, no cluster read and no added RBAC — and it holds whichever upstream stage broke, the image build or the bump.
Failing here rather than mid-run means the existing
NightlyHarnessSuiteFailedalert covers the whole class — an alert that already exists and already pages — and the 2.5-hour run isn't spent on a build nobody advanced.Scope
SEID_IMAGE,SEID_IMAGE_MOCK,SEID_IMAGE_CHAOSSEID_UPGRADE_FROM_IMAGE/_TO_IMAGE— pinned commits, no date, self-excluded by the tag regexSEI_NODE_CLUSTER, where every suite skips anywaySEID_IMAGE_MAX_AGE_HOURS36h tolerates one skipped upstream build without tolerating a stopped pipeline: the bump lands ~01:40 UTC and the suite runs at 08:00, so a healthy image is hours old.
Coupling to name
The tag alternation mirrors the three
filterTagspatterns in the platform repo'sclusters/harbor/flux-system/image-automation.yaml. A tag shape added there needs adding here, or that image silently stops being freshness-checked. Called out in the comment onnightlyTagDate.Verification
TestStaleSeidImagescovers eight cases — same-day passes, five-day-old refused, pinned commit tag not date-checked, unset not checked, both mock flavours checked, budget overridable, local run exempt. Runs without a cluster.go vet -tags integrationandgofmtclean.It would fire right now: deployed is
nightly-20260731-2d2628f, and the ImagePolicy currently resolvesnightly-20260805-0d9c675— five days.🤖 Generated with Claude Code