Skip to content

feat(nightly): refuse to run the harness against a stale sei-chain image - #502

Merged
bdchatham merged 1 commit into
mainfrom
feat/harness-image-freshness-gate
Aug 5, 2026
Merged

feat(nightly): refuse to run the harness against a stale sei-chain image#502
bdchatham merged 1 commit into
mainfrom
feat/harness-image-freshness-gate

Conversation

@bdchatham

Copy link
Copy Markdown
Collaborator

Closes the detection gap held open on sei-protocol/platform#1489: once Flux pushes its image-bump branch the ImageUpdateAutomation reports Ready, so every stage after that — the bump PR opening, its checks, the merge — can fail while the automation still looks healthy. The manifests keep the old tag, the suites pass, and nothing else in the fleet distinguishes that from a good run.

Concretely, all of these currently leave main stale and the 08:00 harness green:

  • Shellcheck red on the bump branch
  • gh pr create denied by the Actions PR-creation setting
  • the bump workflow's path or content guard rejecting
  • mergeStateStatus BLOCKED / BEHIND / DIRTY
  • a run dropped by the one-deep concurrency queue

The check

TestMain parses the date out of each nightly tag before any suite starts and exits non-zero if one is more than 36h behind.

filterTags.pattern: '^nightly-(?P<date>[0-9]{8})-[0-9a-f]{7}$'

The date is structurally guaranteed by the ImagePolicy that selects the tag, so this needs no registry call, no cluster read and no added RBAC — and it holds whichever upstream stage broke, the image build or the bump.

Failing here rather than mid-run means the existing NightlyHarnessSuiteFailed alert covers the whole class — an alert that already exists and already pages — and the 2.5-hour run isn't spent on a build nobody advanced.

Scope

Checked SEID_IMAGE, SEID_IMAGE_MOCK, SEID_IMAGE_CHAOS
Not checked SEID_UPGRADE_FROM_IMAGE / _TO_IMAGE — pinned commits, no date, self-excluded by the tag regex
Local runs inert without SEI_NODE_CLUSTER, where every suite skips anyway
Budget 36h, override with SEID_IMAGE_MAX_AGE_HOURS

36h tolerates one skipped upstream build without tolerating a stopped pipeline: the bump lands ~01:40 UTC and the suite runs at 08:00, so a healthy image is hours old.

Coupling to name

The tag alternation mirrors the three filterTags patterns in the platform repo's clusters/harbor/flux-system/image-automation.yaml. A tag shape added there needs adding here, or that image silently stops being freshness-checked. Called out in the comment on nightlyTagDate.

Verification

TestStaleSeidImages covers eight cases — same-day passes, five-day-old refused, pinned commit tag not date-checked, unset not checked, both mock flavours checked, budget overridable, local run exempt. Runs without a cluster. go vet -tags integration and gofmt clean.

It would fire right now: deployed is nightly-20260731-2d2628f, and the ImagePolicy currently resolves nightly-20260805-0d9c675 — five days.

🤖 Generated with Claude Code

The suites read SEID_IMAGE, SEID_IMAGE_MOCK and SEID_IMAGE_CHAOS from env, and
Flux's image automation advances those values in git. Once Flux pushes its bump
branch it reports Ready, so every stage after that — the bump PR opening, its
checks, the merge — can fail while the automation still looks healthy. The
manifests keep the old tag, the suites pass, and nothing else in the fleet
distinguishes that from a good run.

TestMain now parses the date out of each nightly tag before any suite starts and
exits non-zero if one is more than 36h behind. The date is structurally
guaranteed by the ImagePolicy that selects the tag, so this needs no registry
call, no cluster read and no added RBAC, and it holds whichever upstream stage
broke — the image build or the bump.

Failing here rather than mid-run means the existing NightlyHarnessSuiteFailed
alert covers the whole class, and the two-and-a-half-hour run is not spent on a
build nobody advanced.

Scope: the check skips any tag that is not nightly-shaped, so the upgrade suites'
pinned commit images are untouched, and it is inert without SEI_NODE_CLUSTER so
local runs are unaffected. SEID_IMAGE_MAX_AGE_HOURS overrides the budget.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
@cursor

cursor Bot commented Aug 5, 2026

Copy link
Copy Markdown

PR Summary

Low Risk
Harness-only startup check with no production path changes; mis-tuned regex or age budget could false-fail or miss stale tags until platform tag shapes stay in sync.

Overview
Adds a pre-suite guard in TestMain that fails the integration harness before any nightly suite runs when Flux-managed image env vars point at an outdated nightly tag.

For cluster runs (SEI_NODE_CLUSTER set), it parses the embedded date from SEID_IMAGE, SEID_IMAGE_MOCK, and SEID_IMAGE_CHAOS tags (regex aligned with platform Flux filterTags) and exits with a diagnostic if any matched tag is older than 36 hours (override via SEID_IMAGE_MAX_AGE_HOURS). Pinned commit tags, unset vars, and local runs without a cluster are skipped so upgrade suites and dev workflows are unaffected.

TestStaleSeidImages locks in pass/refuse behavior, mock/chaos tag shapes, budget override, and the local exemption.

Reviewed by Cursor Bugbot for commit b5777de. Bugbot is set up for automated code reviews on this repo. Configure here.

@bdchatham
bdchatham merged commit 2d670ad into main Aug 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant