factory: a pass that cannot see says so, instead of reading as a quiet night - #495
Open
JulienMartel wants to merge 3 commits into
Open
factory: a pass that cannot see says so, instead of reading as a quiet night#495JulienMartel wants to merge 3 commits into
JulienMartel wants to merge 3 commits into
Conversation
…t night The shift's product is a log somebody reads hours later instead of having watched it, so its worst failure is not a crash. It is a pass that sensed nothing, said nothing about that, and ended on output identical to a genuinely quiet night. Two paths did exactly that. `gh run list` failing left `conclusion` empty, and empty is also what a still-running job and a repo with no workflows look like, so the `case` fell through and that repo's main went unreported — including when it was red. Reporting a red main is the entire reason the block exists. `gh repo list` failing left the loop with nothing to iterate, so every repo went unchecked and the pass still printed `pass done: 0 merged`. Neither is hypothetical. Measured 2026-08-29 across three consecutive passes on one machine: four `connection reset by peer` and one `http2: client conn could not be established` against api.github.com, one of them mid-pass. Nothing was red underneath — checked by hand, every family repo green — which is the point: the shift could not have told me either way. `ci-unknown: <repo>` for the first, and a non-zero `pass ABORTED` for the second. `ci-unknown` is a log line rather than a trill card because one blip is noise and it is the foreman that decides whether a repeat is a story. This is the rule the budget block already states about itself, applied to the half that reports on other people's CI: degrade to a named unknown, never to an answer that happens to parse. `test/factory-shift.bats` is new and stubs `gh` to make those passes observable. Four of its six cases fail against the previous script; the other two are the controls that must not move — a green main still says nothing, and a red one still says CI-RED. The workflow now shellchecks all three factory scripts and runs both suites, which meant fixing two warnings that were already there: `notify done` read as a loop terminator (quoted), and factory-lease's third TSV field bound a name nothing used (`_`).
The first commit guarded the CI half and the org listing. An assurance pass against a clean context pointed out that the one failure with a record in tonight's log was in the MERGE half, which I had left alone. `factory-tier` exits 0 tier-1 · 3 refused-with-a-reason · anything else means `set -e` aborted inside it, usually on a `gh` call. `factory-shift` branched on success/failure, so "refused" and "could not be judged" both became `queued:` with whatever was on stderr in the reason column. The nightshift skill tells the foreman that queued rows need nothing, being the morning's by design — so a PR nobody managed to sense was filed as one nobody needs to look at. That is shift-20260829.log:96, verbatim: `queued: perch#120 … — Post "https://api.github.com/graphql": http2: client conn could not be established`. It is now `tier-unknown`. `gh pr list` ran in a process substitution, whose exit status is unreachable, so a failed listing gave zero iterations and a repo whose PRs were never read was indistinguishable from one with none open. That is where the `connection reset by peer` lines went in the two passes before it: straight to the terminal, never into the log, the repo silently skipped. Now `prs-unknown`, and it no longer takes that repo's CI check down with it. The dated measurement in the CI comment claimed counts the log does not support and attributed them to the block they did not come from. Removed rather than restated: the falsifiable evidence belongs in this message, where a reader asking why already is. Both unknown lines now carry the failing command's stderr, one line, truncated: telling a blip from a rate limit is the foreman's only judgement here and it had nothing to make it on. `pass ABORTED`'s card no longer claims "GitHub unreachable" for what may be a token or a rate limit. The tests stub `trill` now, and that is not tidiness. `setup()` prepends to PATH, so `notify`'s `command -v trill` found the real one and three cases reached a `notify fault` — running the suite drew real banners on this desktop. The stub also records its calls, so the notify policy is pinned: one unseeable repo must not card, a dead pass must. docs/factory.md stated the change instead of the contract, which AGENTS.md bans outright. It is now a table of what each line means and what the pass exits, and the nightshift skill — the only consumer of any of this — learns the four lines, that an unknown is never a fixer's cue, and that an unknown or an abort is noop:false rather than another quiet tick. Twelve cases now. Eight fail against main; three are the controls that must not move, and one pins a card main does not draw either.
CI reddened on `script/factory-tier:25`, a line this PR does not touch: apt's shellcheck emits SC2015 on `A && B || C` where 0.11.0 (the nix one I checked with) does not, even at -S style. Putting the factory scripts under the gate is what surfaced it, so it belongs here. The line is an if/then now, which reads better than the guard it replaces and needs no directive. Both steps print their tool's version first. The versions are not pinned and this is the second-order failure that costs the most time: a step reddening on a line nobody edited, with nothing in the log saying the checker moved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
factory-shiftruns unattended and its product is a log you read hours later, so silence in it is a claim — that something was looked at and was fine. Four steps could fail while making that claim on the shift's behalf:gh repo listpass done: 0 mergedpass ABORTEDgh pr list(process substitution, status unreachable)prs-unknown: <repo>, and its CI is still checkedfactory-tierreturning neither 0 nor 3queued:with stderr in the reason columntier-unknown: <repo>#<n>gh run listconclusion→casefalls through → thatmainunreported, including when redci-unknown: <repo>The two
unknownlines carry the failing command's stderr, one line, truncated — telling a blip from a rate limit is the foreman's only judgement here and it had nothing to make it on.tier-unknownis the expensive one..agents/skills/nightshift/SKILL.mdtells the foreman that queued rows need nothing, they are the morning's by design — so a PR nobody managed to sense was being filed as one nobody needs to look at.Why
Not hypothetical, and not theorised from the code. Running the shift tonight, three consecutive passes hit
api.github.com:~/.cache/hausfold-factory/shift-20260829.log:96—queued: perch#120 … — Post "https://api.github.com/graphql": http2: client conn could not be established. A PR filed as judged-and-refused when nothing had judged it.connection reset by peerlines that reached the terminal and never the log, fromgh pr list. That repo was skipped; the pass reported success.Nothing was red underneath — I checked all fourteen repos by hand. That is the point: the shift could not have told me either way, and the next person to run it at 3am will not be checking by hand.
Verify
202/202, shellcheck clean. The factory scripts had neither before; the workflow now covers all three, which meant fixing two warnings already sitting there (
notify doneparsed as a loop terminator,factory-lease's third TSV field binding an unused name — both no-ops at runtime).The tests stub
gh,trillandfactory-tier. Eight of the twelve fail againstmain— I verified that by checking outmain's script under the new suite, because a test that passes before and after guards nothing. Three of the four that pass are declared controls: a green main still says nothing, a red one still saysCI-RED, a judged refusal is stillqueuedwith its reason.Also worth running once on a real org, which I did:
./script/factory-shift --dry-runprints exactly what it printed before, no false unknowns.Watch out
trillwas a bug fix, not tidiness.setup()prepends to PATH, sonotify'scommand -v trillfound the real binary and three cases reach anotify fault. Running the earlier version of this suite drew real--source factorybanners on this desktop. If you have atrillon PATH and pull an older revision of this branch, it will do it again.factory-shiftcan now exit non-zero. Only onpass ABORTED. Its one caller is the/nightshiftskill, updated here; nothing else invokes it andbenchhas no factory verb.maintoo (one unseeable repo does not card) — main does not card there either. It is a forward guard on the policy, not evidence of this change.A && B || Cwhere 0.11.0 does not, even at-S style, which is how this branch went green locally and red in CI on a line it never touched (script/factory-tier:25, now anif/then). Both steps print their version first now, so the next divergence is one line into the log instead of a bisect. Pinning them is a separate decision and yours.ci-unknownon every pass forever —gh run listexits 1 on its 404 rather than returning empty. None of hausfold's are today (verified: all fourteen exit 0, four of them with empty output). Noted in the code; the answer when it happens is a 404 arm, not a wider||.🤖 Generated with Claude Code
https://claude.ai/code/session_01DMJ3jfaov6Q2wKZaTUqLAx