[auto] update translations - #682
Closed
github-actions[bot] wants to merge 1 commit into
Closed
Conversation
chhhee10
added a commit
that referenced
this pull request
Aug 12, 2026
`isCached` is a pure function of the ENGLISH source hash. It records that a page
was translated once — never that the translation is on disk now — and those two
facts came apart in production.
Translations land on an auto-translate PR branch. While that branch sits
unmerged, `main` lacks the files and the cache still reports them done, so they
are never regenerated. Meanwhile `--update-nav` reads the ENGLISH tree and emits
nav entries for them, and `mintlify validate` fails on entries pointing at files
that are not there. Verified on the live repo: `docs/cli/{update,migrate}.mdx`
exist on main, `docs/zh/cli/` has neither, and PR #682 carrying them is still
open — 28 missing files across 14 locales.
That is non-convergent, which is what makes it worth a code change rather than a
merge. A cache HIT writes nothing, so validation fails and the cache is never
saved; a full cache MISS spends 120 runner-minutes and goes green. The pipeline
had no path to a cheap success while #682 stayed open, and Aug 12 is exactly
that: all 14 languages finished in ~20 seconds each, and consolidate failed.
Statting the output makes the cache self-healing against any "translated once,
never landed" gap, whatever opened it — an unmerged PR, a hand-reverted file, a
locale added to the matrix after the fact.
Guarded at all four sites rather than one. `cli.ts` is load-bearing: the batch
path sorts pages into cached/uncached itself and never calls `translateMdxPage`
for a cached one, so guarding only the translator would have fixed nothing. The
two single-page paths are guarded too, or they and the batch path disagree about
what "cached" means.
The tests pin both directions — a missing output re-translates, a present one is
still skipped. The second matters as much as the first: without it a later
refactor could satisfy this commit by making the guard a cache bypass, and every
run would be a full re-translation with the suite still green. Confirmed the
first test fails on the pre-fix code rather than assuming it would.
NOTE: this changes translation OUTPUT, not just caching. The first run after it
lands re-sends those 28 pages to the model, so the text will not be byte-identical
to what sits on #682 — expect a noisy diff there once. ~10 minutes, once.
Co-Authored-By: Claude Opus 5 <[email protected]>
chhhee10
added a commit
that referenced
this pull request
Aug 12, 2026
`isCached` is a pure function of the ENGLISH source hash. It records that a page
was translated once — never that the translation is on disk now — and those two
facts came apart in production.
Translations land on an auto-translate PR branch. While that branch sits
unmerged, `main` lacks the files and the cache still reports them done, so they
are never regenerated. Meanwhile `--update-nav` reads the ENGLISH tree and emits
nav entries for them, and `mintlify validate` fails on entries pointing at files
that are not there. Verified on the live repo: `docs/cli/{update,migrate}.mdx`
exist on main, `docs/zh/cli/` has neither, and PR #682 carrying them is still
open — 28 missing files across 14 locales.
That is non-convergent, which is what makes it worth a code change rather than a
merge. A cache HIT writes nothing, so validation fails and the cache is never
saved; a full cache MISS spends 120 runner-minutes and goes green. The pipeline
had no path to a cheap success while #682 stayed open, and Aug 12 is exactly
that: all 14 languages finished in ~20 seconds each, and consolidate failed.
Statting the output makes the cache self-healing against any "translated once,
never landed" gap, whatever opened it — an unmerged PR, a hand-reverted file, a
locale added to the matrix after the fact.
Guarded at all four sites rather than one. `cli.ts` is load-bearing: the batch
path sorts pages into cached/uncached itself and never calls `translateMdxPage`
for a cached one, so guarding only the translator would have fixed nothing. The
two single-page paths are guarded too, or they and the batch path disagree about
what "cached" means.
The tests pin both directions — a missing output re-translates, a present one is
still skipped. The second matters as much as the first: without it a later
refactor could satisfy this commit by making the guard a cache bypass, and every
run would be a full re-translation with the suite still green. Confirmed the
first test fails on the pre-fix code rather than assuming it would.
NOTE: this changes translation OUTPUT, not just caching. The first run after it
lands re-sends those 28 pages to the model, so the text will not be byte-identical
to what sits on #682 — expect a noisy diff there once. ~10 minutes, once.
Co-Authored-By: Claude Opus 5 <[email protected]>
Contributor
|
Closing this nightly translation dump. At 678 files it is the symptom rather than the change — #685 ("Stop the nightly translation re-translating everything, most days") fixes the cause, so the next run should produce a diff worth reading. Nothing is lost by closing: the job regenerates this PR on its next scheduled run. |
NiveditJain
pushed a commit
that referenced
this pull request
Aug 14, 2026
…it onto one cron box (#694) * canary: move the daily integration suite to a local box and probe the daemon path Daily runs leave GH Actions (runner minutes were the entire cost; the LLM spend is identical either way) for a local canary box driven by a systemd user timer in the retired cron's 06:17 UTC slot. integration-suite/local/ ships the box side: run-local.sh (checkout CANARY_REF → stable leg → beta leg, flock-serialized, with a crash-guard Slack note for a leg that dies BEFORE reporting — the replacement for GHA's red-job email), install.sh (installed copy OUTSIDE the clone the wrapper hard-resets, systemd units, secrets template), and the service/timer units. The workflow keeps workflow_dispatch as the cloud fallback and loses its cron. The stable leg now probes the daemon-configured path (CANARY_DAEMON=1) — the configuration `failproofai config` gives users going forward: ci-entrypoint cross-compiles failproofaid in rust:1-bookworm (glibc-matched to the node:22-bookworm-slim sandbox; a host build can link newer symbols and fail to load inside it), run.sh bind-mounts it into the probe container, and probe-cli.sh sets the daemon.configured fail-closed marker through the real fp-config updateConfig path — shell-appending TOML could produce a duplicate [daemon] table, which parses as NOT configured and silently falls back to in-process. The daemon restarts per probe, not per CLI: the wire protocol forwards {hookEvent, cli, stdin, cwd} and never env, so the warm worker's FAILPROOFAI_HOOK_LOG_FILE is fixed at daemon start — one daemon across both probes would share one oracle dir, and probe A's incidental read-denies would satisfy probe B's grep (false PASS). A dead daemon cannot false-PASS either: its deny is shaped by the synthetic failproofai/daemon-unreachable policy, which the probes' greps never match — those probes go INCONCLUSIVE and re-probe until the daemon path recovers. Verified without LLM or secrets in the real sandbox image: live daemon → canary-bash deny through the socket → warm worker writes the per-probe oracle; killed daemon → fail-closed deny logged as daemon-unreachable, matching neither probe grep; marker cleared → in-process evaluation restored. Tripwires in __tests__/integration-suite/local-runner.test.ts pin the workflow staying cron-free, the unit↔installer paths, the secrets-template↔workflow-env parity, the per-probe daemon restarts, the marker hygiene, and the fail-closed/oracle non-overlap (both sides extracted from the real sources). Co-Authored-By: Claude Fable 5 <[email protected]> * canary: one container, one cron line — repackage the box runner for zero-touch hosts The box story shrinks to Docker + one cron line + one env file: the systemd units, install.sh and host-toolchain requirements are gone. A self-contained runner image (local/Dockerfile.runner — node+bun+git+docker CLIENT) drives the HOST's Docker through the mounted socket, so the sandbox image, the per-channel volumes and every probe container are exactly the ones CI runs, as siblings. Two decisions carry the design: - Path parity. The one work dir is mounted at an IDENTICAL path inside and out (-v "$HOME/fp-canary:$HOME/fp-canary") because paths under it serve both as in-container file paths and as sibling-container -v sources, which the host daemon resolves against the host filesystem. The entrypoint auto-detects the parity mount from its own container's mount table and names the exact flag to add when it is missing. runner-daily.sh pins the daemon build's cargo cache under the work dir — the only harness default rooted outside it ($HOME), where the rust sibling's mount would silently create an empty root-owned host dir and cache nothing. - A thin baked entrypoint, everything else from the checkout. The image carries only runner-entrypoint.sh (preflight, work-dir detection, host-side flock so overlapping cron fires share one lock across containers, clone/ fetch/checkout of $CANARY_REF, Slack crash-note for the checkout phase); it then execs integration-suite/local/runner-daily.sh FROM THE CHECKOUT. Harness changes reach the box through git — nobody rebuilds the boss's image for a leg tweak. runner-daily.sh keeps the leg contract from the systemd iteration verbatim: stable leg daemon-configured (CANARY_DAEMON=1) then beta in-process, per-leg 90-min timeout, crash-guard keyed on the absence of run.sh's own posted-to-Slack line, 14-day log prune. secrets.env.example documents every variable the GHA Environment supplied, in docker --env-file's literal KEY=value format. Tripwires updated in local-runner.test.ts: the image must never bake the daily driver, the crash-guard grep must match run.sh's actual wording, the example must offer every secret-fed env var the workflow maps and must contain no shell expansion on value lines. Co-Authored-By: Claude Fable 5 <[email protected]> * canary: port the fail-closed leg and its live-test lessons from the daemon test session Ports the daemon-leg findings from the parallel host-run session that drove all three legs against 10 real, locally-installed CLIs (2026-08-07): the daemon does not regress enforcement on any CLI, denies land in 2-3ms warm versus 7-8ms cold — and the fail-closed pass surfaced an availability defect (factory fired 202 denied hook calls and antigravity 1,002, retrying a deny that can never succeed until the harness killed them at ten minutes) that only a fail-closed leg keeps visible. CANARY_DAEMON_DEAD=1 is that leg: configure the machine for the daemon exactly as CANARY_DAEMON=1 does, then never start it. Every CLI must DENY; the benign probe command executing anyway means the machine believed it was fail-closed and was not. The deny is scored through the existing daemon-unreachable detector, which also now breaks the probe retry loops early in live-daemon mode (a dead daemon denies everything — further LLM attempts can only reproduce the same deny) and prints a triage note so a mid-probe daemon death reads as DAEMON FAILED CLOSED instead of a quiet INCONCLUSIVE. Two hazards closed on the way in: - The DEAD leg gets its own state lane ($STATE.dead). Its PASS means "denied while dead" — recorded in the enforcement gate it would skip the next REAL probe of the same (CLI, failproofai) pair as already-green. - The daemon.configured marker is now cleared before wire() in EVERY mode and set only after it. wire() runs vendor CLIs whose hooks route through the marker (openclaw onboard), and a marker with no daemon up yet — set too early today, or surviving from yesterday in the persistent volume — would fail-close the wiring itself. Also carried from that session's debugging: the SUN_LEN (108-byte) Unix socket path cap is documented on the socket-path choice. All of it pinned in __tests__/integration-suite/local-runner.test.ts (49 tests). Co-Authored-By: Claude Fable 5 <[email protected]> * chore: fill in the canary PR number in the changelog (#656) Co-Authored-By: Claude Fable 5 <[email protected]> * Make the canary box a one-command install Setting the box up was four commands. Three of them have a failure mode that is silent for a full day, which is the wrong property for the thing whose whole job is to notice silent failures: - the work dir mounted at a different path inside the container than out, so the sibling-container `-v` sources resolve against the host to nothing; - `CANARY_REF` left at the shipped `origin/failproofaid`, a branch that merged in #632 — the box would test a frozen tree forever and never say so; - a filled-in env file with no Slack webhook: a run that works perfectly and reports nowhere, which is worse than no canary because it looks like cover. `install.sh` refuses each at install time, in front of a person, rather than at 06:17 tomorrow in front of nobody. The webhook is required for exactly that reason and not because the run needs it. It builds the image straight from the git URL — Docker takes `<repo>#<ref>:<subdir>` as a build context — so the box never clones anything. The runner re-clones the repo itself on every run, so a checkout here would only go stale. The cron line is rewritten, not appended: it carries a `# failproofai-canary` marker and a re-install strips any previous line first, so running the installer twice upgrades the schedule instead of scheduling two jobs. The marker is a comment rather than a match on the command, because the command changes. The stale `CANARY_REF` default is fixed in `secrets.env.example` too. Catching it in the installer only would leave the wrong value shipping, with a guard as the sole thing standing between it and a year of green runs against a dead ref. `--dry-run` distinguishes what was CHECKED from what would be CHANGED. The preflight really does run in a dry run, so it keeps its ✓; the mutations print "would". A script that reports success for work it did not do is the same defect class this canary exists to find, and it would be a poor advertisement. Co-Authored-By: Claude Opus 5 <[email protected]> * Stop every canary leg building the dashboard it says it skips The build step announces "dist/index.js + dist/cli.mjs — no dashboard" and then builds exactly those two. But the `bun install --frozen-lockfile` above it fires the package `prepare` hook, which is `bun run build` — the FULL build, ending in `bun --bun next build`. So each leg compiled the entire Next.js application first, then built the two artifacts it actually wanted. Found by running the box end to end rather than reading it: the run log shows `Creating an optimized production build` and `Generating static pages (3/3)` underneath a step whose own text says it does not do that. `translate-docs.yml` already carries this guard, with the same reasoning written next to it — the trap is the hook, and every entry point that installs for tooling has to opt out of it individually. Costs two full Next builds a day here (stable + beta), on a box whose entire reason for existing is that runner time was too expensive to keep buying. Co-Authored-By: Claude Opus 5 <[email protected]> * Say that the work dir is root-owned before someone finds out The runner is root inside the container, so everything it creates under the work dir — the clone, logs/, state/, the cargo cache — is root-owned on the host. Only secrets.env, written by the installer, belongs to the user. That is harmless: the next run is root too, and nothing in the pipeline cares. But the first person to `tail` a log or `rm -rf` the clone gets a permission error with no explanation, on a box they were told needs nothing but Docker and a cron line. Found the same way — by doing it. Documented in both places someone would look, with the sudo form of the command they were about to run. Co-Authored-By: Claude Opus 5 <[email protected]> * Stop the cargo cache evicting everything else in the store `rust-quality` cached `~/.cargo` AND `target/` under a combined `actions/cache@v6`, which writes a ref-scoped copy from every branch that misses the exact key. Because the entry carries build output, each copy is 1.5-2.3 GiB, and five were live at once — PR refs 677, 679, 680, 681 and main — putting the repository at 11.56 GiB against GitHub's 10 GiB cap and therefore permanently in LRU eviction. The thing being evicted was not another cargo build. It was the 13 KB doc translation cache, read once every 24 hours by the nightly `translate-docs` run and so always the least-recently-used entry in the store. Losing it re-translated 48 pages into 14 languages the next morning: ~125 runner-minutes and a full LLM pass per language, against a 4-minute baseline when it survives. Six consecutive days of that, Aug 6-11, cost ~750 runner-minutes and six full-corpus passes through the gateway. Restore on every run, save only on a push to main — the split `build-daemon.yml` already uses, whose comment gives the other reason to want it (a PR branch can otherwise write a poisoned `target/` that a later release run restores straight into a published binary). `cache-hit != 'true'` keeps a run that changed nothing from re-uploading 2 GiB. What a PR gives up: one whose `Cargo.lock` moved rebuilds from a stale-but-close main cache. That is already what `restore-keys` hands it today. Co-Authored-By: Claude Opus 5 <[email protected]> * Save the translation cache where the work was proven, not at the end The only save sat in `consolidate`, downstream of both the matrix gate (`if: needs.translate.result == 'success'`) and `mintlify validate`. So the day's cache was contingent on fourteen languages and a nav check all succeeding: Aug 6 discarded ~110 minutes of completed translation because one `ko` page failed validation, and Aug 12 discarded a full run because consolidate's validation failed. In both cases every language had finished its work and uploaded its fragment; the cache was thrown away anyway. Each fragment is already authoritative for its own language, so nothing has to be merged before it can be stored. Each language now saves its own, in the job that produced it, immediately after the step that proved it good. Consolidate's merged save stays as the cross-language fallback. The restore key changes for a related reason. It read `translation-cache-${{ hashFiles('scripts/translate-docs/.translation-cache.json') }}`, which ALWAYS evaluated to the bare literal `translation-cache-`: the file is gitignored, so it is absent at checkout and `hashFiles` returns "" for a path that matches nothing. Every restore that ever worked was a `restore-keys` prefix match. That is not a bug on its own — but it means a total miss and a hit are indistinguishable, so the expensive case was silent. It is now a per-language key with the merged entry as fallback, and a miss emits a `::warning` naming what it is about to cost. Artifact retention 1 → 7 days, so a run that dies mid-pipeline leaves a human a recovery path rather than expiring overnight. Co-Authored-By: Claude Opus 5 <[email protected]> * Treat a cached translation whose file is missing as a miss `isCached` is a pure function of the ENGLISH source hash. It records that a page was translated once — never that the translation is on disk now — and those two facts came apart in production. Translations land on an auto-translate PR branch. While that branch sits unmerged, `main` lacks the files and the cache still reports them done, so they are never regenerated. Meanwhile `--update-nav` reads the ENGLISH tree and emits nav entries for them, and `mintlify validate` fails on entries pointing at files that are not there. Verified on the live repo: `docs/cli/{update,migrate}.mdx` exist on main, `docs/zh/cli/` has neither, and PR #682 carrying them is still open — 28 missing files across 14 locales. That is non-convergent, which is what makes it worth a code change rather than a merge. A cache HIT writes nothing, so validation fails and the cache is never saved; a full cache MISS spends 120 runner-minutes and goes green. The pipeline had no path to a cheap success while #682 stayed open, and Aug 12 is exactly that: all 14 languages finished in ~20 seconds each, and consolidate failed. Statting the output makes the cache self-healing against any "translated once, never landed" gap, whatever opened it — an unmerged PR, a hand-reverted file, a locale added to the matrix after the fact. Guarded at all four sites rather than one. `cli.ts` is load-bearing: the batch path sorts pages into cached/uncached itself and never calls `translateMdxPage` for a cached one, so guarding only the translator would have fixed nothing. The two single-page paths are guarded too, or they and the batch path disagree about what "cached" means. The tests pin both directions — a missing output re-translates, a present one is still skipped. The second matters as much as the first: without it a later refactor could satisfy this commit by making the guard a cache bypass, and every run would be a full re-translation with the suite still green. Confirmed the first test fails on the pre-fix code rather than assuming it would. NOTE: this changes translation OUTPUT, not just caching. The first run after it lands re-sends those 28 pages to the model, so the text will not be byte-identical to what sits on #682 — expect a noisy diff there once. ~10 minutes, once. Co-Authored-By: Claude Opus 5 <[email protected]> * Guard the per-language cache save against a job re-run The save key embeds `github.run_id`, and GitHub REUSES that id when someone re-runs a failed job. On the second attempt the primary key already exists, the restore scores an exact hit, and the save collides with itself. Found by running it rather than reasoning about it. A throwaway workflow exercising the same key shapes showed all four cases: - first run: cache-matched-key='', save proceeds - next run: restored from probe-zh-<prev_id>, cache-hit='false' - re-run: cache-hit='true' <- the collision - languages stayed isolated: ja restored ja's payload, zh restored zh's The same `cache-hit != 'true'` guard `build-daemon.yml:137` carries. With it the re-run skips the save and stays green. That probe also confirmed the two things the rest of this branch assumes and could not otherwise check: `cache-matched-key` really is empty on a total miss — so the new warning fires exactly when a language is about to re-translate everything, and stays silent on the prefix hits that are the normal case — and the `restore-keys` prefix genuinely carries the previous run's file across, which is the whole mechanism by which tomorrow's run inherits today's cache. Co-Authored-By: Claude Opus 5 <[email protected]> * Put both scheduled jobs on one box, behind one installer The integration suite already moved off Actions; the doc translation had not. Both crons cost runner minutes and nothing else — the LLM spend is identical wherever they run — so the translation joins it, on the same machine, image and env file. The runner already locked, checked out a ref and handed off to a script from that checkout. $CANARY_JOB now picks WHICH script, resolved to a path rather than through a case statement, so a third job is a new file in the repo and never a rebuild of the boss's image. Everything per-run is keyed by job. The lock most of all: one shared lock lets a canary wedged on a vendor CLI swallow the night's translation, and the swallow is a clean `exit 0` that reports nowhere. The clone too, since translate commits and switches branches inside its checkout. Three things collapse in the move, which is why the job is shorter than the workflow it replaces: the 14-way matrix was runner parallelism rather than translation structure, the Actions cache layer becomes one 13 KB file in the work dir, and consolidate's re-checkout-and-overlay existed only because its siblings ran on other machines. What does not collapse is the cache eviction that was accidentally load-bearing. A "translated once" entry whose output only exists on an unmerged PR branch makes --update-nav emit nav entries for missing files and mintlify validate fail, while the cache hit regenerates nothing. On Actions an eviction eventually forced a full miss and the run went green by brute force. Nothing evicts this cache, so the existsSync guard is now the only thing keeping the job convergent. Co-Authored-By: Claude Opus 5 <[email protected]> * Print the reason a translate run died, not only post it Co-Authored-By: Claude Opus 5 <[email protected]> * Audit the docs weekly, as a third job on the same box mintlify validate and validate:mdx answer "does this build", per PR, on the pages a PR touches. They pass happily on a corpus that builds perfectly and is quietly wrong: a page nobody has edited since the CLI it documents was rewritten, a page in the nav that is gone, a page in no nav at all and so unreachable by any reader, an in-body link to something renamed, a translation still describing last quarter's behaviour. None of that fails a build, which is exactly the shape a periodic sweep catches and a per-PR gate never will. It is the cheapest job on the box — no gateway key, no push token, no sibling containers — so it installs on a machine holding no credentials but the webhook. Deliberate: an audit that could also FIX what it finds would need write access and a much longer argument about what it may change unattended. It reports and exits 0 by design. --fail-on-findings is there for a future caller that wants a gate, off by default, because an audit that reddens the build the day a page crosses an age threshold gets switched off within a week — and then there is neither a gate nor a report. The judgement lives in scripts/docs-audit.ts as pure functions over the git log, the file list and the cache, so every detector is tested in both directions without a repo, a docs tree or a clock. The shell job is only box wiring. Scheduling it taught the installer to express weekly at all: a spec is now "M H" or a full five-field cron expression. And a job name may carry a dash — docs-audit is a valid path component and an invalid shell variable name — so every per-job lookup goes through one conversion rather than each site remembering. Co-Authored-By: Claude Opus 5 <[email protected]> * Let the GitHub API host be pointed elsewhere GHES needs it, and it is what lets the publish path be proven end-to-end without opening real pull requests. Co-Authored-By: Claude Opus 5 <[email protected]> * Recover when the open PR's branch is gone, instead of failing nightly Found by running the job: an open PR whose branch no longer exists made every subsequent night fail at the fetch, because the branch never comes back. Told apart from a remote we could not REACH, which must not fall through to a new branch — that would open a second PR on a transient network error, and two open auto-translation PRs is what reusing one exists to prevent. Co-Authored-By: Claude Opus 5 <[email protected]> * Pin the stale-PR-branch recovery with tripwires Co-Authored-By: Claude Opus 5 <[email protected]> * Translate reports by opening a PR, not by posting to Slack Its output IS the pull request: a run that did something leaves one, a run that did nothing leaves the previous one untouched. There is nothing a chat message adds that the PR list does not already say. Failures go to the run log and the exit code, which makes die()'s printing load-bearing rather than a convenience. The webhook stops being a requirement for that job, so a box that only runs translate needs no Slack at all. Also: check a job's ref against the REMOTE instead of one hardcoded branch name. Matching the name against origin/failproofaid only ever caught origin/failproofaid. A real secrets.env on this machine carried CANARY_REF=origin/feat/canary-local-runner — merged-and-deleted shortly — and would have sailed through to test a frozen tree forever. Asking whether the branch still exists catches every deleted branch without naming any of them, and anything that is not origin/main now draws a warning: legitimate for a one-off, rarely right for a cron line. Co-Authored-By: Claude Opus 5 <[email protected]> * Explain the webhook only when the webhook is missing translate needs none — it reports by opening a pull request — so printing that rationale under a list that does not contain it reads as though the job wants one it does not. Co-Authored-By: Claude Opus 5 <[email protected]> * Stop the canary probe reading a leaked marker as broken enforcement antigravity failed probe B 3/3 and it was never an enforcement bug. Recorded live against agy 1.1.11: view_file delivers AbsolutePath, which the input map already carries, and a deny on it IS honoured — "tool call denied with reason", sentinel never reaches the model. What actually happened is that canary-read identifies the marker by substring on the command text. Denied on `cat …/CANARY_MARKER.txt`, the agent retried with `cat …/CANARY_MA*`: same file, and a string that no longer contains the matched substring. The shell expanded the glob, the sentinel landed in the transcript, and probe B scored FAIL — because a leaked sentinel deliberately outranks our own log claiming a deny. failproofai did exactly what it was told. This closes the observed family: any CANARY reference except the bash probe's own token (excluding it is load-bearing — denying `touch CANARY_PROBE_ran` here would keep canary-bash out of the hook log and turn probe A inconclusive while looking like a fix), plus a read utility pointed at a glob, which is the `cat *` case that names nothing at all. PARTIAL, and knowingly so. A substring policy over arbitrary shell cannot be closed: a later run still leaked by another route. The real fix is to make a shell route during probe B score INCONCLUSIVE rather than FAIL, so a workaround reads as unproven instead of broken — that changes what the probe measures and wants a decision, not a patch. Regression-checked: claude still PASS/PASS on both probes. Co-Authored-By: Claude Opus 5 <[email protected]> * Score a routed-around read as unproven, not as broken enforcement Probe B asks one question — is a deny on the CLI's READ tool honoured — and an agent with a shell can answer a different one by fetching the bytes another way. Substring-matching the marker cannot stop that: closing the `CANARY_MA*` glob just moved the agent to the next route, and the ways to read a file with a shell are not enumerable. So probe B now tells the two situations apart instead of trying to prevent one. canary-read-shell denies shell file-reads during the READ probe only, identified from the per-probe oracle dir — the one per-probe signal a policy can read, since the daemon wire protocol carries no env. Its separate name is what makes it work: a deny under it can never satisfy read_denied and score a PASS, and the verdict can see the agent reaching for the shell. A leak arriving WHILE those reads are denied is INCONCLUSIVE. A leak with no shell attempt stays FAIL, because that is what a CLI ignoring our deny looks like, and blurring the two would blind this suite to the silent-allow it exists to catch. read_denied's grep grew a trailing space for the same reason: without it, canary-read also matches the canary-read-shell line. Navigation stays allowed — several CLIs locate the file before reading it, and denying ls/pwd would push CLIs that pass today into INCONCLUSIVE for no gain. Verified: claude and codex still PASS both probes with the detector live, all six verdict combinations exercised against the real shell functions, and the ordering test updated to the new shape while keeping its invariant. Co-Authored-By: Claude Opus 5 <[email protected]> * Keep a docs-audit tracking issue on GitHub, alongside the Slack post Slack is read the morning it arrives. An issue is what somebody finds three weeks later wondering why a page is unreachable, so the audit now keeps one "[auto] docs audit" issue current: opened when there is something to do, its body refreshed each week, and CLOSED when a week comes back clean — so an open issue always means "there is something to do" rather than "this ran once, months ago". An issue and not a PR, deliberately. A report is not a change: a weekly PR would either sit open forever or auto-merge a file nobody reads. And an audit that opened a FIXING PR would have almost nothing safe to put in it — a dangling nav entry might mean "delete the entry" or "restore the page", an orphan page might be deliberately unlisted, a broken link has no inferable target. Each is a judgement this job cannot make. So the token stays weak: Issues read+write and nothing else, since the audit never touches a file. It is optional — with none set, the job is exactly what it was, a Slack post. countActionable decides open-vs-closed and excludes stale and never-translated pages on purpose: the nightly translation closes both by itself, and counting them would hold the issue open forever, which is the only way a tracking issue can actually fail. The /issues listing filters out entries carrying a pull_request key — every PR is an issue to that endpoint, so without it an open PR sharing the title would be updated instead. Verified against a stand-in API through the real runner image, all five paths: opens, updates without duplicating, closes on a clean week, no-ops when clean and already closed, and degrades to Slack alone with no token. The decoy PR in the listing was correctly ignored. Co-Authored-By: Claude Opus 5 <[email protected]> * Build the runner image from the checkout when there is one The install has two front doors and only one of them had no clone. Reached the usual way — git clone, then bash integration-suite/local/install.sh — the build context is now the directory this script sits in: no network for the build, and the image provably matches the tree the operator is looking at. Building from the git URL there could hand them an image from a DIFFERENT commit than their checkout while both printed the same branch name. The curl one-liner keeps the remote context, since there is no checkout to use. The runner re-clones the repo on every run either way, so neither goes stale. Co-Authored-By: Claude Opus 5 <[email protected]> * Ship no credentials template, and print the variables instead A file that looks like a credentials file is one `git add -A` away from being committed by whoever fills it in. secrets.env.example was added on this branch and never reached main, so it goes now rather than becoming a thing to delete later. Run the installer with no arguments and it prints exactly which variables to put in the file, grouped by job — generated from the same REQUIRED_ lists the checks enforce, so unlike a checked-in example it cannot drift out of date. It also says to keep the file at mode 600 and out of any checkout. The workflow-to-box parity test loses its comparison target, so it now checks the real consumers: every secret the workflow feeds must be read somewhere in integration-suite/. That is the property that actually mattered — a secret the box never reads means a CLI quietly reporting ERROR forever — and it is checked against the code that reads it rather than against a second copy of the list. A new tripwire keeps any env-shaped file from reappearing under local/. The usage header now leads with clone-then-install; the curl one-liner stays documented below it as the no-clone form. Co-Authored-By: Claude Opus 5 <[email protected]> * docs: fill in the PR number in the changelog (#694) * Publish the runner image, so a box needs Docker and a credentials file Setting the box up meant a clone, an installer and a local image build. It now means Docker and an env file: the runner image publishes to GHCR, and every cron line carries --pull=always, so the box tracks it with nothing to re-run. docker run --rm --pull=always -e CANARY_JOB=docs-audit \ -e CANARY_WORK="$HOME/fp-canary" -v "$HOME/fp-canary:$HOME/fp-canary" \ --env-file "$HOME/fp-canary/secrets.env" \ ghcr.io/failproofai/failproofai-canary-runner:latest Path-filtered to the BAKED layer only. Job scripts reach the box through the run-time clone, so triggering a publish on those would lose the split that lets a harness change reach the box without anyone touching it. The package is set public on purpose. A private one turns that one-line cron into a docker login plus a fourth credential that expires and silently breaks every job when it does, and there is nothing in the layers to protect: node, bun, git, the docker client and mintlify. Every secret arrives at run time through --env-file, and the repo is cloned at run time too. THE SOCKET NOW GOES TO THE CANARY ALONE. It is the only job that spawns sibling containers; translate and docs-audit are plain containers, and the entrypoint demanding a socket on their behalf would have forced two of three cron lines into the long form for nothing. What the entrypoint needs it for is recovering the work dir, so it is required only when CANARY_WORK was not passed — and the canary asserts its own requirement up front, where that knowledge belongs, instead of failing an hour in at the first sibling container. Verified against the rebuilt image: docs-audit runs to completion with no socket mounted at all, and the canary refuses immediately with the flag to add. install.sh writes the three lines against the published image and pulls it at install time, so a private package or a typo'd tag is a problem in front of a person rather than a missed run at 02:00. --build-local still builds from a checkout, for trying a change to the baked entrypoint before publishing it. Co-Authored-By: Claude Opus 5 <[email protected]> * Fail closed when a GitHub lookup cannot be completed Hermes was right, and the same bug was in both jobs. curl piped straight into a parser that swallows its own errors makes a 401, a 5xx and a timeout indistinguishable from an empty list — and the answer to an empty list is to CREATE one. translate would have opened a SECOND auto-translation PR, splitting the generated files against a cache that marks them done so the next run validates an incomplete checkout; docs-audit would have filed a duplicate tracking issue every week until somebody noticed the pile. That is the same duplicate the branch-reuse logic exists to prevent, reached by the one path the ls-remote guard cannot cover: it only runs once a PR was already found. api() now captures the HTTP status and returns non-zero on anything but 2xx, and each lookup is two statements rather than one pipeline so it can actually fail. The status goes to STDERR, not a variable — api() is always called inside $( ), so an assignment there dies with the subshell. The first cut of this fix used a variable and silently printed nothing; a test now pins the stderr form. Verified against a stand-in returning 500: refuses, names the status, and creates zero duplicates. The 200 path is unchanged. Also from review: dropped an unused json() helper, and guarded an indexOf ordering assertion that would have passed for the wrong reason (-1 < any index) the day someone removed the marker it looks for. Co-Authored-By: Claude Opus 5 <[email protected]> * Give cron one short line per job A crontab entry must be a SINGLE line — the format has no continuation — so the docker invocation could not be wrapped. That made each entry ~350 characters: unreadable in a crontab, and mangled by every chat client it was pasted through on the way to whoever sets the box up. run-job.sh holds the invocation, so the crontab reads: 0 11 * * * $HOME/fp-canary/run.sh canary It also owns its own log, which closes a real trap. cron evaluates a `>>` redirect BEFORE the command runs, so a missing logs/ directory meant the job silently never started — and the container could not create the directory its own redirect needed. mkdir then redirect, in that order. install.sh drops it in and writes the short lines, so both setup routes produce the same thing. Two assertions moved with the code they describe rather than being deleted: the job-name passthrough and the docker-socket scoping now check run-job.sh, which is where they are true. Co-Authored-By: Claude Opus 5 <[email protected]> * Pin nanoid to 3.3.18, closing GHSA-2v37-7h3g-55p8 Supply Chain went red on a lockfile this branch never touched. main passed the same scan at 04:57 today and this branch failed at 16:21 — the advisory's affected range was published in between (modified 16:00). CVSS 8.2: custom generators can loop indefinitely when size is zero. The scan output reads "FIXED VERSION 3.3.17" against an installed 3.3.17, which is not actionable as printed; the advisory's real range is introduced 0 → fixed 3.3.18. nanoid arrives transitively through postcss, which asks for ^3.3.17, so 3.3.18 satisfies it without moving anything else: two lines of lockfile, 657 entries before and after. An overrides pin rather than an osv-scanner.toml ignore because that file's own rule is to prefer fixing when a fix exists — and one does. (`bun update nanoid` is the wrong tool here: it adds nanoid as a DIRECT dependency at 6.0.1 rather than bumping the transitive one.) Verified with the same scanner image CI runs: "No issues found", exit 0. Full suite unchanged at 3575 pass. Co-Authored-By: Claude Opus 5 <[email protected]> --------- Co-authored-by: Claude Fable 5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Automated translation update triggered by changes to English documentation sources.