Repository navigation
Conversation
* Admin API lists connected workers and queued demand * Docs: the admin listings * Admin demand listing reads the queue the way the matcher does * Admin demand listing carries the client's last keepalive * Admin demand listing leaves out a record that moved on or went away between reads
…perties cached per pass (TraceMachina#2805) * Add a least_loaded allocation strategy so a newly joined worker takes the next action * best_fit placement with a priority preference, typed properties cached per pass
…unreadable row skipped, a wait for the new index (TraceMachina#2803) * Redis: read a cursor page in the RESP3 shape the first page already parses * Scheduler: the Redis backend gets the retention default, so completed records expire * Scheduler: the listing skips a record it cannot read instead of ending * Redis: wait for a new search index to finish its scan before reading it
…tion, one store read per queued action (TraceMachina#2806) * Give room that opens during a matching pass to the oldest waiting action * Serve a listed action from the record the listing already read * Matching pass: parked actions keep the listing order, a higher memory report is room opening, the parked metric counts confirmed sends
…tdown guard that waits for the drain (TraceMachina#2807) * GoingAway with a drain flag so a shutting-down worker stops receiving work, and reconnect backoff with jitter * Count the original shutdown guard so a SIGTERM waits for the worker to drain
* Dispatch is acknowledged or declined by the worker * Dispatch ack: the answer is recorded before the liveness refresh, a late decline does not pause, counters count requeues, a dead channel is a decline * Dispatch ack: the timeout counts from the scheduler clock at the send, and a late acknowledgement is answered with a kill * Keep the pause through a decline, name the memory property, defer kills on a full channel
…ancelled (TraceMachina#2809) * Kill the whole process group when an action times out or is cancelled * Run the process-group kill test without namespaces * Write the kill test's pid file under the action root * Kill the process group from the child wrapper so every kill path covers it * Use the imported error and log paths in the group kill helper
…d orphaned action directories (TraceMachina#2810) * Bound worker output capture, uploads, precondition scripts and orphan directories * Detach the orphan sweep task so it outlives its spawn handle * Orphan sweep: carry on past a directory it cannot remove, skip symlinks, and log a quiet sweep at debug * Cut log excerpts by bytes and take the cleanup mark in the orphan sweep * Bound open output files with one semaphore and spill outside the file budget * Gate the gated store to unix with the test that uses it
…aceMachina#2802) Co-authored-by: llms bot <[email protected]>
…uard before inputs are fetched (TraceMachina#2812) * Worker: kill an action that exceeds its memory reservation before the pod's cgroup limit does * Kill on a single sample at twice the memory ceiling * Worker honours the platform properties the scheduler reserved for the action * Worker refuses an action whose disk reservation exceeds the free space before fetching its inputs * Use the child wrapper's group kill for the memory ceiling * Log a ceiling breach whose action already ended instead of dropping the send result * Shorten the ceiling-breach log line so both formatters agree * Enforce each axis alone, sample Pss, count admitted disk reservations, overlay properties only with enforcement on * Gate the unix-only test helpers with the tests that use them
…hina#2820) * Worker advertises CPU and memory from its own cgroup limits * Advertise CPU in whole cores by default, millicores only when told * Headroom follows resource_enforcement unless set; refuse to start without a cgroup and without the properties
* Worker readiness follows scheduler registration * Regenerate main's config reference and have the snippet lint read it too * A lost registration reads Initializing, and a readiness path equal to the status path is refused
…on, and resource usage with an outcome (TraceMachina#2822) * Worker: SIGTERM grace, output kept on kill, per-action TMPDIR, timeout clamp, usage v2 with outcome * Read the timed-out result's error from the action result in the grace test * Borrow the finished result's error in the grace test * Drop the wall time assertion from the memory kill test * Namespaces: pass SIGTERM through the stub so a namespaced action gets its kill grace * Follow the split enforcement axes and the regenerated bindings * Use the namespace helpers as main has them and a store key for the kept-output check * Gate the exit-signal read to unix and drop the marker removal result
…aceMachina#2819) Co-authored-by: llms bot <[email protected]>
…lers no longer hold its output open (TraceMachina#2823) * Worker reaps the zombies actions leave behind, and an action's stragglers no longer hold its output open * Take the action's group id before the guard owns the child, and match the zombie sets by lookup * Reaper: own the children we spawn, scan stat off the runtime, forget reaped pids; drain bounded only for killed or straggling actions; biased read * Take the reaper's last sweep by reference
…und over /tmp (TraceMachina#2824) * Give each action a private /tmp: its own tmp directory bound over /tmp inside the mount namespace * Say temporary directory in prose where vale reads tmp as a misspelling * Private /tmp: no mask under a bound /tmp, the size cap only with isolate_tmp, a probe that runs the real sequence, isolate_tmp judged against the resolved mode * Map the gid in the namespace probe as the action path does
…writes (TraceMachina#2825) * Worker: a disk soft limit and peak_disk_kb from the files an action writes * Walk the action directory through the crate's blocking spawn, and expect the sample to catch the write in progress * Give the integration test suite the tempfile dependency the disk walk test uses * Disk limit: the action's own files by modification time, the walk as a polled task, periodic walks only for the soft limit, the disk figure counts as sampled * Pass the walk start to the Linux sampler and give the disk guard test slack * Time the disk walk by the filesystem's clock
…orker as the last step (TraceMachina#2829) * Scheduler: memory escalation with retry budgets by cause, the whole worker as the last step * Escalation: no veto on a whole-worker ask, max_kb caps the ceiling, draining workers left out, percent validated, a zero base refused, a max_steps test, the shared requeue metric
…ion environment, namespaces, measured usage, and a wait at the cap (TraceMachina#2830) * Persistent workers: a configurable pool with an idle sweeper, the action environment, namespaces, measured usage, and a wait at the cap * Start the persistent worker sampler with no ceilings * Persistent worker tests pass PATH to the worker process * Persistent worker processes get the worker's additional environment too * Keep the unix cfg on the shell echo script * Persistent workers: key prints no environment values, the wait at the cap is raced against the kill and bounded by the action's time, CPU as a delta, one environment helper, the pool on by default * Windows persistent worker tests pass the whole environment
…h a cold-start policy, and where each reservation came from (TraceMachina#2832) * Historical resource scheduler: hints by digest keys, size classes with a cold-start policy, and where each reservation came from * Write the eternal test timeout in hours * Hints: every key family a record names, the ladder guarded by what the action carries, zero hints fall to cold start, rules and property types checked at load, disk on the cold start
…aceMachina#2828) Co-authored-by: llms bot <[email protected]>
…aceMachina#2843) * Worker: build the cleanup mark's guard after the lock is released, so a second mark cannot deadlock the runtime on its own mutex * Worker: an action's cleanup waits for the mark instead of stepping aside, and the sweep and a retry's stale removal hold it
…EOF (TraceMachina#2844) * Store: a read that stops short of the digest's size is an error, not EOF, in the Redis store and the verify store * Store: DataLoss outranks the inner send error on an over-long read, and a copy that fails size verification is dropped from the caches below
…rectory (TraceMachina#2845) * Worker: read capacity and free memory from the worker's own cgroup directory, not the mount root * Worker: the cgroup limit comes from the nearest limited ancestor, no limit anywhere keeps the typed properties, and the failure names the cgroup
…l reach it (TraceMachina#2846) * Worker: an input fetch that never completes fails the action at max_download_timeout, and a kill reaches it, instead of holding its slot * Worker: a kill token every phase watches, so a kill ends a fetch or an upload; cleanup retries the removal; the tests clean up
…ina#2811) * Add opt-in generic gRPC event-sink transport Support string-keyed write-only event stores without changing CAS or Action Cache semantics. Carry the actual payload size in ByteStream resources, validate configuration, and regenerate the configuration reference. Refresh the change onto main and retain the declaration-order lint fix. Replaces the same-author PR commits f775a29 and d5b1e4c. AI-assisted implementation. * Validate event-sink acknowledgements and exercise lost-ack retries Reject event-sink responses whose committed size differs from the full payload. Cover incomplete acknowledgements and retries after the receiver consumes an event but loses its response. Document durable receiver acknowledgements, stable event-key deduplication, and the limits of publisher recovery. Validated the 13 gRPC-store and 11 configuration tests with Bazel, including Clippy and rustfmt, and ran the documentation and repository hooks. AI-assisted implementation at Marcus Eagan's direction.
…hina#2826) should_timeout_operation ended an Executing action once Action.timeout had passed since it was assigned. The worker enforces the same timeout, but its clock starts when the command starts, after input fetch, so the scheduler always fired first. The action was requeued onto the worker still running it, which refused with AlreadyExists; the retries ran out, and when the worker's own DEADLINE_EXCEEDED result arrived it was treated as a stray and the worker was disconnected, taking every other action on it down too. The scheduler now enforces Action.timeout in every liveness case, at Action.timeout plus no_event_action_timeout, measured from the later of the assignment and the worker's last update on the action. A live worker reports first, so no liveness special-casing is needed, and the scheduler still ends an action whose worker never reports: a hung input fetch, an unkillable child, a broken timeout_handled_externally wrapper, or an orphan no instance reaps. The client's DEADLINE_EXCEEDED message now names the deadline that fired, instead of always citing no_event_action_timeout. This is the interim: the scheduler cannot observe when the command starts. TraceMachina#2827 tracks the worker reporting command start as an operation transition, so the deadline can be anchored to that event. Co-authored-by: Johnny <[email protected]>
…store, and a missing input names its digest so the client re-uploads it (TraceMachina#2855)
…the worker evicted and idle (TraceMachina#2856) * Worker: no scheduler call awaited in the run loop, a deadline on keepalives and acknowledgements that reconnects, and HTTP/2 keepalive on the scheduler channel * Worker: the GoingAway calls on shutdown carry the same deadline * Worker: the layout CI's rustfmt asks for * Keep the scheduler stream flowing: ordered processing off the reader, no deadlines on stream messages * Qualify Code in the worker API server test * Box the action future before the launch closure * Box the action chain inside the action closure * Raise the binary's recursion limit for the worker's action future * Use try_update for the filesystem store generation counter
…a#2859) Co-authored-by: config reference bot <[email protected]>
…aceMachina#2848) Co-authored-by: llms bot <[email protected]>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
* chore(deps): update rust crate serial_test to v4 * Update MODULE.bazel.lock
…eMachina#2864) Co-authored-by: Marcus Eagan <[email protected]>
…oolchain identity The Nix / Bazel Dev lane gets 0/2723 remote cache hits. It compiles inside `nix develop`, where the default C++/Rust toolchain resolves to nix-store paths (/nix/store/<hash>-clang|gcc|rust) that are baked into every action key, so its digests never match the cache the Native lanes warm off a non-nix toolchain (the Bazel 9 Native lane gets 2273/3915 against the same endpoint). If the remote cache warms one Bazel config it should warm both; this closes that gap by keying the lane on a stable, relocatable toolchain identity rather than host paths. - Factor the hermetic LLVM platform + downloaded Rust toolchains that nl-rbe already uses into a new, endpoint-free `nl-toolchain` config, and have nl-rbe inherit it via `--config=nl-toolchain`. These toolchains key on downloaded CONTENT, not host paths, so they produce identical action digests on a bare runner, inside `nix develop`, or on a remote worker. No behavior change for nl-rbe (same flags, now shared). - Apply `--config=nl-toolchain` to the Bazel Dev lane. This is cache-read only (no remote executor added): the lane keeps running locally, it just keys its compiles against the hermetic identity so they can hit a warm cache. Verification: - PROVEN (measured): action keys are byte-identical. A `bazel aquery` proof run compares three ways and shows the nix+nl-toolchain command lines match the bare-runner nl-toolchain command lines and no longer reference /nix/store: https://github.com/b7r6/nativelink/actions/runs/37059855343 - PROJECTED (not yet measured): with a warm cache the lane should drop from ~670s to ~250s. This is a projection -- proven here is key alignment, not a full warm read. - REQUIRES ONE UPSTREAM CHANGE to realize the win: the trusted main-push cache warmer must also build with `--config=nl-toolchain`, so it warms the same key this lane now requests (today's Native warmer uses bare `--extra_toolchains` without the `@llvm` platform pin). One line; happy to do it in a follow-up once this lands. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This was referenced Oct 3, 2026
Closed
b7r6
force-pushed
the
main
branch
2 times, most recently
from
October 4, 2026 22:41
83c1f81 to
a9c6801
Compare
Owner
Author
|
Closing: this CI experiment concluded — its findings shipped upstream via the TraceMachina#2874/TraceMachina#2877/TraceMachina#2880/TraceMachina#2886/TraceMachina#2897 series. Branch preserved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The common case first. The vast majority of PRs change a small dependency cone (a handful of crates). In that regime, with a warm remote cache, the Bazel lanes are fast because their actions are cacheable — but the Nix / Bazel Dev lane can't participate: it lands 0 remote cache hits of its 2723 actions (
2723 processes: 1142 internal, 1700 processwrapper-sandbox— every compile runs locally), so it pays a full from-source build (~12m) on every PR regardless of cone size. By contrast the Bazel 9 Native lane, against the same cache endpoint, gets 1877 remote cache hits + 803 disk cache hits of 3915 actions (3915 processes: 803 disk cache hit, 1877 remote cache hit, 1351 internal, 3 processwrapper-sandbox). This change targets the cache-key mismatch that keeps the Bazel Dev lane cold so it can read the same warm cache.Why the lane misses on keys: it compiles inside
nix develop, where the default C++/Rust toolchain resolves to nix-store paths (/nix/store/<hash>-clang|gcc|rust) that are baked into every action key — so its digests cannot match a cache warmed off a non-nix toolchain. The fix keys the lane on a stable, relocatable toolchain identity instead of host paths. (On runs where the lane also hitsRemote Cache: UNAVAILABLEtransient connectivity, hits are zero for that separate reason; the key mismatch is the standing cause this change removes, and it is the one demonstrated by the aquery proof below.)The shape of the problem (measured fanout curve)
No-op comment at increasing dependency-cone depth, same read-only staging cache. Every cell is a clickable run. This is the shape that makes the case:
Runs: T1 bazel / cargo · T2 bazel / cargo · T3 bazel / cargo · T4 bazel / cargo
Reading it:
cargoon any platform, so even a 1-crate leaf change pays the full ~12m Windows / ~6m Ubuntu tax. Cargo cannot play this game.(Spread check: the T1 leaf Bazel 9 lane measured 3m02s and 3m25s on two independent runs — tight enough that the curve's shape is not noise.)
So for the common case — warm cache, small-cone PR — this change moves the Bazel Dev lane from a flat ~12m into the fast cacheable regime.
The change
Two files, minimal and self-contained:
ci.bazelrc: factor the hermetic LLVM platform + downloaded Rust toolchains thatnl-rbealready uses into a new, endpoint-freenl-toolchainconfig;nl-rbeinherits it via--config=nl-toolchain(no behavior change fornl-rbe— same flags, now shared). These toolchains key on downloaded content, not host paths, so they produce identical action digests on a bare runner, insidenix develop, or on a remote worker..github/workflows/nix.yaml: apply--config=nl-toolchainto the Bazel Dev lane. Cache-read only — no remote executor is added; the lane keeps running locally, it just keys its compiles against the hermetic identity so they can hit a warm cache.How was this verified?
DEMONSTRATED (measured, linkable): the toolchain path signatures align. A
bazel aqueryproof compares the compile action command lines three ways — (A) insidenix developwith the default toolchain, (B) insidenix developwith--config=nl-toolchain, (C) on a bare runner with--config=nl-toolchain— and shows B matches C (the warmer identity) and no longer references/nix/storetoolchains, whereas A does: https://github.com/b7r6/nativelink/actions/runs/37059855343 . That path-signature alignment is the precondition for a warm read. (The proof prints the signatures/counts for inspection rather than hard-failing the job on mismatch; the comparison is in its uploadedaquery-*artifacts.)Green + the key-match is live, not just theoretical (measured, linkable): on the fork verification run the Bazel Dev lane builds successfully with
--config=nl-toolchainand its remote-cache hit count moves from 0 to 18 (3915 processes: 18 remote cache hit, 1351 internal, 2665 processwrapper-sandbox) — Bazel Dev job, full Nix run, all lanes green. The 18 hits are whatever already sits in the staging cache under the hermetic identity; it is small precisely because nothing has warmed the Bazel-Dev cone under this key yet (see the dependency below). That the number is nonzero at all is the direct, non-aquery confirmation that the lane now keys against the warm identity.PROJECTED (NOT measured end-to-end), with its basis stated: once a writer warms this key, the lane should drop from ~12m (the flat curve above) toward the regime the Native Bazel 9 lane already demonstrates against the same cache endpoint — that lane hits 2680/3915 actions cached (1877 remote + 803 disk, 68%) and finishes in ~5-6m. The projection is "the Bazel Dev lane, once keyed identically, reaches a comparable hit rate," i.e. ~12m → ~4-6m (design target ~250s). It is a projection, not a measurement: see the caveat.
Why it is not measured here (stated plainly): realizing the warm read needs a writer that populates this key, and I verified this fork has no such writer — it carries 0 Actions secrets / 0 variables, so its CI reads the staging cache anonymously and cannot write Bazel-Dev-keyed (
nl-toolchain) artifacts into it, and its main-push warmer is non-functional for the same reason. I could not, at any budget, populate a warmnl-toolchain-keyed cache from this fork. So this PR presents the demonstrated key-match plus the projection anchored to the Native lane's measured 68% warm hit rate, rather than a fork-measured warm number I cannot honestly produce.The one honest caveat (worst case, acknowledged)
The warm hit is not yet demonstrable on this fork, and realizing it needs one upstream line: the trusted main-push cache warmer must also build with
--config=nl-toolchain, so it warms the same key this lane now requests. Today's Native warmer uses bare--extra_toolchainswithout the@llvmplatform pin, so it warms a different key. Until that one-line change lands, this lane's hit rate is unchanged — it does not get slower, it just doesn't yet get faster. I'm happy to send that warmer one-liner as a follow-up the moment this lands, or fold it in here if you'd prefer it in one PR. (The fork's own main-push warmer is broken for unrelated reasons, which is why the end-to-end warm number is projected rather than measured here.)Risk
Low. Cache-read keying change only — no remote executor, no cache-write change, no
ci-modechange. Thenl-toolchainextraction is a pure refactor of flagsnl-rbealready carried. Worst case if the warmer line never lands: the Bazel Dev lane's hit rate is unchanged (never slower). macOS lanes do not apply this config (LRE auto-applies conflicting platforms there, already guarded by the setup action).AI assistance
An agent (Claude Opus 4.8) drafted the config refactor, ran the aquery proof, the fanout-curve measurement, and the fork verification, and wrote this description; a human (b7r6) directed it, reviewed every line, and is answerable for all of it. The commit carries a Co-Authored-By trailer.