[fork-only test v2] scheduler-only diff

.github/workflows/lre.yaml

@@ -20,7 +20,41 @@ concurrency:
   cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
 
 jobs:
+  # Compute whether this diff can affect the LRE example builds. These lanes
+  # build @local-remote-execution//examples, so they are keyed off the LRE
+  # toolchain inputs. Pushes to main always run the full matrix (backstop).
+  changes:
+    name: Compute affected lanes
+    runs-on: ubuntu-24.04
+    outputs:
+      lre: ${{ steps.filter.outputs.lre }}
+    steps:
+      - name: Checkout
+        uses: >- # v6.0.2
+          actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
+
+      - name: Filter LRE-affecting paths
+        uses: >- # v3.0.2
+          dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
+        id: filter
+        with:
+          filters: |
+            lre:
+              - 'local-remote-execution/**'
+              - 'MODULE.bazel'
+              - 'MODULE.bazel.lock'
+              - '*.bazelrc'
+              - 'flake.nix'
+              - 'flake.lock'
+              - 'tools/**'
+              - '.github/workflows/lre.yaml'
+              - '.github/actions/**'
+
   local:
+    needs: [changes]
+    if: >-
+      github.ref == 'refs/heads/main'
+      || needs.changes.outputs.lre == 'true'
     strategy:
       fail-fast: false
       matrix:

.github/workflows/nix.yaml

@@ -157,8 +157,45 @@ jobs:
           nativelink_attic_token: ${{ secrets.NATIVELINK_ATTIC_TOKEN }}
           nix_name: .#${{ matrix.test-name }}-with-nativelink-test
 
+  # Compute which lanes this diff can affect. The rbe-* lanes validate
+  # "does this toolchain still build through NativeLink", so they are keyed
+  # off the toolchain/LRE configuration inputs they consume. Pushes to main
+  # always run the full matrix (backstop), so no change can merge without
+  # the complete matrix running against it.
+  changes:
+    name: Compute affected lanes
+    runs-on: ubuntu-24.04
+    outputs:
+      toolchain: ${{ steps.filter.outputs.toolchain }}
+    steps:
+      - name: Checkout
+        uses: >- # v6.0.2
+          actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
+
+      - name: Filter toolchain-affecting paths
+        uses: >- # v3.0.2
+          dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36
+        id: filter
+        with:
+          filters: |
+            toolchain:
+              - 'toolchain-examples/**'
+              - 'local-remote-execution/**'
+              - 'MODULE.bazel'
+              - 'MODULE.bazel.lock'
+              - '*.bazelrc'
+              - 'flake.nix'
+              - 'flake.lock'
+              - 'tools/**'
+              - '.github/workflows/nix.yaml'
+              - '.github/actions/**'
+
   # rbe-toolchain is slow because it has many subcommands, so split it out
   generate-rbe-commands:
+    needs: [changes]
+    if: >-
+      github.ref == 'refs/heads/main'
+      || needs.changes.outputs.toolchain == 'true'
     runs-on: ubuntu-24.04
     outputs:
       matrix: ${{ steps.set-matrix.outputs.matrix }}

llms-small.txt

@@ -1653,11 +1653,12 @@ reusing a warm process, and a clear-eyed view of the isolation you trade for it.
 
 ## Two things to know before you start
 
-**Almost nothing needs configuring in NativeLink.** No field anywhere in
-[the configuration model](/configuration) turns this on, tunes the pool, or
-selects a protocol; `LocalWorkerConfig` has no persistent-worker section at
-all, and every pool parameter is a compiled-in constant. The feature is
-driven by two platform properties on each action.
+**Little needs configuring in NativeLink.** The pool is on by default and
+the protocol is chosen by two platform properties on each action. The
+worker's `persistent_workers` block (v1.7.3 and later) sizes it: how many
+processes per key, how long an idle one lives, how many requests one serves,
+how long an action at the cap waits for a process to come back, and
+`enabled: false` to run every such action one-shot.
 
 ## Opt an action in
 
@@ -1672,13 +1673,14 @@ Actions are pooled by a key derived from the command line:
 
 ## The isolation you give up
 
-This is the part to read twice. NativeLink applies real isolation to ordinary
-actions and **applies none of it to persistent workers**. The persistent path
-returns before any of it runs.
+This is the part to read twice. A persistent worker process outlives the
+actions it serves, so some of the isolation an ordinary action gets cannot
+apply to it, and some of it did not apply until v1.7.3.
 
 ## The pool's actual behaviour
 
-Every parameter below is a compiled-in constant. None is configurable.
+The pool is configured under `persistent_workers` on the worker; every value
+below is its default.
 
 ## Steps
 
@@ -3457,6 +3459,15 @@ short fix and all five are prerequisites.
 Before you write a readiness probe, understand what `/status` actually
 measures, because the answer is surprising in both directions.
 
+## Readiness is a different question
+
+`/status` answers whether the components that registered a health indicator
+have failed. It stays `200` while a component is still initializing, which is
+the right answer for a liveness probe and the wrong one for readiness. The
+`health` service therefore also serves `/ready` (`readiness_path`, default
+`/ready`), which answers `503` while anything is initializing or failed and
+`200` otherwise.
+
 ## Build the deployment
 
 <li>
@@ -3596,6 +3607,22 @@ in response to what the worker returns.
 Several NativeLink limits queue instead of erroring, which is why "hung" is a
 more common symptom than "failed":
 
+## Every build stalls at `[Sched]` and nothing errors
+
+Clients sit on `WaitExecution` for hours, the scheduler log is quiet apart
+from `Could not find AwaitedAction for the given operation id`, idle workers
+take nothing, and the provisioner reads demand as zero. The scheduler's
+records are being evicted from Redis.
+
+## The scheduler's Redis keeps every completed action
+
+**Symptom.** With the Redis scheduler backend, `DBSIZE` on the scheduler's
+database grows by one `aa_*` hash per action ever run and never falls; the
+records all read `completed` and `TTL` says `-1`. Once the search index holds
+more than one cursor page, listings over it (the admin read the worker
+provisioner makes, the scheduler admin endpoints) can fail with
+`Failed to get operations list! Parse: Non map item`.
+
 ## Restarting the scheduler
 
 Not an incident so much as a procedure that behaves worse than you expect.
@@ -3958,8 +3985,8 @@ Every property key a worker advertises has a declared type:
 
 ## Retries, and what "failed" means
 
-The attempt counter is incremented by two different events, with different
-rules.
+The attempt counter is incremented by one event, and two other events that
+used to spend it now have budgets of their own.
 
 ### Two timeouts with very different defaults
 
@@ -4100,8 +4127,11 @@ result computed with a different value of it.
 ### Actions can run more than once
 
 The default `max_job_retries` of 3 is checked strictly, so an action may
-execute **four** times. A worker disconnect always costs an attempt; a
-`ResourceExhausted` never does.
+execute **four** times on its own failures. A lost worker has its own budget,
+`max_worker_loss_retries` (default 10), and a memory escalation its own
+(`memory_escalation.max_steps`, default 8), so an action can run more often
+than four times when workers die under it or it keeps outgrowing its
+reservation; a `ResourceExhausted` never costs anyth…
 
 ### Some output metadata is discarded