fix(webhook): queue a standing-watch batch at the hook cap, don't drop it

AGENTS.md

@@ -64,6 +64,7 @@ Keep the module self-contained: deployed boxes fetch `modules/agent-box.nix` as
 - `nix build -L .#checks.<system>.checkout-bootstrap` runs `tests/test-checkout-bootstrap.sh` against `modules/src/checkout-cli.sh`, the script that puts this repo ON a deployed box (issue #242). Its assertions are mostly REFUSALS, because that is where the damage would be: it runs unattended at every supervisor start, in a tree sibling sessions are working in, so a realign that moved somebody's branch pointer would destroy work at boot on a box nobody is watching. `origin` is a local repository and `gh` is a shim, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-checkout-bootstrap.sh modules/src/checkout-cli.sh`.
 - `nix build -L .#checks.<system>.source-tree` runs `tests/test-source-tree.sh` against `modules/src/source-tree.sh`, the tree the box is BUILT from and the whole of what an update moves (issue #242). Weighted at the refusals for the same reason: it runs as root, unattended, and the tree it leaves behind is what the next rebuild builds — so a rewritten history, a downgrade, and a baseline the tree has never heard of each get an assertion, as does the realign that makes the fast-forward guard measure ancestry from the rev the box is RUNNING. It also pins the two locks the trust boundary rests on: `check` answers from `git ls-remote` and so creates no tree and fetches into none, and git runs with `core.hooksPath` pointed at nothing, so a `post-checkout`/`post-merge` hook in the tree cannot run as root (that assertion has a negative control — remove the lock and it fails). `origin` is a local repository, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-source-tree.sh modules/src/source-tree.sh`.
 - `nix build -L .#checks.<system>.checkout-options` is the eval regression for `selfUpdate`'s three path assertions (issue #242). It reads `config.assertions` rather than forcing `toplevel`, so a failure names WHICH assertion fired instead of only reporting that something did — and it asserts the accepting cases too, so an assertion that rejects everything fails it as loudly as one that rejects nothing. `selfUpdate.srcDir` is the one that matters most, because root BUILDS the box from that tree: it is confined to a normalized path under `/var/lib`, since owning the directory is not enough — a writable ancestor (`/home/agent/src`, `/tmp/src`) lets an agent swap the whole tree and choose what root builds. For `selfUpdate.checkout.path` the inputs that matter are the ones a first pass at "must be relative" lets through: `../agent-box` escapes the home, `.` and `""` collapse to `/home/<maintainer>` itself — which the agent unit's `ProtectSystem=strict` would refuse as EROFS inside a background job's journal — and `a//b`, which resolves to a perfectly ordinary child path and is refused for a different reason: every empty component is, because one is how the collapsing cases are spelled.
+- `nix build -L .#checks.<system>.webhook-defer` runs `tests/test-webhook-defer.sh`: what `modules/src/webhook-spawn.sh` answers at the hook-session ceiling, and what the pinned `webhook.py` does with that answer (issues #170, #301). A refusal is `exit 75` (`EX_TEMPFAIL`), the one code the dispatcher reads as "declined for now" rather than "this spawner is broken" -- every other code drops the batch, and a standing watch is for events NO session owns, so nothing else is holding them. The second half runs the REAL wrapper as the REAL Dispatcher's spawn command, fills the cap, frees a slot and asserts the declined batch starts by itself, because the bug was the two programs disagreeing about what a non-zero exit meant. Runnable without Nix too: `bash tests/test-webhook-defer.sh modules/src/webhook-spawn.sh /path/to/webhook.py` (the dispatcher half is skipped, and says so, when no `webhook.py` is given).
 - `nix flake metadata` validates flake inputs and basic evaluation.
 - `nix build .#packages.x86_64-linux.vm` builds the bootable qcow2 image under `result/`.
 - `nix build -L .#checks.<system>.multi-user` runs the quick module/configuration assertion.
@@ -264,7 +265,7 @@ get written in — the desktop merely looks a little uneven.
 
 Use `pkgs.testers.runNixOSTest` for service and VM behavior; name tests after the capability under test. Use Playwright `*.spec.ts` files only for behavior requiring a real browser or deployed instance. Add regression coverage with each behavioral fix. There is no numeric coverage threshold; CI expects every relevant named flake check to pass.
 
-A VM test script has a hard ceiling of 128 KiB. nixpkgs passes it to the driver build in one environment variable and Linux caps a single environment string at `MAX_ARG_STRLEN`, so the test past that limit fails to build with `error: executing '.../bin/bash': Argument list too long` and no VM boots -- nothing in that message names the test script or its size. The `testscript-fits` check asserts the limit a page early so the failure is legible. When it fires, the answer is not shorter comments: move the assertions that do not need a VM into a native `runCommand` check (`webhook-spawn-claim` is one such move, out of `tests/webhook.nix`), or split the test the way `tests/sessions-common.nix` split the session tests.
+A VM test script has a hard ceiling of 128 KiB. nixpkgs passes it to the driver build in one environment variable and Linux caps a single environment string at `MAX_ARG_STRLEN`, so the test past that limit fails to build with `error: executing '.../bin/bash': Argument list too long` and no VM boots -- nothing in that message names the test script or its size. The `testscript-fits` check asserts the limit a page early so the failure is legible. When it fires, the answer is not shorter comments: move the assertions that do not need a VM into a native `runCommand` check (`webhook-spawn-claim` and `webhook-defer` are two such moves, out of `tests/webhook.nix`), or split the test the way `tests/sessions-common.nix` split the session tests.
 
 Never end a test pipeline with `grep -q`. The driver runs each command under `set -euo pipefail`, and `grep -q` exits on the FIRST match — the producer upstream then gets EPIPE, and its non-zero status fails the whole assertion even though the pattern matched (a `must succeed` that reports exit 123 with `write error: Broken pipe` in the log). Write `… | grep PATTERN >/dev/null` instead, which drains the input, or capture to a file first and grep the file. `grep -q` is safe only with a file operand.
 

README.md

@@ -562,8 +562,11 @@ On by default (`webhook.enable`), needs `web.enable`. How it fits together:
   coalesce into one session; concurrent spawns are capped, and the wrapper
   refuses to accumulate more than a handful of live `hook-*` sessions — counted
   as sessions that are actually running, so a finished one frees its slot even
-  if nobody delisted it. The spawned session's prompt tells it to remove itself
-  when done.
+  if nobody delisted it. A batch that arrives at that ceiling is **queued, not
+  dropped**: the wrapper declines it with `EX_TEMPFAIL` and the receiver
+  re-offers it as slots free (bounded by `LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S`, an
+  hour here). The spawned session's prompt tells it to remove itself when
+  done.
 - The **settings page**'s **Webhook** panel carries both halves of what a
   sender's form asks for — the payload URL per configured source and that
   source's secret — each with a **copy button**, so registering a webhook needs

bin/agentbox

@@ -2158,8 +2158,17 @@ class Renderer:
         if self.spec.webhook_enable:
             t.file(self.p("/etc/agent-box/units",
                           f"agent-box-webhook-{u.name}.env"),
-                   env_file([("LOCAL_WEBHOOK_STATE_DIR",
-                              f"{u.home}/.local/state/local-webhook")]))
+                   env_file([
+                       ("LOCAL_WEBHOOK_STATE_DIR",
+                        f"{u.home}/.local/state/local-webhook"),
+                       # How long a batch the hook-session cap declined
+                       # (exit 75, modules/src/webhook-spawn.sh) may wait
+                       # for a slot before the receiver drops it.
+                       # local-webhook's own default is 300s, which is
+                       # shorter than a hook session runs — see the module's
+                       # half of this file for the whole argument.
+                       ("LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S", "3600"),
+                   ]))
 
     def user_dropins(self, t, u):
         """Host-level content, per instance.

flake.nix

@@ -1373,6 +1373,52 @@ open(sys.argv[3], "w").write(header + yaml.safe_dump(data, sort_keys=True))' \
               cp log "$out"
             '';
 
+          # What the spawn wrapper ANSWERS at the hook-session ceiling, and
+          # what the pinned local-webhook does with that answer (#170, #301).
+          # A refusal used to be `exit 1`, which the dispatcher cannot tell
+          # from "command not found", so it dropped the batch — and a standing
+          # watch is for events NO session owns, so nothing else held them.
+          # 75 (EX_TEMPFAIL) is the code that means "declined for now", and
+          # the batch is then re-offered until a slot frees.
+          #
+          # Both halves run against the REAL wrapper and the REAL pinned
+          # webhook.py, because the bug was the two disagreeing about what a
+          # non-zero exit meant — the wrapper is the dispatcher's own spawn
+          # command here, and the deferral is driven end to end in seconds.
+          # The VM test keeps the wiring an interpreter cannot show; this is
+          # where the contract between the two programs lives.
+          webhook-defer =
+            pkgs.runCommand "agent-box-webhook-defer"
+              {
+                nativeBuildInputs = [
+                  pkgs.bash
+                  pkgs.coreutils
+                  pkgs.jq
+                  pkgs.python3
+                ];
+                # The whole directory, not the one file: the source form
+                # carries @@include markers and the test resolves them
+                # against its siblings, exactly as the assembler does.
+                src = ./modules/src;
+                tests = ./tests/test-webhook-defer.sh;
+                # The same pin the module and #runtime read, fetched the
+                # same way (nix/webhook-pin.nix).
+                webhookPy =
+                  let pin = import ./nix/webhook-pin.nix; in
+                  builtins.fetchurl {
+                    url = "https://raw.githubusercontent.com/${pin.repo}/${pin.rev}"
+                          + "/local-webhook/webhook.py";
+                    sha256 = pin.sha256;
+                  };
+              } ''
+              bash "$tests" "$src/webhook-spawn.sh" "$webhookPy" > log 2>&1 || {
+                cat log
+                exit 1
+              }
+              cat log
+              cp log "$out"
+            '';
+
           # Unit tests for the durable per-session lease (issue #535):
           # outcome precedence (first ending wins, never the most recent),
           # clear's delete-not-blank resolution, and the read-only accessor