From 876356ed30d7faab748f3eafcd389e3c611d11f7 Mon Sep 17 00:00:00 2001 From: defangdevs Date: Sat, 5 Sep 2026 00:09:02 +0000 Subject: [PATCH 1/2] fix(webhook): queue a standing-watch batch at the hook cap, don't drop it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `modules/src/webhook-spawn.sh` refused a batch at the hook-session ceiling with a message and `exit 1`. The dispatcher cannot tell that apart from "command not found", so it dropped the batch for good — and a standing watch is for events NO session owns, so unlike a failed session delivery there was no peer holding a copy. #170 made the loss visible; the events were still gone. local-webhook reads 75 (EX_TEMPFAIL) as "declined for now": the batch goes back at the head of its key's pending list, is re-offered as the rate window reopens, and starts the moment a slot frees, with every line re-checked against live session ownership first. So the cap — and only the cap — exits 75. A malformed AGENT_BOX_HOOK_SESSION_ARGS or a failed `add` keeps its own status, because re-offering a broken spawner would loop. The receiver unit also raises LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S to an hour. Upstream's 300s default is for a consumer that declines briefly; ours is a session ceiling and a hook session runs for tens of minutes, so five would drop the batch anyway — the same loss, later. Overridable per box in agent-box-webhook-.local.env. The refusal record now says `deferred`, so `status` does not report as lost a batch the receiver is still holding, and the sentence `ls`/`status` print says the batch is queued rather than dropped. New native check `webhook-defer` runs the REAL wrapper as the REAL pinned Dispatcher's spawn command: it fills the cap, asserts exit 75 and the recorded deferral, frees a slot, and asserts the declined batch starts by itself with no second delivery. That contract was the whole bug — the two programs disagreeing about what a non-zero exit meant — and it costs seconds natively. The typo-fallback leg moved there too: tests/webhook.nix sits 384 bytes under the 128 KiB testScript ceiling. Closes #301. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NWmLRsa4keJzBo9Minofxe --- AGENTS.md | 3 +- README.md | 7 +- bin/agentbox | 13 +- flake.nix | 46 ++++ modules/agent-box.nix | 138 +++++++--- modules/agent-box.nix.in | 21 +- modules/src/default-agents-webhook.md | 12 +- modules/src/webhook-cli.sh | 51 ++-- modules/src/webhook-spawn.sh | 56 +++-- .../web/etc/agent-box-guides/AGENTS.agent.md | 12 +- .../units/agent-box-webhook-agent.env | 1 + .../units/agent-box-webhook-robot.env | 1 + .../bin/agent-box-webhook-spawn | 54 ++-- .../agent-box-webhook/bin/agent-box-webhook | 51 ++-- .../etc/agent-box-guides/AGENTS.agent.md | 12 +- .../etc/agent-box-guides/AGENTS.robot.md | 12 +- .../units/agent-box-webhook-agent.env | 1 + .../units/agent-box-webhook-robot.env | 1 + tests/test-webhook-defer.sh | 238 ++++++++++++++++++ tests/webhook.nix | 75 +++--- 20 files changed, 636 insertions(+), 169 deletions(-) create mode 100755 tests/test-webhook-defer.sh diff --git a/AGENTS.md b/AGENTS.md index 05323ef1..d1e9cbff 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,6 +64,7 @@ Keep the module self-contained: deployed boxes fetch `modules/agent-box.nix` as - `nix build -L .#checks..checkout-bootstrap` runs `tests/test-checkout-bootstrap.sh` against `modules/src/checkout-cli.sh`, the script that puts this repo ON a deployed box (issue #242). Its assertions are mostly REFUSALS, because that is where the damage would be: it runs unattended at every supervisor start, in a tree sibling sessions are working in, so a realign that moved somebody's branch pointer would destroy work at boot on a box nobody is watching. `origin` is a local repository and `gh` is a shim, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-checkout-bootstrap.sh modules/src/checkout-cli.sh`. - `nix build -L .#checks..source-tree` runs `tests/test-source-tree.sh` against `modules/src/source-tree.sh`, the tree the box is BUILT from and the whole of what an update moves (issue #242). Weighted at the refusals for the same reason: it runs as root, unattended, and the tree it leaves behind is what the next rebuild builds — so a rewritten history, a downgrade, and a baseline the tree has never heard of each get an assertion, as does the realign that makes the fast-forward guard measure ancestry from the rev the box is RUNNING. It also pins the two locks the trust boundary rests on: `check` answers from `git ls-remote` and so creates no tree and fetches into none, and git runs with `core.hooksPath` pointed at nothing, so a `post-checkout`/`post-merge` hook in the tree cannot run as root (that assertion has a negative control — remove the lock and it fails). `origin` is a local repository, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-source-tree.sh modules/src/source-tree.sh`. - `nix build -L .#checks..checkout-options` is the eval regression for `selfUpdate`'s three path assertions (issue #242). It reads `config.assertions` rather than forcing `toplevel`, so a failure names WHICH assertion fired instead of only reporting that something did — and it asserts the accepting cases too, so an assertion that rejects everything fails it as loudly as one that rejects nothing. `selfUpdate.srcDir` is the one that matters most, because root BUILDS the box from that tree: it is confined to a normalized path under `/var/lib`, since owning the directory is not enough — a writable ancestor (`/home/agent/src`, `/tmp/src`) lets an agent swap the whole tree and choose what root builds. For `selfUpdate.checkout.path` the inputs that matter are the ones a first pass at "must be relative" lets through: `../agent-box` escapes the home, `.` and `""` collapse to `/home/` itself — which the agent unit's `ProtectSystem=strict` would refuse as EROFS inside a background job's journal — and `a//b`, which resolves to a perfectly ordinary child path and is refused for a different reason: every empty component is, because one is how the collapsing cases are spelled. +- `nix build -L .#checks..webhook-defer` runs `tests/test-webhook-defer.sh`: what `modules/src/webhook-spawn.sh` answers at the hook-session ceiling, and what the pinned `webhook.py` does with that answer (issues #170, #301). A refusal is `exit 75` (`EX_TEMPFAIL`), the one code the dispatcher reads as "declined for now" rather than "this spawner is broken" — every other code drops the batch, and a standing watch is for events NO session owns, so nothing else is holding them. The second half runs the REAL wrapper as the REAL Dispatcher's spawn command, fills the cap, frees a slot and asserts the declined batch starts by itself, because the bug was the two programs disagreeing about what a non-zero exit meant. Runnable without Nix too: `bash tests/test-webhook-defer.sh modules/src/webhook-spawn.sh /path/to/webhook.py` (the dispatcher half is skipped, and says so, when no `webhook.py` is given). - `nix flake metadata` validates flake inputs and basic evaluation. - `nix build .#packages.x86_64-linux.vm` builds the bootable qcow2 image under `result/`. - `nix build -L .#checks..multi-user` runs the quick module/configuration assertion. @@ -264,7 +265,7 @@ get written in — the desktop merely looks a little uneven. Use `pkgs.testers.runNixOSTest` for service and VM behavior; name tests after the capability under test. Use Playwright `*.spec.ts` files only for behavior requiring a real browser or deployed instance. Add regression coverage with each behavioral fix. There is no numeric coverage threshold; CI expects every relevant named flake check to pass. -A VM test script has a hard ceiling of 128 KiB. nixpkgs passes it to the driver build in one environment variable and Linux caps a single environment string at `MAX_ARG_STRLEN`, so the test past that limit fails to build with `error: executing '.../bin/bash': Argument list too long` and no VM boots -- nothing in that message names the test script or its size. The `testscript-fits` check asserts the limit a page early so the failure is legible. When it fires, the answer is not shorter comments: move the assertions that do not need a VM into a native `runCommand` check (`webhook-spawn-claim` is one such move, out of `tests/webhook.nix`), or split the test the way `tests/sessions-common.nix` split the session tests. +A VM test script has a hard ceiling of 128 KiB. nixpkgs passes it to the driver build in one environment variable and Linux caps a single environment string at `MAX_ARG_STRLEN`, so the test past that limit fails to build with `error: executing '.../bin/bash': Argument list too long` and no VM boots -- nothing in that message names the test script or its size. The `testscript-fits` check asserts the limit a page early so the failure is legible. When it fires, the answer is not shorter comments: move the assertions that do not need a VM into a native `runCommand` check (`webhook-spawn-claim` and `webhook-defer` are two such moves, out of `tests/webhook.nix`), or split the test the way `tests/sessions-common.nix` split the session tests. Never end a test pipeline with `grep -q`. The driver runs each command under `set -euo pipefail`, and `grep -q` exits on the FIRST match — the producer upstream then gets EPIPE, and its non-zero status fails the whole assertion even though the pattern matched (a `must succeed` that reports exit 123 with `write error: Broken pipe` in the log). Write `… | grep PATTERN >/dev/null` instead, which drains the input, or capture to a file first and grep the file. `grep -q` is safe only with a file operand. diff --git a/README.md b/README.md index 44ac95a7..601ad11c 100644 --- a/README.md +++ b/README.md @@ -562,8 +562,11 @@ On by default (`webhook.enable`), needs `web.enable`. How it fits together: coalesce into one session; concurrent spawns are capped, and the wrapper refuses to accumulate more than a handful of live `hook-*` sessions — counted as sessions that are actually running, so a finished one frees its slot even - if nobody delisted it. The spawned session's prompt tells it to remove itself - when done. + if nobody delisted it. A batch that arrives at that ceiling is **queued, not + dropped**: the wrapper declines it with `EX_TEMPFAIL` and the receiver + re-offers it as slots free (bounded by `LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S`, an + hour here). The spawned session's prompt tells it to remove itself when + done. - The **settings page**'s **Webhook** panel carries both halves of what a sender's form asks for — the payload URL per configured source and that source's secret — each with a **copy button**, so registering a webhook needs diff --git a/bin/agentbox b/bin/agentbox index 3c99e010..e207f52e 100755 --- a/bin/agentbox +++ b/bin/agentbox @@ -2158,8 +2158,17 @@ class Renderer: if self.spec.webhook_enable: t.file(self.p("/etc/agent-box/units", f"agent-box-webhook-{u.name}.env"), - env_file([("LOCAL_WEBHOOK_STATE_DIR", - f"{u.home}/.local/state/local-webhook")])) + env_file([ + ("LOCAL_WEBHOOK_STATE_DIR", + f"{u.home}/.local/state/local-webhook"), + # How long a batch the hook-session cap declined + # (exit 75, modules/src/webhook-spawn.sh) may wait + # for a slot before the receiver drops it. + # local-webhook's own default is 300s, which is + # shorter than a hook session runs — see the module's + # half of this file for the whole argument. + ("LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S", "3600"), + ])) def user_dropins(self, t, u): """Host-level content, per instance. diff --git a/flake.nix b/flake.nix index 21716c7c..84873c06 100644 --- a/flake.nix +++ b/flake.nix @@ -1373,6 +1373,52 @@ open(sys.argv[3], "w").write(header + yaml.safe_dump(data, sort_keys=True))' \ cp log "$out" ''; + # What the spawn wrapper ANSWERS at the hook-session ceiling, and + # what the pinned local-webhook does with that answer (#170, #301). + # A refusal used to be `exit 1`, which the dispatcher cannot tell + # from "command not found", so it dropped the batch — and a standing + # watch is for events NO session owns, so nothing else held them. + # 75 (EX_TEMPFAIL) is the code that means "declined for now", and + # the batch is then re-offered until a slot frees. + # + # Both halves run against the REAL wrapper and the REAL pinned + # webhook.py, because the bug was the two disagreeing about what a + # non-zero exit meant — the wrapper is the dispatcher's own spawn + # command here, and the deferral is driven end to end in seconds. + # The VM test keeps the wiring an interpreter cannot show; this is + # where the contract between the two programs lives. + webhook-defer = + pkgs.runCommand "agent-box-webhook-defer" + { + nativeBuildInputs = [ + pkgs.bash + pkgs.coreutils + pkgs.jq + pkgs.python3 + ]; + # The whole directory, not the one file: the source form + # carries @@include markers and the test resolves them + # against its siblings, exactly as the assembler does. + src = ./modules/src; + tests = ./tests/test-webhook-defer.sh; + # The same pin the module and #runtime read, fetched the + # same way (nix/webhook-pin.nix). + webhookPy = + let pin = import ./nix/webhook-pin.nix; in + builtins.fetchurl { + url = "https://raw.githubusercontent.com/${pin.repo}/${pin.rev}" + + "/local-webhook/webhook.py"; + sha256 = pin.sha256; + }; + } '' + bash "$tests" "$src/webhook-spawn.sh" "$webhookPy" > log 2>&1 || { + cat log + exit 1 + } + cat log + cp log "$out" + ''; + # Unit tests for the durable per-session lease (issue #535): # outcome precedence (first ending wins, never the most recent), # clear's delete-not-blank resolution, and the read-only accessor diff --git a/modules/agent-box.nix b/modules/agent-box.nix index 03cb877e..c595b867 100644 --- a/modules/agent-box.nix +++ b/modules/agent-box.nix @@ -181,10 +181,14 @@ let never parked, so it stays listed and attachable for you to read - `rm` it once you have. That cleanup is load-bearing: at most 4 `hook-*` sessions may RUN at once, and once that ceiling is reached EVERY - watch on the box is inert - a matching batch is refused and dropped, never - queued. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook - status`: its `dispatch` object has the live count against the ceiling and the - last batch the ceiling dropped. + watch on the box is stalled - a matching batch starts nothing until a slot + frees. It is no longer LOST while it waits: the wrapper declines it and the + receiver keeps it, re-offers it as slots free, and drops it only after an hour + of waiting. A stopped session frees its slot even before it is delisted. So + before you conclude a repo has been quiet, run `agent-box-webhook status`: its + `dispatch` object has the live count against the ceiling and the last batch the + ceiling turned away (`lastRefusal.deferred` says whether that batch is still + waiting). Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on @@ -4312,12 +4316,16 @@ esac RUN at once (AGENT_BOX_HOOK_SESSION_MAX in the receiver daemon's environment). Hook sessions are removed by the agent they start, so four of them - still running wedge every watch on the box — a - refused batch is DROPPED, never queued. Stopping one - frees its slot; `agent-box-session rm NAME` also - delists it. `status` reports the count, the ceiling - and the last refusal, and `ls` says so too once the - box is at the ceiling. + still running stall every watch on the box. A batch + that arrives then is QUEUED, not dropped: the spawn + wrapper declines it (exit 75) and the receiver + re-offers it as slots free, giving up only after + LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + Stopping a session frees its slot; + `agent-box-session rm NAME` also delists it. + `status` reports the count, the ceiling and the last + refusal, and `ls` says so too once the box is at the + ceiling. --ignore-sender LOGIN mutes echoes of that sender's own comments and pushes ("@self" is $LOCAL_WEBHOOK_SELF — the login this box acts as, resolved from @@ -4400,9 +4408,11 @@ esac Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` - is the batch the ceiling most recently dropped (with a running total), and - `warning` — the same field `ls` sets when the receiver has no spawn command - — is present exactly when a match right now would spawn nothing. + is the batch the ceiling most recently turned away (with a running total, and + `deferred: true` when the receiver kept it to re-offer as a slot frees rather + than dropping it), and `warning` — the same field `ls` sets when the receiver + has no spawn command — is present exactly when a match right now would spawn + nothing. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -4535,11 +4545,16 @@ esac # ----------------------------------------------------- standing-watch cap --- # Standing watches are the one delivery shape with no session behind it, so # when the spawn wrapper refuses a batch (too many hook-* sessions running) - # webhook.py drops it and NOBODY got those events. That refusal used to - # reach only the receiver daemon journal while every listing here still said - # "subscribed" — four hook sessions whose agents forgot `agent-box-session rm` - # made the whole box inert and it read like a quiet week (issue #170). These - # three read the state the wrapper decides on, so `status` and `ls` can say it. + # nothing else is holding those events. That refusal used to reach only the + # receiver daemon journal while every listing here still said "subscribed" — + # four hook sessions whose agents forgot `agent-box-session rm` made the whole + # box inert and it read like a quiet week (issue #170). These three read the + # state the wrapper decides on, so `status` and `ls` can say it. + # + # Since #301 the refusal is a DEFERRAL (the wrapper exits 75), so the batch + # waits for a slot instead of dying at the cap — but the cap is still what an + # operator has to clear, and the wait is bounded, so the reporting below stays + # exactly as loud as it was. hook_sessions() { # The capacity in use, counted the way the wrapper counts it @@ -4588,14 +4603,18 @@ esac hook_capacity_warning() { # hook_capacity_warning LIVE MAX — the one sentence `status` and `ls` both - # print when the ceiling has made the watches inert, and nothing when it has + # print when the ceiling has stalled the watches, and nothing when it has # not. One wording in one place: two copies would drift, and this is the # sentence the reader acts on. [ "$1" -ge "$2" ] || return 0 printf '%s' "$1 of at most $2 hook-* sessions are running, so every \ - standing watch is inert: a matching event batch is refused and DROPPED, never \ - queued. Free a slot (agent-box-session ls, then agent-box-session stop NAME, \ - or agent-box-session rm NAME to delist it for good), or raise \ + standing watch is stalled: a matching event batch starts nothing now. The \ + spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ + as slots free and dropping it only once the wait outlasts \ + LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ + lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ + a slot (agent-box-session ls, then agent-box-session stop NAME, or \ + agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." } @@ -6643,18 +6662,38 @@ fi # # The cap itself is right — webhook.py rate-limits and coalesces spawns but # bounds nothing over time, so agents that forget their `agent-box-session rm` -# would otherwise fill the box. What was wrong is where the news went: on the -# refusal below webhook.py DROPS the batch (deliberately — retrying a broken -# spawner would loop), and for a standing watch there is no session peer that -# received those events anyway, so the loss is total. Its only trace was the -# receiver daemon's journal, while `agent-box-webhook ls` and `status` kept -# reporting a healthy subscription: four wedged hook sessions made every watch -# on the box inert, and that reads exactly like a quiet repo (issue #170). +# would otherwise fill the box. What was wrong was the ANSWER it gave. A +# refusal used to print a message and `exit 1`, which the dispatcher cannot +# tell apart from "command not found", so it dropped the batch for good — and +# for a standing watch there is no session peer that received those events +# anyway, so the loss was total. Its only trace was the receiver daemon's +# journal, while `agent-box-webhook ls` and `status` kept reporting a healthy +# subscription: four wedged hook sessions made every watch on the box inert, +# and that reads exactly like a quiet repo (issue #170). +# +# The exit code is a three-way answer since local-webhook 0.16.0 +# (local-channels#28, agent-box#301): 0 accepted, 75 (EX_TEMPFAIL) declines +# for NOW, anything else says the spawner is broken. Only the last drops the +# batch. So the cap exits 75 and nothing else in this script does — a +# malformed AGENT_BOX_HOOK_SESSION_ARGS or a failed `add` really is a broken +# spawner, and re-offering those would loop. A declined batch goes back at the +# head of its key's pending list, is re-offered as the rate window reopens +# (LOCAL_WEBHOOK_SPAWN_WINDOW, 60s) and starts the moment a slot frees, with +# every line re-checked against live session ownership first. It is dropped +# only if the whole streak outlasts LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — which +# the receiver unit raises well past the upstream 300s default, because a hook +# session runs for tens of minutes and five is not a wait, it is a slower drop. # -# So a refusal is written down where the CLI can find it, next to the other -# per-user agent-box state. Cumulative, never cleared: the dropped batches do -# not come back, so "5 dropped, the last one 20 minutes ago" is the standing -# fact an agent needs, not something to forget on the next successful spawn. +# On a box pinned to local-webhook < 0.16.0 the 75 reads as a broken spawner +# and the batch is dropped exactly as it was before, so this is never worse +# than what it replaces. +# +# Either way the refusal is written down where the CLI can find it, next to +# the other per-user agent-box state. Cumulative, never cleared: "5 refused, +# the last one 20 minutes ago" is the standing fact an agent needs, not +# something to forget on the next successful spawn. `deferred` records which +# answer this wrapper gave, so `status` does not report as lost a batch the +# receiver is still holding. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -6675,7 +6714,7 @@ record_refusal() { # $1 = the used hook-* capacity that triggered the refusal — the same number # the message above printed, which is what is running or queued to start and # NOT the raw registry key count (issue #280). Best effort: a state file that - # cannot be written must not turn a dropped batch into a crashed spawner, so + # cannot be written must not turn a refused batch into a crashed spawner, so # every failure here is silent and the journal line above stays the fallback. mkdir -p "$BOX_STATE" 2>/dev/null || return 0 prev=0 @@ -6691,7 +6730,7 @@ record_refusal() { --arg topic "''${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "''${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). The batch was DROPPED, not queued. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then @@ -6763,7 +6802,8 @@ if [ -s "$REGISTRY_FILE" ]; then fi if [ "$used" -ge "$MAX" ]; then echo "agent-box-webhook-spawn: $used hook-* sessions are running or queued to" \ - "start (max $MAX); dropping this batch — 'agent-box-session ls' shows" \ + "start (max $MAX); declining this batch for now — the receiver keeps it" \ + "and offers it again when a slot frees. 'agent-box-session ls' shows" \ "which; stopping one frees its slot and 'agent-box-session rm NAME'" \ "delists it for good" >&2 # The number recorded is the number refused on: `agent-box-webhook status` @@ -6771,7 +6811,10 @@ if [ -s "$REGISTRY_FILE" ]; then record_refusal "$used" echo "agent-box-webhook-spawn: recorded in $REFUSED;" \ "'agent-box-webhook status' reports it" >&2 - exit 1 + # 75, not 1: EX_TEMPFAIL is the only code the dispatcher reads as "declined + # for now" rather than "this spawner is broken" (agent-box#301). Every + # other exit in this script stays what it was. + exit 75 fi fi @@ -20027,7 +20070,21 @@ if __name__ == "__main__": ); }) terminalUsers) // lib.optionalAttrs webhookEnabled (lib.listToAttrs (map (name: lib.nameValuePair "agent-box/units/agent-box-webhook-${name}.env" { - text = "LOCAL_WEBHOOK_STATE_DIR=${webhookStateDirOf name}\n"; + text = "LOCAL_WEBHOOK_STATE_DIR=${webhookStateDirOf name}\n" + # How long a batch the hook-session cap declined (exit 75, see + # src/webhook-spawn.sh) may wait for a slot before the receiver + # gives up and drops it. local-webhook's own default is 300s, + # chosen for a consumer that declines briefly; ours is a session + # ceiling, and a hook session runs for tens of minutes, so five + # would mean the batch is dropped anyway — a slower version of + # the bug #301 set out to fix. An hour is the other bound: past + # it the event has usually been overtaken (the PR merged, the + # branch re-run), and the pending list is trimmed to + # LOCAL_WEBHOOK_SPAWN_PENDING_MAX lines meanwhile, so the wait + # costs a bounded amount of memory and no duplicate sessions. + # Overridable per box in agent-box-webhook-.local.env, + # which this file's unit reads after this one. + + "LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600\n"; }) terminalUsers)); # ttyd/settings/webhook per terminal user. The unit text (see @@ -20219,8 +20276,9 @@ if __name__ == "__main__": # ExecStartPre that enforces the governed watch policy. Once nix # garbage collection reclaims that generation the spawn command is # simply gone, and every matching standing-watch batch is DROPPED - # (never queued, agent-box#301) with the reason only in the - # root-owned journal. The window opens on essentially every + # with the reason only in the root-owned journal — the deferral + # #301 added does not save this one, because a command that cannot + # be run is a broken spawner and not a decline. The window opens on essentially every # rebuild, since a store-path bump is enough to change this # drop-in — and an update is exactly when a merge's own CI # deliveries are in flight. diff --git a/modules/agent-box.nix.in b/modules/agent-box.nix.in index 088512b5..482474d4 100644 --- a/modules/agent-box.nix.in +++ b/modules/agent-box.nix.in @@ -3375,7 +3375,21 @@ in ); }) terminalUsers) // lib.optionalAttrs webhookEnabled (lib.listToAttrs (map (name: lib.nameValuePair "agent-box/units/agent-box-webhook-${name}.env" { - text = "LOCAL_WEBHOOK_STATE_DIR=${webhookStateDirOf name}\n"; + text = "LOCAL_WEBHOOK_STATE_DIR=${webhookStateDirOf name}\n" + # How long a batch the hook-session cap declined (exit 75, see + # src/webhook-spawn.sh) may wait for a slot before the receiver + # gives up and drops it. local-webhook's own default is 300s, + # chosen for a consumer that declines briefly; ours is a session + # ceiling, and a hook session runs for tens of minutes, so five + # would mean the batch is dropped anyway — a slower version of + # the bug #301 set out to fix. An hour is the other bound: past + # it the event has usually been overtaken (the PR merged, the + # branch re-run), and the pending list is trimmed to + # LOCAL_WEBHOOK_SPAWN_PENDING_MAX lines meanwhile, so the wait + # costs a bounded amount of memory and no duplicate sessions. + # Overridable per box in agent-box-webhook-.local.env, + # which this file's unit reads after this one. + + "LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600\n"; }) terminalUsers)); # ttyd/settings/webhook per terminal user. The unit text (see @@ -3567,8 +3581,9 @@ in # ExecStartPre that enforces the governed watch policy. Once nix # garbage collection reclaims that generation the spawn command is # simply gone, and every matching standing-watch batch is DROPPED - # (never queued, agent-box#301) with the reason only in the - # root-owned journal. The window opens on essentially every + # with the reason only in the root-owned journal — the deferral + # #301 added does not save this one, because a command that cannot + # be run is a broken spawner and not a decline. The window opens on essentially every # rebuild, since a store-path bump is enough to change this # drop-in — and an update is exactly when a merge's own CI # deliveries are in flight. diff --git a/modules/src/default-agents-webhook.md b/modules/src/default-agents-webhook.md index a6ac16eb..6d461ed0 100644 --- a/modules/src/default-agents-webhook.md +++ b/modules/src/default-agents-webhook.md @@ -141,10 +141,14 @@ sooner. What is NOT reaped is a hook session that CRASHED: a non-zero exit is never parked, so it stays listed and attachable for you to read - `rm` it once you have. That cleanup is load-bearing: at most 4 `hook-*` sessions may RUN at once, and once that ceiling is reached EVERY -watch on the box is inert - a matching batch is refused and dropped, never -queued. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook -status`: its `dispatch` object has the live count against the ceiling and the -last batch the ceiling dropped. +watch on the box is stalled - a matching batch starts nothing until a slot +frees. It is no longer LOST while it waits: the wrapper declines it and the +receiver keeps it, re-offers it as slots free, and drops it only after an hour +of waiting. A stopped session frees its slot even before it is delisted. So +before you conclude a repo has been quiet, run `agent-box-webhook status`: its +`dispatch` object has the live count against the ceiling and the last batch the +ceiling turned away (`lastRefusal.deferred` says whether that batch is still +waiting). Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/modules/src/webhook-cli.sh b/modules/src/webhook-cli.sh index f3e071ef..e9ad1f35 100644 --- a/modules/src/webhook-cli.sh +++ b/modules/src/webhook-cli.sh @@ -77,12 +77,16 @@ Two delivery shapes: RUN at once (AGENT_BOX_HOOK_SESSION_MAX in the receiver daemon's environment). Hook sessions are removed by the agent they start, so four of them - still running wedge every watch on the box — a - refused batch is DROPPED, never queued. Stopping one - frees its slot; `agent-box-session rm NAME` also - delists it. `status` reports the count, the ceiling - and the last refusal, and `ls` says so too once the - box is at the ceiling. + still running stall every watch on the box. A batch + that arrives then is QUEUED, not dropped: the spawn + wrapper declines it (exit 75) and the receiver + re-offers it as slots free, giving up only after + LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + Stopping a session frees its slot; + `agent-box-session rm NAME` also delists it. + `status` reports the count, the ceiling and the last + refusal, and `ls` says so too once the box is at the + ceiling. --ignore-sender LOGIN mutes echoes of that sender's own comments and pushes ("@self" is $LOCAL_WEBHOOK_SELF — the login this box acts as, resolved from @@ -165,9 +169,11 @@ branch — and is reported in the JSON (`plugin.skew`) without a warning. Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` -is the batch the ceiling most recently dropped (with a running total), and -`warning` — the same field `ls` sets when the receiver has no spawn command -— is present exactly when a match right now would spawn nothing. +is the batch the ceiling most recently turned away (with a running total, and +`deferred: true` when the receiver kept it to re-offer as a slot frees rather +than dropping it), and `warning` — the same field `ls` sets when the receiver +has no spawn command — is present exactly when a match right now would spawn +nothing. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -300,11 +306,16 @@ peer_kinds() { # ----------------------------------------------------- standing-watch cap --- # Standing watches are the one delivery shape with no session behind it, so # when the spawn wrapper refuses a batch (too many hook-* sessions running) -# webhook.py drops it and NOBODY got those events. That refusal used to -# reach only the receiver daemon journal while every listing here still said -# "subscribed" — four hook sessions whose agents forgot `agent-box-session rm` -# made the whole box inert and it read like a quiet week (issue #170). These -# three read the state the wrapper decides on, so `status` and `ls` can say it. +# nothing else is holding those events. That refusal used to reach only the +# receiver daemon journal while every listing here still said "subscribed" — +# four hook sessions whose agents forgot `agent-box-session rm` made the whole +# box inert and it read like a quiet week (issue #170). These three read the +# state the wrapper decides on, so `status` and `ls` can say it. +# +# Since #301 the refusal is a DEFERRAL (the wrapper exits 75), so the batch +# waits for a slot instead of dying at the cap — but the cap is still what an +# operator has to clear, and the wait is bounded, so the reporting below stays +# exactly as loud as it was. hook_sessions() { # The capacity in use, counted the way the wrapper counts it @@ -353,14 +364,18 @@ dispatch_topics() { hook_capacity_warning() { # hook_capacity_warning LIVE MAX — the one sentence `status` and `ls` both - # print when the ceiling has made the watches inert, and nothing when it has + # print when the ceiling has stalled the watches, and nothing when it has # not. One wording in one place: two copies would drift, and this is the # sentence the reader acts on. [ "$1" -ge "$2" ] || return 0 printf '%s' "$1 of at most $2 hook-* sessions are running, so every \ -standing watch is inert: a matching event batch is refused and DROPPED, never \ -queued. Free a slot (agent-box-session ls, then agent-box-session stop NAME, \ -or agent-box-session rm NAME to delist it for good), or raise \ +standing watch is stalled: a matching event batch starts nothing now. The \ +spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ +as slots free and dropping it only once the wait outlasts \ +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ +lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ +a slot (agent-box-session ls, then agent-box-session stop NAME, or \ +agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." } diff --git a/modules/src/webhook-spawn.sh b/modules/src/webhook-spawn.sh index 8010ba55..ff98010a 100644 --- a/modules/src/webhook-spawn.sh +++ b/modules/src/webhook-spawn.sh @@ -415,18 +415,38 @@ fi # # The cap itself is right — webhook.py rate-limits and coalesces spawns but # bounds nothing over time, so agents that forget their `agent-box-session rm` -# would otherwise fill the box. What was wrong is where the news went: on the -# refusal below webhook.py DROPS the batch (deliberately — retrying a broken -# spawner would loop), and for a standing watch there is no session peer that -# received those events anyway, so the loss is total. Its only trace was the -# receiver daemon's journal, while `agent-box-webhook ls` and `status` kept -# reporting a healthy subscription: four wedged hook sessions made every watch -# on the box inert, and that reads exactly like a quiet repo (issue #170). -# -# So a refusal is written down where the CLI can find it, next to the other -# per-user agent-box state. Cumulative, never cleared: the dropped batches do -# not come back, so "5 dropped, the last one 20 minutes ago" is the standing -# fact an agent needs, not something to forget on the next successful spawn. +# would otherwise fill the box. What was wrong was the ANSWER it gave. A +# refusal used to print a message and `exit 1`, which the dispatcher cannot +# tell apart from "command not found", so it dropped the batch for good — and +# for a standing watch there is no session peer that received those events +# anyway, so the loss was total. Its only trace was the receiver daemon's +# journal, while `agent-box-webhook ls` and `status` kept reporting a healthy +# subscription: four wedged hook sessions made every watch on the box inert, +# and that reads exactly like a quiet repo (issue #170). +# +# The exit code is a three-way answer since local-webhook 0.16.0 +# (local-channels#28, agent-box#301): 0 accepted, 75 (EX_TEMPFAIL) declines +# for NOW, anything else says the spawner is broken. Only the last drops the +# batch. So the cap exits 75 and nothing else in this script does — a +# malformed AGENT_BOX_HOOK_SESSION_ARGS or a failed `add` really is a broken +# spawner, and re-offering those would loop. A declined batch goes back at the +# head of its key's pending list, is re-offered as the rate window reopens +# (LOCAL_WEBHOOK_SPAWN_WINDOW, 60s) and starts the moment a slot frees, with +# every line re-checked against live session ownership first. It is dropped +# only if the whole streak outlasts LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — which +# the receiver unit raises well past the upstream 300s default, because a hook +# session runs for tens of minutes and five is not a wait, it is a slower drop. +# +# On a box pinned to local-webhook < 0.16.0 the 75 reads as a broken spawner +# and the batch is dropped exactly as it was before, so this is never worse +# than what it replaces. +# +# Either way the refusal is written down where the CLI can find it, next to +# the other per-user agent-box state. Cumulative, never cleared: "5 refused, +# the last one 20 minutes ago" is the standing fact an agent needs, not +# something to forget on the next successful spawn. `deferred` records which +# answer this wrapper gave, so `status` does not report as lost a batch the +# receiver is still holding. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -447,7 +467,7 @@ record_refusal() { # $1 = the used hook-* capacity that triggered the refusal — the same number # the message above printed, which is what is running or queued to start and # NOT the raw registry key count (issue #280). Best effort: a state file that - # cannot be written must not turn a dropped batch into a crashed spawner, so + # cannot be written must not turn a refused batch into a crashed spawner, so # every failure here is silent and the journal line above stays the fallback. mkdir -p "$BOX_STATE" 2>/dev/null || return 0 prev=0 @@ -463,7 +483,7 @@ record_refusal() { --arg topic "${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). The batch was DROPPED, not queued. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then @@ -535,7 +555,8 @@ if [ -s "$REGISTRY_FILE" ]; then fi if [ "$used" -ge "$MAX" ]; then echo "agent-box-webhook-spawn: $used hook-* sessions are running or queued to" \ - "start (max $MAX); dropping this batch — 'agent-box-session ls' shows" \ + "start (max $MAX); declining this batch for now — the receiver keeps it" \ + "and offers it again when a slot frees. 'agent-box-session ls' shows" \ "which; stopping one frees its slot and 'agent-box-session rm NAME'" \ "delists it for good" >&2 # The number recorded is the number refused on: `agent-box-webhook status` @@ -543,7 +564,10 @@ if [ -s "$REGISTRY_FILE" ]; then record_refusal "$used" echo "agent-box-webhook-spawn: recorded in $REFUSED;" \ "'agent-box-webhook status' reports it" >&2 - exit 1 + # 75, not 1: EX_TEMPFAIL is the only code the dispatcher reads as "declined + # for now" rather than "this spawner is broken" (agent-box#301). Every + # other exit in this script stays what it was. + exit 75 fi fi diff --git a/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md b/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md index ae3fe6ae..b384f7f6 100644 --- a/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md +++ b/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md @@ -316,10 +316,14 @@ sooner. What is NOT reaped is a hook session that CRASHED: a non-zero exit is never parked, so it stays listed and attachable for you to read - `rm` it once you have. That cleanup is load-bearing: at most 4 `hook-*` sessions may RUN at once, and once that ceiling is reached EVERY -watch on the box is inert - a matching batch is refused and dropped, never -queued. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook -status`: its `dispatch` object has the live count against the ceiling and the -last batch the ceiling dropped. +watch on the box is stalled - a matching batch starts nothing until a slot +frees. It is no longer LOST while it waits: the wrapper declines it and the +receiver keeps it, re-offers it as slots free, and drops it only after an hour +of waiting. A stopped session frees its slot even before it is delisted. So +before you conclude a repo has been quiet, run `agent-box-webhook status`: its +`dispatch` object has the live count against the ceiling and the last batch the +ceiling turned away (`lastRefusal.deferred` says whether that batch is still +waiting). Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/tests/golden/web/etc/agent-box/units/agent-box-webhook-agent.env b/tests/golden/web/etc/agent-box/units/agent-box-webhook-agent.env index ecfa4ab9..77c64334 100644 --- a/tests/golden/web/etc/agent-box/units/agent-box-webhook-agent.env +++ b/tests/golden/web/etc/agent-box/units/agent-box-webhook-agent.env @@ -1 +1,2 @@ LOCAL_WEBHOOK_STATE_DIR=/home/agent/.local/state/local-webhook +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600 diff --git a/tests/golden/web/etc/agent-box/units/agent-box-webhook-robot.env b/tests/golden/web/etc/agent-box/units/agent-box-webhook-robot.env index 9c4abd6d..7ef551d0 100644 --- a/tests/golden/web/etc/agent-box/units/agent-box-webhook-robot.env +++ b/tests/golden/web/etc/agent-box/units/agent-box-webhook-robot.env @@ -1 +1,2 @@ LOCAL_WEBHOOK_STATE_DIR=/home/robot/.local/state/local-webhook +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600 diff --git a/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn b/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn index 1724e5dd..9d4d1f34 100644 --- a/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn +++ b/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn @@ -901,18 +901,38 @@ fi # # The cap itself is right — webhook.py rate-limits and coalesces spawns but # bounds nothing over time, so agents that forget their `agent-box-session rm` -# would otherwise fill the box. What was wrong is where the news went: on the -# refusal below webhook.py DROPS the batch (deliberately — retrying a broken -# spawner would loop), and for a standing watch there is no session peer that -# received those events anyway, so the loss is total. Its only trace was the -# receiver daemon's journal, while `agent-box-webhook ls` and `status` kept -# reporting a healthy subscription: four wedged hook sessions made every watch -# on the box inert, and that reads exactly like a quiet repo (issue #170). +# would otherwise fill the box. What was wrong was the ANSWER it gave. A +# refusal used to print a message and `exit 1`, which the dispatcher cannot +# tell apart from "command not found", so it dropped the batch for good — and +# for a standing watch there is no session peer that received those events +# anyway, so the loss was total. Its only trace was the receiver daemon's +# journal, while `agent-box-webhook ls` and `status` kept reporting a healthy +# subscription: four wedged hook sessions made every watch on the box inert, +# and that reads exactly like a quiet repo (issue #170). # -# So a refusal is written down where the CLI can find it, next to the other -# per-user agent-box state. Cumulative, never cleared: the dropped batches do -# not come back, so "5 dropped, the last one 20 minutes ago" is the standing -# fact an agent needs, not something to forget on the next successful spawn. +# The exit code is a three-way answer since local-webhook 0.16.0 +# (local-channels#28, agent-box#301): 0 accepted, 75 (EX_TEMPFAIL) declines +# for NOW, anything else says the spawner is broken. Only the last drops the +# batch. So the cap exits 75 and nothing else in this script does — a +# malformed AGENT_BOX_HOOK_SESSION_ARGS or a failed `add` really is a broken +# spawner, and re-offering those would loop. A declined batch goes back at the +# head of its key's pending list, is re-offered as the rate window reopens +# (LOCAL_WEBHOOK_SPAWN_WINDOW, 60s) and starts the moment a slot frees, with +# every line re-checked against live session ownership first. It is dropped +# only if the whole streak outlasts LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — which +# the receiver unit raises well past the upstream 300s default, because a hook +# session runs for tens of minutes and five is not a wait, it is a slower drop. +# +# On a box pinned to local-webhook < 0.16.0 the 75 reads as a broken spawner +# and the batch is dropped exactly as it was before, so this is never worse +# than what it replaces. +# +# Either way the refusal is written down where the CLI can find it, next to +# the other per-user agent-box state. Cumulative, never cleared: "5 refused, +# the last one 20 minutes ago" is the standing fact an agent needs, not +# something to forget on the next successful spawn. `deferred` records which +# answer this wrapper gave, so `status` does not report as lost a batch the +# receiver is still holding. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -933,7 +953,7 @@ record_refusal() { # $1 = the used hook-* capacity that triggered the refusal — the same number # the message above printed, which is what is running or queued to start and # NOT the raw registry key count (issue #280). Best effort: a state file that - # cannot be written must not turn a dropped batch into a crashed spawner, so + # cannot be written must not turn a refused batch into a crashed spawner, so # every failure here is silent and the journal line above stays the fallback. mkdir -p "$BOX_STATE" 2>/dev/null || return 0 prev=0 @@ -949,7 +969,7 @@ record_refusal() { --arg topic "${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). The batch was DROPPED, not queued. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then @@ -1021,7 +1041,8 @@ if [ -s "$REGISTRY_FILE" ]; then fi if [ "$used" -ge "$MAX" ]; then echo "agent-box-webhook-spawn: $used hook-* sessions are running or queued to" \ - "start (max $MAX); dropping this batch — 'agent-box-session ls' shows" \ + "start (max $MAX); declining this batch for now — the receiver keeps it" \ + "and offers it again when a slot frees. 'agent-box-session ls' shows" \ "which; stopping one frees its slot and 'agent-box-session rm NAME'" \ "delists it for good" >&2 # The number recorded is the number refused on: `agent-box-webhook status` @@ -1029,7 +1050,10 @@ if [ -s "$REGISTRY_FILE" ]; then record_refusal "$used" echo "agent-box-webhook-spawn: recorded in $REFUSED;" \ "'agent-box-webhook status' reports it" >&2 - exit 1 + # 75, not 1: EX_TEMPFAIL is the only code the dispatcher reads as "declined + # for now" rather than "this spawner is broken" (agent-box#301). Every + # other exit in this script stays what it was. + exit 75 fi fi diff --git a/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook b/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook index 17a68780..203ef46b 100644 --- a/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook +++ b/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook @@ -87,12 +87,16 @@ Two delivery shapes: RUN at once (AGENT_BOX_HOOK_SESSION_MAX in the receiver daemon's environment). Hook sessions are removed by the agent they start, so four of them - still running wedge every watch on the box — a - refused batch is DROPPED, never queued. Stopping one - frees its slot; `agent-box-session rm NAME` also - delists it. `status` reports the count, the ceiling - and the last refusal, and `ls` says so too once the - box is at the ceiling. + still running stall every watch on the box. A batch + that arrives then is QUEUED, not dropped: the spawn + wrapper declines it (exit 75) and the receiver + re-offers it as slots free, giving up only after + LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + Stopping a session frees its slot; + `agent-box-session rm NAME` also delists it. + `status` reports the count, the ceiling and the last + refusal, and `ls` says so too once the box is at the + ceiling. --ignore-sender LOGIN mutes echoes of that sender's own comments and pushes ("@self" is $LOCAL_WEBHOOK_SELF — the login this box acts as, resolved from @@ -175,9 +179,11 @@ branch — and is reported in the JSON (`plugin.skew`) without a warning. Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` -is the batch the ceiling most recently dropped (with a running total), and -`warning` — the same field `ls` sets when the receiver has no spawn command -— is present exactly when a match right now would spawn nothing. +is the batch the ceiling most recently turned away (with a running total, and +`deferred: true` when the receiver kept it to re-offer as a slot frees rather +than dropping it), and `warning` — the same field `ls` sets when the receiver +has no spawn command — is present exactly when a match right now would spawn +nothing. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -310,11 +316,16 @@ peer_kinds() { # ----------------------------------------------------- standing-watch cap --- # Standing watches are the one delivery shape with no session behind it, so # when the spawn wrapper refuses a batch (too many hook-* sessions running) -# webhook.py drops it and NOBODY got those events. That refusal used to -# reach only the receiver daemon journal while every listing here still said -# "subscribed" — four hook sessions whose agents forgot `agent-box-session rm` -# made the whole box inert and it read like a quiet week (issue #170). These -# three read the state the wrapper decides on, so `status` and `ls` can say it. +# nothing else is holding those events. That refusal used to reach only the +# receiver daemon journal while every listing here still said "subscribed" — +# four hook sessions whose agents forgot `agent-box-session rm` made the whole +# box inert and it read like a quiet week (issue #170). These three read the +# state the wrapper decides on, so `status` and `ls` can say it. +# +# Since #301 the refusal is a DEFERRAL (the wrapper exits 75), so the batch +# waits for a slot instead of dying at the cap — but the cap is still what an +# operator has to clear, and the wait is bounded, so the reporting below stays +# exactly as loud as it was. hook_sessions() { # The capacity in use, counted the way the wrapper counts it @@ -363,14 +374,18 @@ dispatch_topics() { hook_capacity_warning() { # hook_capacity_warning LIVE MAX — the one sentence `status` and `ls` both - # print when the ceiling has made the watches inert, and nothing when it has + # print when the ceiling has stalled the watches, and nothing when it has # not. One wording in one place: two copies would drift, and this is the # sentence the reader acts on. [ "$1" -ge "$2" ] || return 0 printf '%s' "$1 of at most $2 hook-* sessions are running, so every \ -standing watch is inert: a matching event batch is refused and DROPPED, never \ -queued. Free a slot (agent-box-session ls, then agent-box-session stop NAME, \ -or agent-box-session rm NAME to delist it for good), or raise \ +standing watch is stalled: a matching event batch starts nothing now. The \ +spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ +as slots free and dropping it only once the wait outlasts \ +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ +lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ +a slot (agent-box-session ls, then agent-box-session stop NAME, or \ +agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." } diff --git a/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md b/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md index 781fad17..da697031 100644 --- a/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md +++ b/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md @@ -353,10 +353,14 @@ sooner. What is NOT reaped is a hook session that CRASHED: a non-zero exit is never parked, so it stays listed and attachable for you to read - `rm` it once you have. That cleanup is load-bearing: at most 4 `hook-*` sessions may RUN at once, and once that ceiling is reached EVERY -watch on the box is inert - a matching batch is refused and dropped, never -queued. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook -status`: its `dispatch` object has the live count against the ceiling and the -last batch the ceiling dropped. +watch on the box is stalled - a matching batch starts nothing until a slot +frees. It is no longer LOST while it waits: the wrapper declines it and the +receiver keeps it, re-offers it as slots free, and drops it only after an hour +of waiting. A stopped session frees its slot even before it is delisted. So +before you conclude a repo has been quiet, run `agent-box-webhook status`: its +`dispatch` object has the live count against the ceiling and the last batch the +ceiling turned away (`lastRefusal.deferred` says whether that batch is still +waiting). Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md b/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md index 781fad17..da697031 100644 --- a/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md +++ b/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md @@ -353,10 +353,14 @@ sooner. What is NOT reaped is a hook session that CRASHED: a non-zero exit is never parked, so it stays listed and attachable for you to read - `rm` it once you have. That cleanup is load-bearing: at most 4 `hook-*` sessions may RUN at once, and once that ceiling is reached EVERY -watch on the box is inert - a matching batch is refused and dropped, never -queued. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook -status`: its `dispatch` object has the live count against the ceiling and the -last batch the ceiling dropped. +watch on the box is stalled - a matching batch starts nothing until a slot +frees. It is no longer LOST while it waits: the wrapper declines it and the +receiver keeps it, re-offers it as slots free, and drops it only after an hour +of waiting. A stopped session frees its slot even before it is delisted. So +before you conclude a repo has been quiet, run `agent-box-webhook status`: its +`dispatch` object has the live count against the ceiling and the last batch the +ceiling turned away (`lastRefusal.deferred` says whether that batch is still +waiting). Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/tests/native/expected/etc/agent-box/units/agent-box-webhook-agent.env b/tests/native/expected/etc/agent-box/units/agent-box-webhook-agent.env index b49bc40c..af456be2 100644 --- a/tests/native/expected/etc/agent-box/units/agent-box-webhook-agent.env +++ b/tests/native/expected/etc/agent-box/units/agent-box-webhook-agent.env @@ -1,2 +1,3 @@ # Generated by `agentbox apply` - do not edit. LOCAL_WEBHOOK_STATE_DIR=/home/agent/.local/state/local-webhook +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600 diff --git a/tests/native/expected/etc/agent-box/units/agent-box-webhook-robot.env b/tests/native/expected/etc/agent-box/units/agent-box-webhook-robot.env index 7d91847c..627a9927 100644 --- a/tests/native/expected/etc/agent-box/units/agent-box-webhook-robot.env +++ b/tests/native/expected/etc/agent-box/units/agent-box-webhook-robot.env @@ -1,2 +1,3 @@ # Generated by `agentbox apply` - do not edit. LOCAL_WEBHOOK_STATE_DIR=/home/robot/.local/state/local-webhook +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S=3600 diff --git a/tests/test-webhook-defer.sh b/tests/test-webhook-defer.sh new file mode 100755 index 00000000..fe1146b9 --- /dev/null +++ b/tests/test-webhook-defer.sh @@ -0,0 +1,238 @@ +#!/usr/bin/env bash +# What the spawn wrapper ANSWERS at the hook-session ceiling, and what the +# pinned local-webhook does with that answer (issues #170, #301). +# +# The cap itself is right: webhook.py rate-limits and coalesces spawns but +# bounds nothing over time, so agents that forget `agent-box-session rm` would +# otherwise fill the box. The answer was wrong. A refusal used to print a +# message and `exit 1`, which the dispatcher cannot tell apart from "command +# not found", so it dropped the batch — and a standing watch is for events NO +# session owns, so nothing else was holding them. Since local-channels#28 the +# exit code is a three-way answer, and 75 (EX_TEMPFAIL) is "declined for now": +# the batch goes back at the head of its key's pending list and is re-offered +# until a slot frees. +# +# Both halves are asserted here, against the REAL wrapper and the REAL pinned +# webhook.py, because the whole bug was the two disagreeing about what a +# non-zero exit meant. The VM test owns the wiring an interpreter cannot show +# (a signed delivery reaching a watch, a session landing in the registry); this +# owns the contract between the two programs, which costs a VM boot there and +# a few seconds here. +set -u + +SCRIPT=${1:?usage: test-webhook-defer.sh PATH/TO/webhook-spawn.sh [WEBHOOK.PY]} +WEBHOOK_PY=${2:-} +[ -f "$SCRIPT" ] || { echo "no such script: $SCRIPT" >&2; exit 2; } +SRC=$(cd "$(dirname "$SCRIPT")" && pwd) + +BASH_BIN=$(command -v bash) +work=$(mktemp -d) +trap 'rm -rf "$work"' EXIT + +# The source form carries @@include markers the assembler resolves; expand +# them the same way (nested includes and all) so the test drives the same text +# the box runs. +python3 - "$SRC" "$SCRIPT" "$work/spawn.sh" <<'PY' +import pathlib, re, sys +src, script, out = (pathlib.Path(p) for p in sys.argv[1:4]) +def expand(path, depth=0): + if depth > 8: + raise SystemExit(f"include loop at {path}") + lines = [] + for line in path.read_text().splitlines(keepends=True): + m = re.match(r"\s*@@include:(\S+)@@\s*$", line) + lines.append(expand(src / m.group(1), depth + 1) if m else line) + return "".join(lines) +out.write_text(expand(script)) +PY + +export HOME="$work/home"; mkdir -p "$HOME/.config/agent-box" +export LOCAL_WEBHOOK_STATE_DIR="$work/state"; mkdir -p "$LOCAL_WEBHOOK_STATE_DIR" +REGISTRY="$HOME/.config/agent-box/sessions.json" +REFUSED="$HOME/.local/state/agent-box/webhook-spawn-refused.json" + +mkdir -p "$work/bin" +# `add` is the wrapper's exec target and `peers` a read the preamble embeds. +# Neither is under test, so the shim records the call and succeeds — the +# recording is what says a spawn was ACCEPTED, since no session registry +# entry is written when `add` is a stub. +cat > "$work/bin/session" <> "$work/session.log" +exit 0 +EOF +# No hook profile and no extra args: the env store answers "unset" (rc 1). +cat > "$work/bin/envstore" < "$work/spawn.out" 2>&1 <<< 'hi' + printf '%s' "$?" +} + +# --- at the ceiling: declined, not failed ----------------------------------- +# +# 75 and 1 differ in nothing an operator sees and everything the receiver does, +# so the code is the assertion. Anything else in this script keeps its own +# status: a broken spawner must still be dropped rather than retried forever. +hook_registry 2 +rc=$(spawn defangdevs/blocked 2) +if [ "$rc" = 75 ]; then + ok "the hook-session cap exits 75 (EX_TEMPFAIL), not 1" +else + fail "the hook-session cap exits 75 (EX_TEMPFAIL), not 1 — got $rc" + sed 's/^/ /' "$work/spawn.out" +fi +if grep -q 'declining this batch for now' "$work/spawn.out"; then + ok "it says the batch is kept, not dropped" +else + fail "it says the batch is kept, not dropped" + sed 's/^/ /' "$work/spawn.out" +fi +if grep -q 'cannot ask tmux' "$work/spawn.out"; then + ok "a probe that cannot run falls back to the key count and says so" +else + fail "a probe that cannot run falls back to the key count and says so" +fi +if [ ! -s "$work/session.log" ]; then + ok "no session is started at the ceiling" +else + fail "no session is started at the ceiling" +fi + +# The record #170 leaves now carries WHICH answer was given, so `status` cannot +# report as lost a batch the receiver is still holding. +if jq -e '.count == 1 and .deferred == true and .live == 2 and .max == 2 + and .key == "defangdevs/blocked"' "$REFUSED" >/dev/null 2>&1; then + ok "the refusal is recorded as deferred, with the capacity it was refused on" +else + fail "the refusal is recorded as deferred, with the capacity it was refused on" + jq -c . "$REFUSED" 2>/dev/null | sed 's/^/ /' +fi + +# --- below the ceiling, and a ceiling nobody can read ------------------------ +hook_registry 1 +rc=$(spawn defangdevs/free 2) +if [ "$rc" = 0 ] && grep -q 'add hook-defangdevs-free-' "$work/session.log"; then + ok "a slot below the cap spawns" +else + fail "a slot below the cap spawns — exit $rc" + sed 's/^/ /' "$work/spawn.out" +fi +# A knob that --help documents is a knob someone will typo, and an unusable +# value must not refuse every batch for a reason nobody can see: `[ n -ge foo ]` +# is fatal under set -e, which would decline every batch on every box that +# typed it. It says so and falls back to the built-in 4. +: > "$work/session.log" +hook_registry 2 +rc=$(spawn defangdevs/typo lots) +if [ "$rc" = 0 ] && grep -q 'is not a number' "$work/spawn.out" \ + && grep -q 'add hook-defangdevs-typo-' "$work/session.log"; then + ok "an unusable AGENT_BOX_HOOK_SESSION_MAX says so and falls back to 4" +else + fail "an unusable AGENT_BOX_HOOK_SESSION_MAX says so and falls back to 4 — exit $rc" + sed 's/^/ /' "$work/spawn.out" +fi + +# --- and the dispatcher KEEPS what the wrapper declined ---------------------- +# +# The half the exit code exists for, driven through the pinned webhook.py's own +# Dispatcher with the REAL wrapper as its spawn command: at the cap the batch +# waits, and it starts by itself once a slot frees, with no second delivery to +# prompt it. Skipped rather than silently passed when no webhook.py was given. +if [ -z "$WEBHOOK_PY" ]; then + printf 'skip the dispatcher half (no webhook.py given)\n' +elif [ ! -f "$WEBHOOK_PY" ]; then + fail "webhook.py not found: $WEBHOOK_PY" +else + : > "$work/session.log" + hook_registry 1 + if AGENT_BOX_HOOK_SESSION_MAX=1 SPAWN="$BASH_BIN $work/spawn.sh" \ + REGISTRY="$REGISTRY" LOG="$work/session.log" \ + python3 - "$WEBHOOK_PY" > "$work/dispatch.out" 2>&1 <<'PY' +import importlib.util, json, os, sys, time + +spec = importlib.util.spec_from_file_location("webhook_pinned", sys.argv[1]) +mod = importlib.util.module_from_spec(spec) +spec.loader.exec_module(mod) + +log, registry = os.environ["LOG"], os.environ["REGISTRY"] +# window 1s so a re-offer costs a second rather than the 60 a box waits; +# owner_of is stubbed to "nobody owns it", the answer a box with no live peer +# sockets gives anyway, so the deferral is what decides the outcome here. +d = mod.Dispatcher(os.environ["SPAWN"], 1, 1, 30, + owner_of=lambda env: None, defer_max_s=60, pending_max=200) +meta = {"source": "github", "key": "defangdevs/deferred", "event": "issues", + "topic": "github:defangdevs/*", "note": "standing watch"} +d.add("defangdevs/deferred", "issue #30 opened on defangdevs/deferred", meta, {}) + +# At the cap: declined every time the window reopens, and never dropped. +deadline = time.monotonic() + 6 +while time.monotonic() < deadline: + time.sleep(0.25) +if os.path.getsize(log): + raise SystemExit("a session started while the cap was full: " + + open(log).read()) +st = d.keys[d._bucket("defangdevs/deferred", meta)] +if st["defer_n"] < 2: + raise SystemExit(f"the batch was offered {st['defer_n']} time(s), not re-offered") +# Pending OR in flight: a re-offer that is mid-decline has already taken the +# batch off the list, and which side of that the sample lands on is a race. +if len(st["pending"]) + (1 if st["running"] else 0) != 1: + raise SystemExit(f"the batch is not waiting: {st['pending']!r}") + +# A slot frees. Nothing new is delivered — the batch that was declined is the +# one that has to start. +json.dump({"version": 1, "sessions": {}}, open(registry, "w")) +deadline = time.monotonic() + 30 +while time.monotonic() < deadline and not os.path.getsize(log): + time.sleep(0.25) +started = open(log).read() +if "add hook-defangdevs-deferred-" not in started: + raise SystemExit(f"the deferred batch never started: {started!r}") +print("started:", started.strip()) +PY + then + ok "a batch declined at the cap starts by itself when a slot frees" + else + fail "a batch declined at the cap starts by itself when a slot frees" + sed 's/^/ /' "$work/dispatch.out" + fi +fi + +if [ "$fails" -eq 0 ]; then + printf '\nall webhook-defer assertions passed\n' +else + printf '\n%s webhook-defer assertion(s) failed\n' "$fails" + exit 1 +fi diff --git a/tests/webhook.nix b/tests/webhook.nix index 8d42e55d..46f6038c 100644 --- a/tests/webhook.nix +++ b/tests/webhook.nix @@ -2163,12 +2163,17 @@ # --- the hook-session ceiling is visible, not merely enforced (#170) ----- # A standing watch is the one delivery shape with no session behind it, so - # when the spawn wrapper refuses a batch webhook.py drops it and NOBODY got - # those events. That refusal used to reach the receiver daemon's journal - # and nowhere else, while `ls` and `status` kept reporting a healthy - # subscription: four hook sessions whose agents forgot `agent-box-session - # rm` made every watch on the box inert, and a wedged box read exactly like - # a quiet week. So a refusal is written down, and both listings say it. + # when the spawn wrapper refuses a batch nothing else holds those events. + # That refusal used to reach the receiver daemon's journal and nowhere + # else, while `ls` and `status` kept reporting a healthy subscription: four + # hook sessions whose agents forgot `agent-box-session rm` made every watch + # on the box inert, and read exactly like a quiet week. So a refusal is + # written down, and both listings say it. + # + # Since #301 the refusal is also a DEFERRAL: the wrapper answers 75, the + # one code local-webhook reads as "declined for now" instead of "this + # spawner is broken", so the exit code is asserted on every refusal below. + # What the receiver then does with it is `webhook-defer`, a native check. refused = "/home/agent/.local/state/agent-box/webhook-spawn-refused.json" def dispatch_status(cap="", expect=None): @@ -2226,8 +2231,8 @@ f" {sw}/sh -c 'echo hi | {spawn_cmd}' 2>&1" ) rc, refusal_log = machine.execute(drop_env) - assert rc == 1, (rc, refusal_log) - assert "dropping this batch" in refusal_log, refusal_log + assert rc == 75, (rc, refusal_log) + assert "declining this batch for now" in refusal_log, refusal_log # What the refusal is recorded against is what it was refused ON: since # #280 that is the capacity in USE — hook-* sessions running or queued to # start — and no longer the raw key count. Nothing here is `stopped`, so @@ -2238,14 +2243,16 @@ ][0] refused_used = int(refused_line.split(":", 1)[1].split()[0]) assert refused_used == live, (refusal_log, live) - # The batch is gone, not queued — no session, and the record is the only - # thing left of it. + # No session started — the cap is still a cap — and the record says the + # wrapper answered "not yet" rather than "never", so `status` cannot + # report as lost a batch the receiver is still holding. machine.fail( "jq -e '.sessions | keys[] | select(startswith(\"hook-defangdevs-dropped\"))'" " /home/agent/.config/agent-box/sessions.json" ) rec = json.loads(machine.succeed(f"cat {refused}")) assert rec["count"] == 1, rec + assert rec["deferred"] is True, rec assert rec["live"] == refused_used, (rec, refusal_log) assert rec["max"] == 1, rec assert rec["topic"] == "github:defangdevs/*", rec @@ -2259,9 +2266,10 @@ d = dispatch_status(cap="AGENT_BOX_HOOK_SESSION_MAX=1", expect="hook-* sessions are running") assert d["hookSessions"] == {"live": live, "max": 1, "atCapacity": True}, d - assert "DROPPED" in d["warning"], d + assert "DECLINES it (exit 75)" in d["warning"], d assert "agent-box-session rm NAME" in d["warning"], d assert d["lastRefusal"]["count"] == 1, d + assert d["lastRefusal"]["deferred"] is True, d # ls says it too — a listing of standing watches is where someone goes to # ask why nothing fires — but on stderr, so its stdout stays byte-for-byte @@ -2270,15 +2278,14 @@ f"{hookenv} AGENT_BOX_HOOK_SESSION_MAX=1 agent-box-webhook ls 2>/tmp/cap.err" ) ls_err = machine.succeed("cat /tmp/cap.err") - assert "every standing watch is inert" in ls_err, ls_err + assert "every standing watch is stalled" in ls_err, ls_err assert '"dispatch"' in ls_out, ls_out - assert "every standing watch is inert" not in ls_out, ls_out + assert "every standing watch is stalled" not in ls_out, ls_out - # Cumulative and never cleared: the dropped batches do not come back, so - # "N dropped since T" is the standing fact, not something the next spawn - # gets to forget. + # Cumulative and never cleared: a refused batch is at best late, so "N + # refused since T" is the standing fact, not something to forget. rc, _ = machine.execute(drop_env) - assert rc == 1, rc + assert rc == 75, rc rec2 = json.loads(machine.succeed(f"cat {refused}")) assert rec2["count"] == 2, rec2 assert rec2["firstAt"] == rec["firstAt"], (rec, rec2) @@ -2303,23 +2310,9 @@ machine.succeed(f"sudo -u agent env HOME=/home/agent agent-box-session rm {again}") machine.succeed(f"jq -e '.count == 2' {refused} >/dev/null") - # A knob that --help documents is a knob someone will typo, and an unusable - # value must not refuse every batch for a reason nobody can see: it says so - # and falls back to the built-in ceiling. - rc, typo_log = machine.execute( - "sudo -u agent env HOME=/home/agent" - " LOCAL_WEBHOOK_STATE_DIR=/home/agent/.local/state/local-webhook" - " AGENT_BOX_HOOK_SESSION_MAX=lots" - " LOCAL_WEBHOOK_SPAWN_SOURCE=github LOCAL_WEBHOOK_SPAWN_KEY=defangdevs/typo" - f" {sw}/sh -c 'echo hi | {spawn_cmd}' 2>&1" - ) - assert rc == 0, (rc, typo_log) - assert "is not a number" in typo_log, typo_log - typo = machine.succeed( - "jq -r '.sessions | keys[] | select(startswith(\"hook-defangdevs-typo-\"))'" - " /home/agent/.config/agent-box/sessions.json" - ).strip() - machine.succeed(f"sudo -u agent env HOME=/home/agent agent-box-session rm {typo}") + # An unusable AGENT_BOX_HOOK_SESSION_MAX falling back to the built-in + # ceiling is asserted in the `webhook-defer` native check instead: it is a + # pure function of one environment variable, and it cost a VM boot here. # --- the dispatch cap counts what RUNS, not registry keys (issue #280) --- # The cap used to count hook-* keys in sessions.json, and nothing expires a @@ -2366,7 +2359,13 @@ f" LOCAL_WEBHOOK_SPAWN_KEY=defangdevs/{key}" f" {sw}/sh -c 'echo hi | {spawn_cmd}' 2>&1" ) - return machine.succeed(cmd) if want else machine.fail(cmd) + if want: + return machine.succeed(cmd) + # 75 exactly: any other code drops the batch (#301), and a + # `machine.fail` would pass on the `exit 1` this replaced. + rc, out = machine.execute(cmd) + assert rc == 75, (rc, out) + return out def cap_session(key): return machine.succeed( @@ -2437,8 +2436,8 @@ ) machine.wait_until_fails(f"{hook_ls} | grep -x {done} >/dev/null", timeout=60) - # Two hook-* keys at MAX=2, only one of them running. The old count dropped - # this batch for good; the new one spends the slot the finished session was + # Two hook-* keys at MAX=2, only one of them running. The old count refused + # this batch; the new one spends the slot the finished session was # sitting on. (Asserted, not assumed: a stray coalesced dispatch landing # here would otherwise turn the arithmetic below into a puzzle.) assert machine.succeed(hook_keys).strip() == "2" @@ -2447,7 +2446,7 @@ machine.wait_until_succeeds(f"{hook_ls} | grep -x {free} >/dev/null", timeout=60) # ...and the brake is not simply gone: the SAME cap, with both slots now - # genuinely running, still drops the batch and says so — the only + # genuinely running, still refuses the batch and says so — the only # difference between this call and the one above is liveness, which is # therefore worth restating as this assertion's precondition. machine.succeed(f"{hook_ls} | grep -x {busy} >/dev/null") From c6e6151feb6efc15cb6ddcde947546c791a7d888 Mon Sep 17 00:00:00 2001 From: defangdevs Date: Sat, 5 Sep 2026 00:21:13 +0000 Subject: [PATCH 2/2] fix(webhook): say what lastRefusal.deferred is - a record, not a queue CodeRabbit on #584: `record_refusal` writes `deferred: true` when the wrapper answers 75, and nothing ever rewrites it. It stays true after the batch starts, and after the receiver gives up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S. So "says whether the last batch is waiting or was lost" was wrong in `status`, in `--help`, in the shipped guide and in the record's own `//` note. It records the ANSWER that batch got - declined for retry rather than a failure that drops it - and the wrapper is gone before either ending, so nothing there could keep live queue state honest. Said that way in all four places, with the "not a queue" reason attached where a reader would otherwise assume one. Also ASCII on the new AGENTS.md line, per the repo's Markdown rule. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01NWmLRsa4keJzBo9Minofxe --- AGENTS.md | 2 +- modules/agent-box.nix | 30 ++++++++++++------- modules/src/default-agents-webhook.md | 5 ++-- modules/src/webhook-cli.sh | 18 ++++++----- modules/src/webhook-spawn.sh | 7 +++-- .../web/etc/agent-box-guides/AGENTS.agent.md | 5 ++-- .../bin/agent-box-webhook-spawn | 7 +++-- .../agent-box-webhook/bin/agent-box-webhook | 18 ++++++----- .../etc/agent-box-guides/AGENTS.agent.md | 5 ++-- .../etc/agent-box-guides/AGENTS.robot.md | 5 ++-- 10 files changed, 64 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d1e9cbff..108359e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -64,7 +64,7 @@ Keep the module self-contained: deployed boxes fetch `modules/agent-box.nix` as - `nix build -L .#checks..checkout-bootstrap` runs `tests/test-checkout-bootstrap.sh` against `modules/src/checkout-cli.sh`, the script that puts this repo ON a deployed box (issue #242). Its assertions are mostly REFUSALS, because that is where the damage would be: it runs unattended at every supervisor start, in a tree sibling sessions are working in, so a realign that moved somebody's branch pointer would destroy work at boot on a box nobody is watching. `origin` is a local repository and `gh` is a shim, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-checkout-bootstrap.sh modules/src/checkout-cli.sh`. - `nix build -L .#checks..source-tree` runs `tests/test-source-tree.sh` against `modules/src/source-tree.sh`, the tree the box is BUILT from and the whole of what an update moves (issue #242). Weighted at the refusals for the same reason: it runs as root, unattended, and the tree it leaves behind is what the next rebuild builds — so a rewritten history, a downgrade, and a baseline the tree has never heard of each get an assertion, as does the realign that makes the fast-forward guard measure ancestry from the rev the box is RUNNING. It also pins the two locks the trust boundary rests on: `check` answers from `git ls-remote` and so creates no tree and fetches into none, and git runs with `core.hooksPath` pointed at nothing, so a `post-checkout`/`post-merge` hook in the tree cannot run as root (that assertion has a negative control — remove the lock and it fails). `origin` is a local repository, so there is no network and it runs natively on every architecture. Runnable without Nix too: `bash tests/test-source-tree.sh modules/src/source-tree.sh`. - `nix build -L .#checks..checkout-options` is the eval regression for `selfUpdate`'s three path assertions (issue #242). It reads `config.assertions` rather than forcing `toplevel`, so a failure names WHICH assertion fired instead of only reporting that something did — and it asserts the accepting cases too, so an assertion that rejects everything fails it as loudly as one that rejects nothing. `selfUpdate.srcDir` is the one that matters most, because root BUILDS the box from that tree: it is confined to a normalized path under `/var/lib`, since owning the directory is not enough — a writable ancestor (`/home/agent/src`, `/tmp/src`) lets an agent swap the whole tree and choose what root builds. For `selfUpdate.checkout.path` the inputs that matter are the ones a first pass at "must be relative" lets through: `../agent-box` escapes the home, `.` and `""` collapse to `/home/` itself — which the agent unit's `ProtectSystem=strict` would refuse as EROFS inside a background job's journal — and `a//b`, which resolves to a perfectly ordinary child path and is refused for a different reason: every empty component is, because one is how the collapsing cases are spelled. -- `nix build -L .#checks..webhook-defer` runs `tests/test-webhook-defer.sh`: what `modules/src/webhook-spawn.sh` answers at the hook-session ceiling, and what the pinned `webhook.py` does with that answer (issues #170, #301). A refusal is `exit 75` (`EX_TEMPFAIL`), the one code the dispatcher reads as "declined for now" rather than "this spawner is broken" — every other code drops the batch, and a standing watch is for events NO session owns, so nothing else is holding them. The second half runs the REAL wrapper as the REAL Dispatcher's spawn command, fills the cap, frees a slot and asserts the declined batch starts by itself, because the bug was the two programs disagreeing about what a non-zero exit meant. Runnable without Nix too: `bash tests/test-webhook-defer.sh modules/src/webhook-spawn.sh /path/to/webhook.py` (the dispatcher half is skipped, and says so, when no `webhook.py` is given). +- `nix build -L .#checks..webhook-defer` runs `tests/test-webhook-defer.sh`: what `modules/src/webhook-spawn.sh` answers at the hook-session ceiling, and what the pinned `webhook.py` does with that answer (issues #170, #301). A refusal is `exit 75` (`EX_TEMPFAIL`), the one code the dispatcher reads as "declined for now" rather than "this spawner is broken" -- every other code drops the batch, and a standing watch is for events NO session owns, so nothing else is holding them. The second half runs the REAL wrapper as the REAL Dispatcher's spawn command, fills the cap, frees a slot and asserts the declined batch starts by itself, because the bug was the two programs disagreeing about what a non-zero exit meant. Runnable without Nix too: `bash tests/test-webhook-defer.sh modules/src/webhook-spawn.sh /path/to/webhook.py` (the dispatcher half is skipped, and says so, when no `webhook.py` is given). - `nix flake metadata` validates flake inputs and basic evaluation. - `nix build .#packages.x86_64-linux.vm` builds the bootable qcow2 image under `result/`. - `nix build -L .#checks..multi-user` runs the quick module/configuration assertion. diff --git a/modules/agent-box.nix b/modules/agent-box.nix index c595b867..a469ed1d 100644 --- a/modules/agent-box.nix +++ b/modules/agent-box.nix @@ -187,8 +187,9 @@ let of waiting. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook status`: its `dispatch` object has the live count against the ceiling and the last batch the - ceiling turned away (`lastRefusal.deferred` says whether that batch is still - waiting). + ceiling turned away. `lastRefusal.deferred` records the ANSWER that batch got - + declined for retry rather than dropped - and stays true afterwards, so it is + history and never a list of what is waiting now. Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on @@ -4321,6 +4322,8 @@ esac wrapper declines it (exit 75) and the receiver re-offers it as slots free, giving up only after LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + `status` reports which answer the LAST refused batch + got, never whether it is still waiting. Stopping a session frees its slot; `agent-box-session rm NAME` also delists it. `status` reports the count, the ceiling and the last @@ -4408,11 +4411,14 @@ esac Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` - is the batch the ceiling most recently turned away (with a running total, and - `deferred: true` when the receiver kept it to re-offer as a slot frees rather - than dropping it), and `warning` — the same field `ls` sets when the receiver - has no spawn command — is present exactly when a match right now would spawn - nothing. + is the batch the ceiling most recently turned away (with a running total), and + `warning` — the same field `ls` sets when the receiver has no spawn command — + is present exactly when a match right now would spawn nothing. + `lastRefusal.deferred` records the ANSWER the wrapper gave that batch — 75, + declined for retry, rather than a failure that drops it. It is history, not a + queue: nothing rewrites it once the batch starts, or once the receiver gives + up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S. Whether a batch is waiting RIGHT + NOW is the receiver's own state and is not reported here. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -4611,8 +4617,7 @@ esac standing watch is stalled: a matching event batch starts nothing now. The \ spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ as slots free and dropping it only once the wait outlasts \ - LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ - lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ + LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix. Free \ a slot (agent-box-session ls, then agent-box-session stop NAME, or \ agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." @@ -6693,7 +6698,10 @@ fi # the last one 20 minutes ago" is the standing fact an agent needs, not # something to forget on the next successful spawn. `deferred` records which # answer this wrapper gave, so `status` does not report as lost a batch the -# receiver is still holding. +# receiver may still be holding. It is that answer and nothing more: this +# program is gone by the time the batch starts or is finally dropped, so +# nothing here could keep a live queue state honest, and the field must not be +# read as one. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -6730,7 +6738,7 @@ record_refusal() { --arg topic "''${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "''${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` records the ANSWER this wrapper gave that batch: 75, which local-webhook >= 0.16.0 reads as declined-for-retry rather than a failure that drops it (agent-box#301). It is history and is never rewritten, so it stays true after the batch starts or after the receiver gives up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S; its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then diff --git a/modules/src/default-agents-webhook.md b/modules/src/default-agents-webhook.md index 6d461ed0..6004a293 100644 --- a/modules/src/default-agents-webhook.md +++ b/modules/src/default-agents-webhook.md @@ -147,8 +147,9 @@ receiver keeps it, re-offers it as slots free, and drops it only after an hour of waiting. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook status`: its `dispatch` object has the live count against the ceiling and the last batch the -ceiling turned away (`lastRefusal.deferred` says whether that batch is still -waiting). +ceiling turned away. `lastRefusal.deferred` records the ANSWER that batch got - +declined for retry rather than dropped - and stays true afterwards, so it is +history and never a list of what is waiting now. Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/modules/src/webhook-cli.sh b/modules/src/webhook-cli.sh index e9ad1f35..06b93269 100644 --- a/modules/src/webhook-cli.sh +++ b/modules/src/webhook-cli.sh @@ -82,6 +82,8 @@ Two delivery shapes: wrapper declines it (exit 75) and the receiver re-offers it as slots free, giving up only after LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + `status` reports which answer the LAST refused batch + got, never whether it is still waiting. Stopping a session frees its slot; `agent-box-session rm NAME` also delists it. `status` reports the count, the ceiling and the last @@ -169,11 +171,14 @@ branch — and is reported in the JSON (`plugin.skew`) without a warning. Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` -is the batch the ceiling most recently turned away (with a running total, and -`deferred: true` when the receiver kept it to re-offer as a slot frees rather -than dropping it), and `warning` — the same field `ls` sets when the receiver -has no spawn command — is present exactly when a match right now would spawn -nothing. +is the batch the ceiling most recently turned away (with a running total), and +`warning` — the same field `ls` sets when the receiver has no spawn command — +is present exactly when a match right now would spawn nothing. +`lastRefusal.deferred` records the ANSWER the wrapper gave that batch — 75, +declined for retry, rather than a failure that drops it. It is history, not a +queue: nothing rewrites it once the batch starts, or once the receiver gives +up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S. Whether a batch is waiting RIGHT +NOW is the receiver's own state and is not reported here. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -372,8 +377,7 @@ hook_capacity_warning() { standing watch is stalled: a matching event batch starts nothing now. The \ spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ as slots free and dropping it only once the wait outlasts \ -LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ -lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix. Free \ a slot (agent-box-session ls, then agent-box-session stop NAME, or \ agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." diff --git a/modules/src/webhook-spawn.sh b/modules/src/webhook-spawn.sh index ff98010a..e1d7e777 100644 --- a/modules/src/webhook-spawn.sh +++ b/modules/src/webhook-spawn.sh @@ -446,7 +446,10 @@ fi # the last one 20 minutes ago" is the standing fact an agent needs, not # something to forget on the next successful spawn. `deferred` records which # answer this wrapper gave, so `status` does not report as lost a batch the -# receiver is still holding. +# receiver may still be holding. It is that answer and nothing more: this +# program is gone by the time the batch starts or is finally dropped, so +# nothing here could keep a live queue state honest, and the field must not be +# read as one. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -483,7 +486,7 @@ record_refusal() { --arg topic "${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` records the ANSWER this wrapper gave that batch: 75, which local-webhook >= 0.16.0 reads as declined-for-retry rather than a failure that drops it (agent-box#301). It is history and is never rewritten, so it stays true after the batch starts or after the receiver gives up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S; its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then diff --git a/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md b/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md index b384f7f6..5925ace2 100644 --- a/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md +++ b/tests/golden/web/etc/agent-box-guides/AGENTS.agent.md @@ -322,8 +322,9 @@ receiver keeps it, re-offers it as slots free, and drops it only after an hour of waiting. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook status`: its `dispatch` object has the live count against the ceiling and the last batch the -ceiling turned away (`lastRefusal.deferred` says whether that batch is still -waiting). +ceiling turned away. `lastRefusal.deferred` records the ANSWER that batch got - +declined for retry rather than dropped - and stays true afterwards, so it is +history and never a list of what is waiting now. Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn b/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn index 9d4d1f34..3a90544c 100644 --- a/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn +++ b/tests/golden/web/payloads/agent-box-webhook-spawn/bin/agent-box-webhook-spawn @@ -932,7 +932,10 @@ fi # the last one 20 minutes ago" is the standing fact an agent needs, not # something to forget on the next successful spawn. `deferred` records which # answer this wrapper gave, so `status` does not report as lost a batch the -# receiver is still holding. +# receiver may still be holding. It is that answer and nothing more: this +# program is gone by the time the batch starts or is finally dropped, so +# nothing here could keep a live queue state honest, and the field must not be +# read as one. BOX_STATE="$HOME/.local/state/agent-box" REFUSED="$BOX_STATE/webhook-spawn-refused.json" @@ -969,7 +972,7 @@ record_refusal() { --arg topic "${LOCAL_WEBHOOK_SPAWN_TOPIC:-}" \ --arg key "${LOCAL_WEBHOOK_SPAWN_KEY:-}" \ --argjson live "$1" --argjson max "$MAX" --argjson count "$((prev + 1))" \ - '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` says the wrapper exited 75, so local-webhook >= 0.16.0 KEEPS the batch and re-offers it until a slot frees or LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S runs out (agent-box#301); its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + '{"//": "Written by agent-box-webhook-spawn when the hook-* session ceiling refused a standing-watch batch (agent-box#170). `deferred` records the ANSWER this wrapper gave that batch: 75, which local-webhook >= 0.16.0 reads as declined-for-retry rather than a failure that drops it (agent-box#301). It is history and is never rewritten, so it stays true after the batch starts or after the receiver gives up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S; its absence means the batch was dropped outright. `live` is the capacity in use: hook-* sessions running or queued to start (agent-box#280). Cumulative since firstAt; agent-box-webhook status reads it.", at: $at, firstAt: $first, count: $count, live: $live, max: $max, deferred: true} + (if $topic == "" then {} else {topic: $topic} end) + (if $key == "" then {} else {key: $key} end)' \ > "$REFUSED.$$" 2>/dev/null; then diff --git a/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook b/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook index 203ef46b..b720a5f7 100644 --- a/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook +++ b/tests/golden/web/payloads/agent-box-webhook/bin/agent-box-webhook @@ -92,6 +92,8 @@ Two delivery shapes: wrapper declines it (exit 75) and the receiver re-offers it as slots free, giving up only after LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S (an hour here). + `status` reports which answer the LAST refused batch + got, never whether it is still waiting. Stopping a session frees its slot; `agent-box-session rm NAME` also delists it. `status` reports the count, the ceiling and the last @@ -179,11 +181,14 @@ branch — and is reported in the JSON (`plugin.skew`) without a warning. Its `dispatch` object is where to look when standing watches seem dead: `hookSessions` is the running hook-* count against the ceiling, `lastRefusal` -is the batch the ceiling most recently turned away (with a running total, and -`deferred: true` when the receiver kept it to re-offer as a slot frees rather -than dropping it), and `warning` — the same field `ls` sets when the receiver -has no spawn command — is present exactly when a match right now would spawn -nothing. +is the batch the ceiling most recently turned away (with a running total), and +`warning` — the same field `ls` sets when the receiver has no spawn command — +is present exactly when a match right now would spawn nothing. +`lastRefusal.deferred` records the ANSWER the wrapper gave that batch — 75, +declined for retry, rather than a failure that drops it. It is history, not a +queue: nothing rewrites it once the batch starts, or once the receiver gives +up on it at LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S. Whether a batch is waiting RIGHT +NOW is the receiver's own state and is not reported here. One-time per SENDER, to make its deliveries possible at all: agent-box-webhook setup github # prints that source's URL and its HMAC @@ -382,8 +387,7 @@ hook_capacity_warning() { standing watch is stalled: a matching event batch starts nothing now. The \ spawn wrapper DECLINES it (exit 75) and the receiver holds it, re-offering it \ as slots free and dropping it only once the wait outlasts \ -LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix, and \ -lastRefusal.deferred says whether the last batch is waiting or was lost. Free \ +LOCAL_WEBHOOK_SPAWN_DEFER_MAX_S — so clearing the cap is still the fix. Free \ a slot (agent-box-session ls, then agent-box-session stop NAME, or \ agent-box-session rm NAME to delist it for good), or raise \ AGENT_BOX_HOOK_SESSION_MAX on the receiver daemon unit." diff --git a/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md b/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md index da697031..c04e207f 100644 --- a/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md +++ b/tests/native/expected/etc/agent-box-guides/AGENTS.agent.md @@ -359,8 +359,9 @@ receiver keeps it, re-offers it as slots free, and drops it only after an hour of waiting. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook status`: its `dispatch` object has the live count against the ceiling and the last batch the -ceiling turned away (`lastRefusal.deferred` says whether that batch is still -waiting). +ceiling turned away. `lastRefusal.deferred` records the ANSWER that batch got - +declined for retry rather than dropped - and stays true afterwards, so it is +history and never a list of what is waiting now. Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on diff --git a/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md b/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md index da697031..c04e207f 100644 --- a/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md +++ b/tests/native/expected/etc/agent-box-guides/AGENTS.robot.md @@ -359,8 +359,9 @@ receiver keeps it, re-offers it as slots free, and drops it only after an hour of waiting. A stopped session frees its slot even before it is delisted. So before you conclude a repo has been quiet, run `agent-box-webhook status`: its `dispatch` object has the live count against the ceiling and the last batch the -ceiling turned away (`lastRefusal.deferred` says whether that batch is still -waiting). +ceiling turned away. `lastRefusal.deferred` records the ANSWER that batch got - +declined for retry rather than dropped - and stays true afterwards, so it is +history and never a list of what is waiting now. Payload rules (`--when` / `--drop`, JSON predicates over payload paths) ARE a watch's spawn policy - see `agent-box-webhook --help`. This box's watches on