From 604a6cafb3f9aee82342de55f782946fe59bb740 Mon Sep 17 00:00:00 2001 From: santoshkumarradha Date: Wed, 16 Sep 2026 16:29:42 -0400 Subject: [PATCH 1/8] chore: clean internal experiment records and revive test/ux for codeaf Public-repository cleanup ahead of opening the docs to outside readers. - Remove four internal DOE/experiment record directories that reference private issue and pull-request numbers and are not current-product documentation: docs/design/pin-doe, plan-gate-doe, turn-wall-share-doe, and workspace-foundation. - Neutralize internal machine references in benchmark and design docs: the bench host ("spark"), its secrets.env path, and a personal working branch name, all now generic placeholders. - Revive test/ux/, the user-facing journey suite, for the renamed product: collapse the removed v1/v2 dual-surface logic into the single v3 surface, repoint the binary to cmd/codeaf and state to ~/.codeaf/CODEAF_*, and delete gate.sh whose premise was the v1-vs-v2 parity table. Co-Authored-By: codeaf --- bench/prompt-diet/README.md | 5 +- bench/prompt-diet/run.sh | 10 +- .../CALIBRATION-20260905.md | 2 +- .../conversation-runtime/COMMUNICATION.md | 2 +- .../conversation-runtime/IMPLEMENTATION.md | 2 +- docs/design/conversation-runtime/MODULES.md | 2 +- .../design/conversation-runtime/VALIDATION.md | 4 +- docs/design/pin-doe/README.md | 39 -- docs/design/pin-doe/baseline-35c1a79e.md | 8 - docs/design/pin-doe/order.txt | 32 -- docs/design/pin-doe/prereg.md | 57 --- docs/design/pin-doe/report-table.md | 50 -- docs/design/pin-doe/report.md | 36 -- docs/design/plan-gate-doe/REPORT.md | 122 ----- docs/design/prompt-diet/BENCH.md | 28 +- docs/design/prompt-diet/DESIGN.md | 4 +- docs/design/turn-wall-share-doe/README.md | 120 ----- .../workspace-foundation/CONSTRAINTS.md | 279 ----------- .../workspace-foundation/IMPLEMENTATION.md | 102 ---- test/ux/journeys/01-ask-small.sh | 24 + test/ux/journeys/02-commission.sh | 56 +++ test/ux/journeys/03-status-read.sh | 35 ++ test/ux/journeys/04-steer.sh | 67 +++ test/ux/journeys/05-correct.sh | 47 ++ test/ux/journeys/06-answer-question.sh | 48 ++ test/ux/journeys/07-teach.sh | 36 ++ test/ux/journeys/08-charter.sh | 64 +++ test/ux/journeys/09-consent.sh | 58 +++ test/ux/journeys/10-money.sh | 44 ++ test/ux/journeys/11-what-did-you-learn.sh | 41 ++ test/ux/journeys/12-leave-return.sh | 60 +++ test/ux/journeys/13-second-window.sh | 52 +++ test/ux/journeys/14-craft.sh | 57 +++ test/ux/journeys/15-fanout.sh | 79 ++++ test/ux/journeys/16-model-words.sh | 57 +++ test/ux/journeys/17-attachments.sh | 63 +++ test/ux/journeys/18-take-out.sh | 52 +++ test/ux/journeys/19-self-pitch.sh | 39 ++ test/ux/judge.py | 104 +++++ test/ux/lib.sh | 336 ++++++++++++++ test/ux/quality/20-coding.sh | 115 +++++ test/ux/quality/21-research.sh | 69 +++ test/ux/quality/22-quicker.sh | 59 +++ test/ux/quality/23-learning-applied.sh | 60 +++ test/ux/quality/24-shape.sh | 106 +++++ test/ux/quality/25-writing.sh | 56 +++ test/ux/quality/26-data.sh | 61 +++ test/ux/quality/27-errand.sh | 54 +++ test/ux/run.sh | 435 ++++++++++++++++++ test/ux/tier-matrix.sh | 134 ++++++ 50 files changed, 2597 insertions(+), 875 deletions(-) delete mode 100644 docs/design/pin-doe/README.md delete mode 100644 docs/design/pin-doe/baseline-35c1a79e.md delete mode 100644 docs/design/pin-doe/order.txt delete mode 100644 docs/design/pin-doe/prereg.md delete mode 100644 docs/design/pin-doe/report-table.md delete mode 100644 docs/design/pin-doe/report.md delete mode 100644 docs/design/plan-gate-doe/REPORT.md delete mode 100644 docs/design/turn-wall-share-doe/README.md delete mode 100644 docs/design/workspace-foundation/CONSTRAINTS.md delete mode 100644 docs/design/workspace-foundation/IMPLEMENTATION.md create mode 100755 test/ux/journeys/01-ask-small.sh create mode 100755 test/ux/journeys/02-commission.sh create mode 100755 test/ux/journeys/03-status-read.sh create mode 100755 test/ux/journeys/04-steer.sh create mode 100755 test/ux/journeys/05-correct.sh create mode 100755 test/ux/journeys/06-answer-question.sh create mode 100755 test/ux/journeys/07-teach.sh create mode 100755 test/ux/journeys/08-charter.sh create mode 100755 test/ux/journeys/09-consent.sh create mode 100755 test/ux/journeys/10-money.sh create mode 100755 test/ux/journeys/11-what-did-you-learn.sh create mode 100755 test/ux/journeys/12-leave-return.sh create mode 100755 test/ux/journeys/13-second-window.sh create mode 100755 test/ux/journeys/14-craft.sh create mode 100755 test/ux/journeys/15-fanout.sh create mode 100755 test/ux/journeys/16-model-words.sh create mode 100755 test/ux/journeys/17-attachments.sh create mode 100755 test/ux/journeys/18-take-out.sh create mode 100755 test/ux/journeys/19-self-pitch.sh create mode 100644 test/ux/judge.py create mode 100644 test/ux/lib.sh create mode 100755 test/ux/quality/20-coding.sh create mode 100755 test/ux/quality/21-research.sh create mode 100755 test/ux/quality/22-quicker.sh create mode 100755 test/ux/quality/23-learning-applied.sh create mode 100755 test/ux/quality/24-shape.sh create mode 100755 test/ux/quality/25-writing.sh create mode 100755 test/ux/quality/26-data.sh create mode 100755 test/ux/quality/27-errand.sh create mode 100755 test/ux/run.sh create mode 100755 test/ux/tier-matrix.sh diff --git a/bench/prompt-diet/README.md b/bench/prompt-diet/README.md index 0ca5cb9028..867ccac4e2 100644 --- a/bench/prompt-diet/README.md +++ b/bench/prompt-diet/README.md @@ -48,9 +48,8 @@ comparing two harnesses rather than two prompts. ## Running it -Everything runs on the Spark. `~/.local/bin/go`, the key in -`~/.config/fleet/secrets.env`, and `tmux` are all there; the laptop has the -first two and the owner's standing order says the suites do not run on it. +Everything runs on a bench host. `~/.local/bin/go`, the key in a sourced +`secrets.env`, and `tmux` are all there. ```sh bench/prompt-diet/run.sh 6aa6a946e dev --layers a # free, seconds diff --git a/bench/prompt-diet/run.sh b/bench/prompt-diet/run.sh index 4102fd8ae0..5515fdd935 100755 --- a/bench/prompt-diet/run.sh +++ b/bench/prompt-diet/run.sh @@ -19,11 +19,11 @@ # side — and a rig that moved with the subject would be measuring two harnesses # rather than two prompts. # -# IT RUNS ON THE SPARK. Layer B alone is seventeen minutes of a real binary in a +# IT RUNS ON THE BENCH HOST. Layer B alone is seventeen minutes of a real binary in a # real terminal against a real model, and the owner's standing order is that no # full suite runs on the laptop. Nothing here refuses to run elsewhere — a # refusal keyed off a hostname would be a lie on the next machine — but every -# recipe in docs/design/prompt-diet/BENCH.md is spelled for `ssh spark`. +# recipe in docs/design/prompt-diet/BENCH.md is spelled for `ssh benchhost`. # # IT SPENDS REAL MONEY. Layers B and C call a real model. Layer A does not, and # is worth running on its own while editing. @@ -72,7 +72,7 @@ LAYERS="a,c,d" # THE PIN IS THE ONE THE BATTERIES WERE CALIBRATED ON, and it is not the one # this wave started with. # -# MEASURED on the Spark, 2026-09-10. Pinned to `deepseek/deepseek-v4.1-flash`, +# MEASURED on a bench host, 2026-09-10. Pinned to `deepseek/deepseek-v4.1-flash`, # cells died mid-turn with the provider's own sentence: "0 endpoints out of 1 # requested are available matching your guardrail restrictions and data policy # … Paid model training violation (account settings): 1 endpoint excluded". It @@ -169,14 +169,14 @@ if git -C "$OUT" rev-parse --show-toplevel >/dev/null 2>&1; then exit 1 fi -# GO IS NOT ON THE SPARK'S NON-INTERACTIVE PATH. Finding it here rather than +# GO IS NOT ON THE BENCH HOST'S NON-INTERACTIVE PATH. Finding it here rather than # asking every caller to export it is what keeps the ssh recipes in BENCH.md to # one line, and a missing toolchain says so once instead of failing four layers # deep with "go: command not found". GO="${GO:-$(command -v go || true)}" [ -n "$GO" ] && [ -x "$GO" ] || GO="$HOME/.local/bin/go" if [ ! -x "$GO" ]; then - warn "no go toolchain: set GO= (the Spark's lives at ~/.local/bin/go)" + warn "no go toolchain: set GO= (the bench host's lives at ~/.local/bin/go)" exit 1 fi diff --git a/docs/design/conversation-runtime/CALIBRATION-20260905.md b/docs/design/conversation-runtime/CALIBRATION-20260905.md index ab8442bf9b..c6ac736d98 100644 --- a/docs/design/conversation-runtime/CALIBRATION-20260905.md +++ b/docs/design/conversation-runtime/CALIBRATION-20260905.md @@ -1,6 +1,6 @@ # Calibration 02 — measured candidate and next decisions -Candidate `ec9f5a0c2`, local branch `santosh/conversation-runtime`. The full repository check passed before the binary and rig were frozen. This is calibration, not a demonstrated Pareto frontier. +Candidate `ec9f5a0c2`, local branch `a local working branch`. The full repository check passed before the binary and rig were frozen. This is calibration, not a demonstrated Pareto frontier. ## Protocol and completeness diff --git a/docs/design/conversation-runtime/COMMUNICATION.md b/docs/design/conversation-runtime/COMMUNICATION.md index 805b223634..49af465509 100644 --- a/docs/design/conversation-runtime/COMMUNICATION.md +++ b/docs/design/conversation-runtime/COMMUNICATION.md @@ -1,6 +1,6 @@ # Communication ownership and extension boundary -This is the implementation target, not a claim about current behavior. The local wave remains on `santosh/conversation-runtime`; no push, pull request or trunk merge is authorized for this wave. Coding delegates initially used Claude Code Opus; the owner subsequently authorized OMP with GLM 5.3 as its replacement. Live model tests use DeepSeek V4 Flash through OpenRouter, pinned across roles. Commercial OpenRouter models are excluded. +This is the implementation target, not a claim about current behavior. The local wave remains on a working branch. Live model tests use DeepSeek V4 Flash through OpenRouter, pinned across roles. Commercial OpenRouter models are excluded. ## One delivery mechanism, distinct meanings diff --git a/docs/design/conversation-runtime/IMPLEMENTATION.md b/docs/design/conversation-runtime/IMPLEMENTATION.md index 586a0aa0f0..3ace6ef8ac 100644 --- a/docs/design/conversation-runtime/IMPLEMENTATION.md +++ b/docs/design/conversation-runtime/IMPLEMENTATION.md @@ -1,6 +1,6 @@ # Conversation runtime implementation -Local integration line: `santosh/conversation-runtime` (2026-09-04 working wave). This is a working record of what has merged into that +Local integration line: `a local working branch` (2026-09-04 working wave). This is a working record of what has merged into that line, what is still open, and what the available evidence does and does not establish. It describes the local integration only; nothing here is published or claimed as released. diff --git a/docs/design/conversation-runtime/MODULES.md b/docs/design/conversation-runtime/MODULES.md index ccb9601554..82c02cecda 100644 --- a/docs/design/conversation-runtime/MODULES.md +++ b/docs/design/conversation-runtime/MODULES.md @@ -1,6 +1,6 @@ # Modules: where the responsibilities are, and where the seams are -**Read as of the local `santosh/conversation-runtime` wave, 2026-09-04.** Every row marked *implemented* names +**Read as of the local `a local working branch` wave, 2026-09-04.** Every row marked *implemented* names code in this tree. Every row marked *proposed* is a plan and nothing more. `PLAN.md`, `PRODUCT.md` and `COMMUNICATION.md` describe intended behavior and are not claims about what ships. diff --git a/docs/design/conversation-runtime/VALIDATION.md b/docs/design/conversation-runtime/VALIDATION.md index d1a544ce46..2699bfd172 100644 --- a/docs/design/conversation-runtime/VALIDATION.md +++ b/docs/design/conversation-runtime/VALIDATION.md @@ -21,7 +21,7 @@ tests passed and no skip was added. See the calibration report for exact scope. This is a local development wave, not a release or a claim that the harness is on a universal cost/time/quality frontier. The integration branch is -`santosh/conversation-runtime`; the shared `dev` checkout and the prior consolidated +`a local working branch`; the shared `dev` checkout and the prior consolidated QA branch were not used as mutable workspaces. Independent Claude Opus lanes were reviewed and merged locally. After the Opus session limit, the owner authorized OMP with GLM 5.3 for the remaining delegated review. All live comparison calls use @@ -178,7 +178,7 @@ ledger has not grown. Focused budget tests passed under the race detector (2.207 as did causal-wake and prefix regressions (10.353s). The unchanged fixed-prefix budget remains 48,000 bytes. -Work is integrated locally on `santosh/conversation-runtime` in +Work is integrated locally on `a local working branch` in `/private/tmp/af-conversation`, with its own `bin/codeaf`. The original QA checkout and shared `dev` were not modified by this wave. Owned merged delegate worktrees, CLI profile and GLM forwarding process were removed; raw evidence and its short diff --git a/docs/design/pin-doe/README.md b/docs/design/pin-doe/README.md deleted file mode 100644 index 88793a371f..0000000000 --- a/docs/design/pin-doe/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Pin semantics after a refusal: the pre-registered experiment (2026-09-02) - -This folder is the record of the design-of-experiments run that decided what a -`lane.talk` pin does when the router refuses the pinned lane for the model -(issues #375 and #456; landed as arm B in PR #533). The decision and the fix -shape are on issue #456; this is the evidence behind them. - -## What is here - -- `prereg.md` — the approved pre-registration. -- `report.md` and `report-table.md` — the summarized result and per-cell table. -- `order.txt` — the seeded run order. -- `baseline-35c1a79e.md` — the comparison baseline. - -The raw cell folders, sealed map, and golden captures were removed after these -retained reports recorded their findings. - -## What was left out, and where the raw run lived - -Each cell also produced `frames.log` (every sampled frame), `f2p.log` and `suite.log` -(pytest output), `entry.json` and `prompt.txt` (the brief, reproducible from the canary -pool by id), and the full `home/` and `work/` directories (the isolated `CODEAF_HOME` -with its transcripts, and the checkout the model edited). Together they were 2.8 GB -and are not in the repository. The raw run lived in a temporary scratchpad on the -shared runner and does not survive a reboot of that machine. The -arm binaries were built from `exp/pin-yields-line` (arm B, 082a0050) and -`exp/pin-yields-silent` (arm C, e967ad57) on top of PR #368's tip 6b35eb52 (arm A, -the shipped behaviour byte-for-byte). - -## Reading the numbers - -Arm means (n = 8): A pin absolute 0/8 tests pass, $0.0001, wall 902 s, 2.1 re-demands; -B yields with a line 8/8, $0.0356, 668 s, 3.2; C yields silently 8/8, $0.0271, 598 s, -3.0; N unpinned 8/8, $0.0288, 688 s, 0. Noise band: cost $0.0105, wall 109 s. B and C -tie inside the band on every metric and both tie with N; A is on the front only because -a dead turn is free. The user chose B (say so) over C (silent) by the no-silent- -substitution law. Two findings from the cells shaped the fix: the sentence was never -observed in any frame, and every task child re-paid the 404; both causes are recorded on -#456 and in PR #533's change entry. diff --git a/docs/design/pin-doe/baseline-35c1a79e.md b/docs/design/pin-doe/baseline-35c1a79e.md deleted file mode 100644 index d247a009c9..0000000000 --- a/docs/design/pin-doe/baseline-35c1a79e.md +++ /dev/null @@ -1,8 +0,0 @@ -# Canary baseline chat-door rows on dev 35c1a79e (from codeaf-14, #407 "BASELINE COMPLETE"), no pin -| brief | door / tests | wall s | cost $ | ttft s | note | -|---|---|---|---|---|---| -| reef-145 | ok / green | 483 | 0.0163 | 1.9 | | -| click-3740 | ok / green | 248 | 0.0157 | 2.5 | | -| attrs-1416 | wall / green | 902 | 0.0269 | | 2 turns, 37 calls; task landed at 12m35s, conversation kept working to the wall | -| tox-4031 | wall / green | 904 | 0.0376 | | 3 turns, 42 calls; pre-TMPDIR rig, turns spent on /tmp/pytest-of-santosh garbage | -Walls at 900 s are product behaviour (the chat does not stop after its task lands), not the driver's rule. diff --git a/docs/design/pin-doe/order.txt b/docs/design/pin-doe/order.txt deleted file mode 100644 index 94f87fe9c7..0000000000 --- a/docs/design/pin-doe/order.txt +++ /dev/null @@ -1,32 +0,0 @@ -fcd2e6e48d -4835a3b6fc -68c038cb44 -151c89cd30 -e13be0103b -1b57877a3a -b4c5c1b3f7 -43c455ed9b -676fd2f3ac -e9f9996059 -d7733921ef -bed059cbad -a40753e97e -2b173a4896 -b71a68febf -77bc6318e1 -afb7e903bc -4d2a84a40b -0f0ebcea38 -3d5f19609c -43a638a550 -8d64be03a4 -9b00fe72e0 -b9b14f5221 -bb3d742840 -c5854addfc -13fa8fd8e7 -15775ac481 -7a638f9163 -28f3a56faa -67c028ec58 -4bf9eecf1f diff --git a/docs/design/pin-doe/prereg.md b/docs/design/pin-doe/prereg.md deleted file mode 100644 index b0b5303751..0000000000 --- a/docs/design/pin-doe/prereg.md +++ /dev/null @@ -1,57 +0,0 @@ -# Pre-registration: what a pin means after the wire refuses it - -Owner codeaf-e6. Written 2026-09-02 before any arm was built or any cell run. - -## Question -After the router refuses a person's pinned lane for a model (`permits only:` 404), -which pin semantics is Pareto-best on cost, quality and wall time, on real issues, -on deepseek-v4-flash, through the chat door? - -## Arms (factor 1, four levels) -- A pin absolute — santosh-75's #368 branch at its rebased tip, byte for byte. Re-demands the pinned lane on every request; the ladder drops the demand after each 404. -- B pin yields after refusal, one visible line: `X cannot serve this model; routing on auto for this model until you pin again`. Built on A; retirement in lanepin.go consuming laneRefusal at laneChoiceFor; the line travels as RescueNews. -- C pin yields silently — control only. Same as B without the line. Cannot win (no-silent-substitution law); its numbers bound the cost of the line. -- N null baseline — A's binary with no pin (`lane.talk` unset, auto). Bounds what pinning costs at all. - -## Briefs (factor 2) and replicates -k = 4 to 6 validated anchors from codeaf-14's pool (`~/af-canary/bench/canary/pool.json`), cited by pool id, chosen BEFORE any arm runs as the first k validated in pool order; n = 2 per (arm, brief). Cells = 4·k·2 (32 to 48). -k is fixed after the #407 baseline table: the largest k in [4,6] with 4·k·2·(mean baseline cost per cell) ≤ $8, keeping $2 reserve under the $10 budget. - -## Held constant -Model deepseek/deepseek-v4-flash on model.talk and all four tiers, `-one-model`; chat door via canary_chat (lib/chat.sh) with canary_home rows; cap $1 per cell, wall 900 s; the pin is `lane.talk: anthropic` on A, B, C — a lane that never serves this model, so the refusal is deterministic (confirmed on the live wire 2026-09-02: HTTP 404, "…permits only: anthropic", metadata.failed_routing_step "Filter by Allowed Providers"). Same price table: OpenRouter's flash endpoint prices snapshotted at run start and applied to every cell from token counts; the ledger's own cost is reported beside it as a check. Same base sha under all four binaries. - -## Order and blinding -Run order is a seeded random permutation of all cells, blocked by brief so each brief's 8 cells are spread across the run, and no arm runs twice in a row where avoidable (router load drifts by the hour). Each cell's output dir is named by an opaque id; the id→(arm, brief, replicate) map is written sealed before the run and opened only after every judge.json exists. Scoring is lib/judge.py, mechanical: fix-PR tests overlaid after the door stops, f2p must be green, no new failures against base. - -## Metrics -Primary: cost_usd (price table × tokens; ledger cost as check), wall_s (door start to the canary end rule), quality (judge pass, 0/1). Secondary: redemand_count (calls whose end row is the router's routing refusal, i.e. the pinned lane demanded again after a refusal), ttft_ms (median over the cell's calls, and the first call alone), and visible_lines: the count of person-visible sentences the arm prints about the pin, the refusal, or auto — status-line rows and conversation lines — counted mechanically from the cell's frames.log and screen.txt by a fixed needle list written before the run (`refused`, `permits only`, `cannot serve`, `routing on auto`, `switch to auto`, `pinned`). What the person had to read is part of quality under the user's law; B's one line, A's repeated refused rows and C's silence are the difference a person feels. - -## Pre-registered expectations -H1 redemand_count(A) ≥ number of turns; (B) = (C) = 1; (N) = 0. H2 wall(A) − wall(B) ≥ one 404 round trip per turn; wall(B) ≈ wall(C). H3 quality equal across A, B, C (same model answers every turn; the pin only adds refused round trips) — any gap is noise at n=2 and will be reported as such. H4 cost equal across A, B, C (a 404 is not billed). If these hold the front is {B, C} and B is the pick because C cannot win. If H2 fails (A's penalty inside the noise band) the front is {A, B, C} and the decision reverts to argument; that outcome is reported, not hidden. - -## Decision rule and tiebreak -Minimise (cost, 1−quality, wall). Front = non-dominated arm means. Tiebreak, in order: quality (passes out of 2k), then cost, then wall. Noise band: two arms tie on a metric when their means differ by less than the mean absolute difference between replicates over all cells for that metric. - -## Exclusions (reported, never silent) -A cell with an unclosed ledger row (codeaf-14's sighting; likely #334, fix #357) is excluded from wall and ttft and listed. A cell that hits the cap counts as quality 0 and is listed. A cell where the refusal did not occur on A/B/C is invalid and re-run once. - -## Pilot before the main run (one cell, arm A, anchor 1) -Verifies that the 404 refusal is observable in `/logs/calls.jsonl` and the turn recovers, and that all five metrics extract from the cell's files. If the ledger does not record the refusal, redemand_count is taken from codeaf's debug log; if neither, the arms run against santosh-75's lanestub with SheetOnly for redemand and wall only, and quality is not claimed from that run (the stub does not answer with a real model). - -## Builds -A: #368 tip, `make build` in its own worktree. B, C: Opus lanes (Fable if Opus is still limited) on branches off A; diff confined to lanepin.go, laneChoiceFor, one RescueNews line; santosh-75's names kept. Each arm's bin/codeaf is passed as CANARY_BIN. The winning arm lands later as a PR on top of #368's merge, pin policy only. - -## Deliverable -The table of arm means with replicate ranges, the Pareto front, the secondary table, the exclusion list, and the sealed map. The user or operator picks. - -## Amendment before the first cell (2026-09-02 15:52 EDT) -The pool holds nine validated anchors in this order: reef-145, attrs-1416, packaging-1318, click-3740, tox-4031, build-860, virtualenv-3072, astroid-2305, sphinx-11437. The rule "first k in pool order" would take packaging-1318, whose fail-to-pass set is 53,420 tests and whose grade is slow enough that eight cells of it would dominate the run's wall for a reason that has nothing to do with the arms. It is skipped, and the briefs are the next four in pool order: reef-145, attrs-1416, click-3740, tox-4031. Declared here before any cell ran; the sealed map was written with these four. Cells run three at a time; the box's one-minute load is recorded at each cell's start and end and reported beside wall. - -## Scoring note added during the run (16:20 EDT), before any cell was scored -codeaf-14 reports that pytest's cleanup of the shared can die under other sessions' pytest runs and leave a grade reading "0 passed 0 failed" with an rm_rf traceback in f2p.log. The driver copy this run uses predates the per-cell TMPDIR fix and is not edited while cells are in flight. Rule: after the last cell, every cell whose judge shows zero tests collected and an rm_rf traceback in f2p.log is re-graded with CANARY_REGRADE=1 through the canary lib on dev (no door re-run, no spend), and the report lists which cells were re-graded. A grade is never changed by hand. - -## Amendments at scoring (18:25 EDT), declared before the front was read -1. The exclusion for unclosed ledger rows is withdrawn: they proved endemic (a start row per cancelled hedge arm, the #334 accounting class), and wall is the door's own clock while first-token latency sits on end rows, so neither depends on them. The count is reported per cell instead. -2. One cell (tox-4031, arm N, replicate 1) has no judge record because the judge crashed when the whole-suite step hit its cap; its f2p.log shows the issue's own tests passing, so the pre-registered quality (the issue's tests pass) is read from that log and the cell is marked. -3. Quality is exactly the pre-registered one, the issue's tests pass with no regression; the driver's "pass" also folds in the wall and is not used. -4. Visible lines count distinct sentences, not sampled frames. diff --git a/docs/design/pin-doe/report-table.md b/docs/design/pin-doe/report-table.md deleted file mode 100644 index a8afccab2f..0000000000 --- a/docs/design/pin-doe/report-table.md +++ /dev/null @@ -1,50 +0,0 @@ -# pin experiment — arm means (n per arm = 8) - -| arm | passes | cost $ | wall s | redemand | ttft med ms | visible lines | excluded | -|---|---|---|---|---|---|---|---| -| A | 0/8 | 0.0001 | 902 | 2.1 | 1435 | 1.0 | 0 | -| B | 8/8 | 0.0356 | 668 | 3.2 | 1488 | 0.8 | 0 | -| C | 8/8 | 0.0271 | 598 | 3.0 | 1213 | 0.6 | 0 | -| N | 8/8 | 0.0288 | 688 | 0.0 | 1378 | 0.0 | 0 | - -noise band (mean |replicate difference|): cost 0.0105, wall 109, quality 0.00 - -Pareto front (minimise cost, wall, 1-quality; ties inside the band): A, B, C, N -Tiebreak order (quality, cost, wall): C, N, B, A - -## Cells - -| id | arm | brief | rep | pass | reason | cost | wall | redemand | ttft first | visible | unclosed | -|---|---|---|---|---|---|---|---|---|---|---|---| -| 1b57877a3a | A | Human-Agent-Society-reef-145 | 1 | 0 | wall / red | 0.0001 | 902 | 2 | 1808 | 1 | 0 | -| 68c038cb44 | A | Human-Agent-Society-reef-145 | 2 | 0 | wall / red | 0.0001 | 903 | 2 | 1526 | 1 | 0 | -| 43c455ed9b | B | Human-Agent-Society-reef-145 | 1 | 1 | wall / green | 0.0425 | 901 | 3 | 573 | 1 | 50 | -| fcd2e6e48d | B | Human-Agent-Society-reef-145 | 2 | 1 | ok / green | 0.0220 | 577 | 4 | 828 | 0 | 26 | -| b4c5c1b3f7 | C | Human-Agent-Society-reef-145 | 1 | 1 | wall / green | 0.0525 | 902 | 7 | 612 | 0 | 64 | -| 151c89cd30 | C | Human-Agent-Society-reef-145 | 2 | 1 | ok / green | 0.0252 | 452 | 4 | 534 | 0 | 41 | -| 4835a3b6fc | N | Human-Agent-Society-reef-145 | 1 | 1 | ok / green | 0.0460 | 714 | 0 | 334 | 0 | 70 | -| e13be0103b | N | Human-Agent-Society-reef-145 | 2 | 1 | wall / green | 0.0218 | 903 | 0 | 527 | 0 | 39 | -| 43a638a550 | A | pallets-click-3740 | 1 | 0 | wall / red | 0.0001 | 903 | 2 | 2160 | 1 | 0 | -| 0f0ebcea38 | A | pallets-click-3740 | 2 | 0 | wall / red | 0.0001 | 903 | 2 | 1830 | 1 | 0 | -| afb7e903bc | B | pallets-click-3740 | 1 | 1 | ok / green | 0.0066 | 135 | 1 | 590 | 0 | 0 | -| 9b00fe72e0 | B | pallets-click-3740 | 2 | 1 | ok / green | 0.0068 | 326 | 2 | 900 | 0 | 15 | -| 8d64be03a4 | C | pallets-click-3740 | 1 | 1 | ok / green | 0.0074 | 133 | 1 | 886 | 0 | 0 | -| 4d2a84a40b | C | pallets-click-3740 | 2 | 1 | ok / green | 0.0053 | 99 | 1 | 666 | 0 | 0 | -| b9b14f5221 | N | pallets-click-3740 | 1 | 1 | ok / green | 0.0069 | 214 | 0 | 642 | 0 | 0 | -| 3d5f19609c | N | pallets-click-3740 | 2 | 1 | ok / green | 0.0045 | 71 | 0 | 950 | 0 | 0 | -| bed059cbad | A | python-attrs-attrs-1416 | 1 | 0 | wall / red | 0.0001 | 901 | 2 | 579 | 1 | 0 | -| a40753e97e | A | python-attrs-attrs-1416 | 2 | 0 | wall / red | 0.0001 | 901 | 2 | 1890 | 1 | 0 | -| 676fd2f3ac | B | python-attrs-attrs-1416 | 1 | 1 | ok / green | 0.0399 | 701 | 3 | 532 | 1 | 27 | -| 2b173a4896 | B | python-attrs-attrs-1416 | 2 | 1 | wall / green | 0.0690 | 903 | 3 | 566 | 2 | 52 | -| d7733921ef | C | python-attrs-attrs-1416 | 1 | 1 | ok / green | 0.0355 | 799 | 3 | 593 | 1 | 62 | -| b71a68febf | C | python-attrs-attrs-1416 | 2 | 1 | ok / green | 0.0203 | 589 | 2 | 529 | 1 | 36 | -| e9f9996059 | N | python-attrs-attrs-1416 | 1 | 1 | wall / green | 0.0424 | 902 | 0 | 595 | 0 | 55 | -| 77bc6318e1 | N | python-attrs-attrs-1416 | 2 | 1 | wall / green | 0.0376 | 902 | 0 | 440 | 0 | 57 | -| 28f3a56faa | A | tox-dev-tox-4031 | 1 | 0 | wall / red | 0.0001 | 901 | 2 | 689 | 1 | 0 | -| 4bf9eecf1f | A | tox-dev-tox-4031 | 2 | 0 | wall / red | 0.0002 | 901 | 3 | 724 | 1 | 0 | -| 67c028ec58 | B | tox-dev-tox-4031 | 1 | 1 | wall / green | 0.0439 | 904 | 7 | 561 | 1 | 27 | -| c5854addfc | B | tox-dev-tox-4031 | 2 | 1 | wall / green | 0.0540 | 901 | 3 | 312 | 1 | 50 | -| 15775ac481 | C | tox-dev-tox-4031 | 1 | 1 | wall / green | 0.0254 | 904 | 3 | 561 | 1 | 56 | -| 7a638f9163 | C | tox-dev-tox-4031 | 2 | 1 | wall / green | 0.0450 | 904 | 3 | 297 | 2 | 48 | -| 13fa8fd8e7 | N | tox-dev-tox-4031 | 1 | 1 | wall / no grade | 0.0293 | 901 | 0 | 318 | 0 | 38 | -| bb3d742840 | N | tox-dev-tox-4031 | 2 | 1 | wall / green | 0.0421 | 901 | 0 | 324 | 0 | 57 | diff --git a/docs/design/pin-doe/report.md b/docs/design/pin-doe/report.md deleted file mode 100644 index faebf27933..0000000000 --- a/docs/design/pin-doe/report.md +++ /dev/null @@ -1,36 +0,0 @@ -# Pin semantics after a refusal — result of the pre-registered run (2026-09-02, 15:57–18:14 EDT) - -Four arms × four briefs (reef-145, attrs-1416, click-3740, tox-4031) × two replicates = 32 cells, chat door, deepseek-v4-flash on every seat, live OpenRouter, `lane.talk: anthropic` on A/B/C (a lane that never serves the model), N unpinned. Total spend $0.73. Sealed map opened only after every cell had a record. Amendments are in prereg.md (packaging skipped before the first cell; the unclosed-row exclusion withdrawn at scoring; one crashed grade read from its f2p log). - -## Arm means (n = 8 each) - -| arm | quality (issue's tests pass) | cost $ | wall s | re-demands (404 with `permits only`) | ttft median ms | visible routing sentences | -|---|---|---|---|---|---|---| -| A pin absolute (shipped, #368 tip) | 0/8 | 0.0001 | 902 | 2.1 | 1435 | 1.0 (the router's own 404 text) | -| B pin yields, one line | 8/8 | 0.0356 | 668 | 3.2 | 1488 | 0.8 (see below: none of them is B's line) | -| C pin yields, silent | 8/8 | 0.0271 | 598 | 3.0 | 1213 | 0.6 | -| N unpinned null | 8/8 | 0.0288 | 688 | 0.0 | 1378 | 0.0 | - -Noise band (mean absolute replicate difference): cost $0.0105, wall 109 s, quality 0. - -## The front - -Non-dominated on (cost, wall, 1−quality): A, B, C, N. A is on the front only because doing nothing is free: it fixed no issue, and its "wall" is the driver's 900 s limit reached with a dead turn. On the pre-registered tiebreak (quality, then cost, then wall): **C, N, B, A** — and C and B tie inside the band on every metric (cost differs by $0.0085 against a band of $0.0105; wall by 70 s against 109 s). B and C also tie with N: pinning-then-yielding costs nothing measurable against not pinning at all. - -**So: C cannot win by the no-silent-substitution law, B is indistinguishable from C on every number, and A is not a contender — it is a defect (#456).** The line costs nothing. The pick between B and C is the law, not the data, exactly as the pre-registration said it would be if H3 and H4 held. - -## What the pre-registered hypotheses did - -- H1 (re-demands): A ≥ turns — A paid 2.1 per cell (aux call + the turn), then the turn died, so there were no more turns to re-demand on. B and C were expected at exactly 1 and measured ~3: one on the first aux call, then **one more per task child**, because the retirement lives in the process and a task runs in its own session; each child re-pays a 404 before it, too, yields. Plus the router's other 404 class (`All providers have been ignored`) which is not the pin and not counted. -- H2 (wall): expected A to pay a round trip per turn. Measured: A pays the whole session. On the live wire the turn path does not recover from the pinned refusal at all (#456). -- H3 (quality equal across A/B/C): refuted for A by #456; B = C = N = 8/8. -- H4 (cost equal): B ≈ C ≈ N inside the band; A's $0.0001 is the cost of a dead turn. - -## Two findings the numbers do not show - -1. **B's line was never observed.** The visible-sentence metric found, in every B and C cell, only #368's own `refused · trying coreweave…` status rows (from the tasks' `All providers have been ignored` refusals), and in no B cell the sentence `anthropic cannot serve this model; routing on auto for this model until you pin again`. The wire-level retirement fired (one 404, then the turn went out on auto), so either the line is drawn for less than the 3 s frame sampling interval or it is not drawn on the live path. Before B ships, its line needs a golden frame from the live walk, not the unit test's rendering. -2. **Retirement does not reach task children** (above). If B ships, the retired pair has to travel to the child, or every task starts by paying the refusal and the person sees the line once per task rather than once. - -## Cells - -See report-table.md (per-cell rows with id, arm, brief, replicate, verdicts, cost, wall, re-demands, first-token latency, visible sentences, unclosed ledger rows). Baseline rows from the canary on 35c1a79e are in baseline-35c1a79e.md; N's walls agree with them (two briefs run to the wall even unpinned, because the chat keeps working after its task lands). diff --git a/docs/design/plan-gate-doe/REPORT.md b/docs/design/plan-gate-doe/REPORT.md deleted file mode 100644 index 3b88a64f39..0000000000 --- a/docs/design/plan-gate-doe/REPORT.md +++ /dev/null @@ -1,122 +0,0 @@ -# The plan-gate experiment — planner arm × split-gate mode - -*On 2026-09-02 the owner chose planner B with the split gate OFF from this front. -The gate had shipped armed, with a six-item floor under every division, and -[#418](https://github.com/Agent-Field/codeaf/issues/418) said the floor folds -real divisions; #384 proposed three planner repairs. Neither question was going -to be settled by argument, so both were made factors of one designed experiment -and the winner was taken from the measurement. What landed in PR #435 is the -decision: unpinned, the gate has no say; `CODEAF_SPLITGATE=1` puts the shipped -count back and `judgment` asks the plan's own sizing. The `lanes` counting arm -lost and its code is gone.* - -*What follows is the report as it was written, and it is -planner arm × split-gate mode, plan door, deepseek-v4-flash (kimi-k3 on R1d), 2026-09-02.* - -## DECISION - -**Planner B (stages), split gate off.** From the front table below, on the -pre-registered tiebreak of quality > cost > wall, with unrecoverable draws as -the disqualifier — the report's own front table, quoted: - -## The front (arm × gate) -| arm | gate | draws | quality | unrecoverable | cost $ | wall s | front | -|---|---|---|---|---|---|---|---| -| D | lanes | 21 | 0.71 | 12/21 | 0.0277 | 25 | YES | -| C | lanes | 21 | 0.71 | 18/21 | 0.0209 | 22 | YES | -| B | off | 21 | 0.71 | 4/21 | 0.0129 | 27 | YES | -| B | lanes | 21 | 0.61 | 6/21 | 0.0186 | 25 | YES | -| A | lanes | 21 | 0.60 | 4/21 | 0.0142 | 21 | YES | -| D | off | 21 | 0.56 | 12/21 | 0.0143 | 28 | | -| C | off | 21 | 0.56 | 15/21 | 0.0136 | 28 | | -| B | shipped | 14 | 0.50 | 3/14 | 0.0105 | 24 | YES | -| A | off | 14 | 0.48 | 5/14 | 0.0194 | 28 | | -| A | shipped | 14 | 0.43 | 1/14 | 0.0175 | 22 | YES | -| B | judgment | 14 | 0.36 | 1/14 | 0.0138 | 22 | YES | -| A | judgment | 14 | 0.33 | 4/14 | 0.0144 | 28 | | -| D | judgment | 14 | 0.29 | 5/14 | 0.0141 | 26 | | -| D | shipped | 14 | 0.29 | 6/14 | 0.0160 | 25 | | -| C | shipped | 14 | 0.29 | 6/14 | 0.0255 | 21 | | -| C | judgment | 14 | 0.21 | 3/14 | 0.0120 | 19 | YES | - - -Three cells tie at the top on quality, 0.71. **B off** carries the fewest -unrecoverable draws of the three — 4 of 21, against 12 and 18 for the two -`lanes` cells that reach the same quality — and the lowest cost, $0.0129 against -D lanes' $0.0277 for no more quality. `shipped`, the gate as it was, is on the -front only at 0.50 and 0.43, below the same arm's off cell in both cases. - -The gate's own main effect is **not consistent across planners**: `lanes` beats -`off` under planner A (0.60 vs 0.48) and loses to it under planner B (0.61 vs -0.71). An interaction that changes sign is not an improvement to ship, and it is -why the counting arm was deleted rather than kept as a fourth pin. `judgment` is -kept as a pin because it is one-directional — it can only add keeps to the count -— and because nothing in the front argues it does harm where somebody wants a -floor with an escape. - -**What this decision does not say.** No cell satisfied the pre-registered null on -every brief, and n=3 differences under ~0.15 are coins (planner A under `lanes` -fell from 0.76 to 0.60 on one more draw per brief). This picked the best front -cell on the responses that were measured; it did not prove the gate harmful. -Tier 2 — full runs on real issues — was never run. - -Pre-registered before any cell ran: the factors, the briefs, the responses, the tiebreak. Every draw was judged -blind from the graph JSON alone, against a lanes-matched rule written for each brief before its first plan. -Draws: 16 cells × 7 briefs × 2, plus a third draw on the seven cells within reach of the n=2 front (273 judged draws, ≈$5). -Tiebreak stated in advance: quality > cost > wall; unrecoverable = oversized-undivided / no graph / dead fan-out. - -## The front (arm × gate) -| arm | gate | draws | quality | unrecoverable | cost $ | wall s | front | -|---|---|---|---|---|---|---|---| -| D | lanes | 21 | 0.71 | 12/21 | 0.0277 | 25 | YES | -| C | lanes | 21 | 0.71 | 18/21 | 0.0209 | 22 | YES | -| B | off | 21 | 0.71 | 4/21 | 0.0129 | 27 | YES | -| B | lanes | 21 | 0.61 | 6/21 | 0.0186 | 25 | YES | -| A | lanes | 21 | 0.60 | 4/21 | 0.0142 | 21 | YES | -| D | off | 21 | 0.56 | 12/21 | 0.0143 | 28 | | -| C | off | 21 | 0.56 | 15/21 | 0.0136 | 28 | | -| B | shipped | 14 | 0.50 | 3/14 | 0.0105 | 24 | YES | -| A | off | 14 | 0.48 | 5/14 | 0.0194 | 28 | | -| A | shipped | 14 | 0.43 | 1/14 | 0.0175 | 22 | YES | -| B | judgment | 14 | 0.36 | 1/14 | 0.0138 | 22 | YES | -| A | judgment | 14 | 0.33 | 4/14 | 0.0144 | 28 | | -| D | judgment | 14 | 0.29 | 5/14 | 0.0141 | 26 | | -| D | shipped | 14 | 0.29 | 6/14 | 0.0160 | 25 | | -| C | shipped | 14 | 0.29 | 6/14 | 0.0255 | 21 | | -| C | judgment | 14 | 0.21 | 3/14 | 0.0120 | 19 | YES | - -## Per brief, front cells against the two nulls (mean; draws; u = unrecoverable draws) -| brief | A shipped | A off | A lanes | B off | B lanes | D lanes | C lanes | -|---|---|---|---|---|---|---|---| -| R1a | 1.00 (1.00/1.00) u0 | 0.00 (0.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u0 | 1.00 (1.00/1.00/1.00) u0 | 0.33 (0.00/0.00/1.00) u0 | 0.67 (0.00/1.00/1.00) u2 | 0.33 (0.00/1.00/0.00) u1 | -| R1b | 0.50 (0.00/1.00) u0 | 1.00 (1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u0 | 0.67 (1.00/0.00/1.00) u0 | 1.00 (1.00/1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u3 | -| R1c | 0.50 (1.00/0.00) u1 | 0.50 (0.00/1.00) u2 | 0.11 (0.33/0.00/0.00) u1 | 0.33 (0.00/0.00/1.00) u1 | 0.67 (0.00/1.00/1.00) u1 | 0.00 (0.00/0.00/0.00) u0 | 0.67 (1.00/0.00/1.00) u2 | -| R1d | 1.00 (1.00/1.00) u0 | 0.50 (1.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u1 | 0.56 (0.00/1.00/0.67) u1 | 0.33 (0.00/1.00/0.00) u3 | 1.00 (1.00/1.00/1.00) u1 | 0.33 (0.00/1.00/0.00) u3 | -| H1 | 0.00 (0.00/0.00) u0 | 0.38 (0.75/0.00) u0 | 0.42 (0.25/0.00/1.00) u0 | 0.50 (0.00/1.00/0.50) u0 | 0.25 (0.50/0.00/0.25) u1 | 1.00 (1.00/1.00/1.00) u3 | 0.75 (0.25/1.00/1.00) u3 | -| H2 | 0.00 (0.00/0.00) u0 | 0.50 (1.00/0.00) u0 | 1.00 (1.00/1.00/1.00) u1 | 0.89 (0.67/1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u3 | 1.00 (1.00/1.00/1.00) u3 | -| H3 | 0.00 (0.00/0.00) u0 | 0.50 (1.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u2 | 1.00 (1.00/1.00/1.00) u0 | 0.67 (1.00/0.00/1.00) u2 | 0.89 (0.67/1.00/1.00) u3 | - -## Reading -- No cell satisfies the pre-registered null on every brief: the original #252 brief on the default model (R1c) is a coin for - every arm (A shipped 0.50, B off 0.33, D lanes 0.00, C lanes 0.67), and R1d (kimi-k3) stays at 0.56–1.00. -- Among the 0.71 tie, B off has the fewest unrecoverable draws (4/21) and the lowest cost ($0.013); D lanes and C lanes - reach the same quality only by carrying the measurement flag on 12/21 and 18/21 draws (see finding 2), and D costs 2×. -- Gate main effect is not consistent across arms (interaction): lanes beats off in A (0.60 vs 0.48) and loses to off in - B (0.61 vs 0.71), because B under lanes collapses R1a on 2 of 3 draws (one atomic node, "no two pieces could be named"). - Shipped and judgment fold every held-out brief in every arm (judgment's fallback count is the shipped one). -- Planner main effect at a fixed gate: B ≥ A (off 0.71 vs 0.48; shipped 0.50 vs 0.43); C and D are dominated by their flag mass. -- Unrecoverable-draw recovery (#384 null): A off 5/14 (R1a, R1c×2, R1d, H3) → B off 4/21 (R1c, R1d, H3×2): R1a's collapse is - recovered, R1c/R1d are not. -- n=3 is still a coin: A lanes fell from 0.76 (n=2) to 0.60 with one more draw per brief. Differences under ~0.15 are not decisions. - -## Findings for other owners -1. Plan door verdict vs tests (#428 family, for codeaf-3f): the door exits 0 with a graph on 103 of 273 draws that hold an - oversized node carrying a refusal word (A 14, B 13, C 42, D 34); cell ids and node ids are with the raw draws. -2. Measurement defect in #425 (arms C, D): a per-lane node still names the whole file, so "its named material exceeds what one - worker holds" is stamped on correctly divided lanes (R1a 3/3 lanes, R1b, H2, H3), and expansion cannot divide a lane over one - file further. The measure needs the lane's share of the file, or must not veto a node whose summary scopes a region. -3. #424 under lanes: a stage that holds three lanes is sized atomic with no parts and refused as unnamed, so the stage question - is never asked (B lanes R1a 2/3 draws). The atomic-until-proven pin and the measured window disagree on a 144 KB file. -4. Panel copies: C draws a three-identical-pass Panel on R1c 6/8 and R1b 3/8 — the ensemble hook fires where A splits. - -## Tier 2 (full runs on real issues) not run: the pool has one anchor plus isort-2562, and the WIP freeze holds new work. diff --git a/docs/design/prompt-diet/BENCH.md b/docs/design/prompt-diet/BENCH.md index 89e9f1b172..f6817ce659 100644 --- a/docs/design/prompt-diet/BENCH.md +++ b/docs/design/prompt-diet/BENCH.md @@ -62,7 +62,7 @@ would have to be written was describing a session that never saved one. ## 1. The baseline — `dev` at `6aa6a946e` -Measured on the Spark, 2026-09-10. +Measured on a bench host, 2026-09-10. ### The static prefix (layer A) @@ -127,7 +127,7 @@ every lane of the pinned model refusing. `dev` at `6aa6a946e` against `prompt-diet/integrate` at `2a90be7e8` (its tip when the run fetched it; the branch has moved since). Same rig, same pin, same cells, -back to back on the Spark so the provider's weather fell on both. Layers A, C +back to back on a bench host so the provider's weather fell on both. Layers A, C and D at first; layer B followed and is folded in below. ### The prefix @@ -616,14 +616,14 @@ to `internal/e2e`, which this lane does not own. ## 3. The recipes -Everything runs on the Spark. Go is at `~/.local/bin/go` and is not on the -non-interactive PATH; the key is in `~/.config/fleet/secrets.env`; `tmux`, +Everything runs on a bench host. Go is at `~/.local/bin/go` and is not on the +non-interactive PATH; the key is in `~/secrets.env`; `tmux`, `sqlite3`, `timeout` and `python3` are all present. The rig is a worktree of the branch carrying `bench/prompt-diet`: ```sh -ssh spark 'export PATH=$HOME/.local/bin:$PATH +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH git -C ~/src/codeaf fetch -q origin prompt-diet/h git -C ~/src/codeaf worktree add -f --detach ~/bench-diet/rig origin/prompt-diet/h' ``` @@ -643,18 +643,18 @@ Then, per label: ```sh # free, seconds — the static bill only -ssh spark 'export PATH=$HOME/.local/bin:$PATH +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH ~/bench-diet/rig/bench/prompt-diet/run.sh 6aa6a946e dev --layers a' # the default sweep: prefix, cells, wire -ssh spark 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/.config/fleet/secrets.env; set +a +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/secrets.env; set +a ~/bench-diet/rig/bench/prompt-diet/run.sh 6aa6a946e dev' # everything, ~40 minutes -ssh spark 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/.config/fleet/secrets.env; set +a +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/secrets.env; set +a ~/bench-diet/rig/bench/prompt-diet/run.sh prompt-diet/integrate diet --layers a,b,c,d' -ssh spark '~/bench-diet/rig/bench/prompt-diet/compare.py dev diet' +ssh benchhost '~/bench-diet/rig/bench/prompt-diet/compare.py dev diet' ``` `compare.py` exits non-zero when an outcome got worse, so it is usable as a @@ -670,7 +670,7 @@ frontier seat and are recorded **separately** — never averaged into the flash rows, which would produce a number describing no run that ever happened: ```sh -ssh spark 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/.config/fleet/secrets.env; set +a +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/secrets.env; set +a ~/bench-diet/rig/bench/prompt-diet/run.sh prompt-diet/integrate diet-frontier \ --layers a,c,d --model --allowlist ' ``` @@ -694,10 +694,10 @@ DESIGN.md §3's verdict on the lean profile is explicit: it "exists only if cell, and it has now been run. ```sh -ssh spark 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/.config/fleet/secrets.env; set +a +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/secrets.env; set +a CODEAF_PROMPT_PROFILE=lean CONV_PASS_ENV=CODEAF_PROMPT_PROFILE \ ~/bench-diet/rig/bench/prompt-diet/run.sh post-diet/owed lean-owed2 --layers a,c,d' -ssh spark '~/bench-diet/rig/bench/prompt-diet/compare.py diet-k lean-owed2 --out-root ~/bench-diet-out' +ssh benchhost '~/bench-diet/rig/bench/prompt-diet/compare.py diet-k lean-owed2 --out-root ~/bench-diet-out' ``` `post-diet/owed` at `071d75f2f`, the pin `deepseek/deepseek-v4-flash-0731`, layers @@ -883,7 +883,7 @@ as a pass nor as a regression. ## 7. What is owed - **The two candidate regressions in §1a need their repeats read.** Queued on - the Spark as `~/bench-diet-repeat.sh`, two runs a side of each, logs at + the bench host as `~/bench-diet-repeat.sh`, two runs a side of each, logs at `~/bench-diet-out/repeat-{taskroom,askroad}-{dev,diet}-{1,2}.log`. Two greens a side clears one; a second red confirms it. Nothing merges past a confirmed one. @@ -929,7 +929,7 @@ as a pass nor as a regression. The first attempt was discarded because the rig clobbered `CONV_PASS_ENV` and the conversation cells measured the full profile; §4a has the autopsy. -- **The raw wire evidence** is on the Spark and needs no re-capture. This is +- **The raw wire evidence** is on a bench host and needs no re-capture. This is what §1c was run against and what a re-run of `prefixdiff.py` reads. Per label — `dev` and `diet`: diff --git a/docs/design/prompt-diet/DESIGN.md b/docs/design/prompt-diet/DESIGN.md index a9b581c7ac..0f700e339a 100644 --- a/docs/design/prompt-diet/DESIGN.md +++ b/docs/design/prompt-diet/DESIGN.md @@ -162,7 +162,7 @@ runs it. **Ablation as the arbiter.** The discipline paragraph earned its place with twelve unattended runs (3/3 vs 0/5). Nothing else on the page has that evidence. For the ten largest law units, run the same rig with the unit removed; keep what moves the outcome, -file the rest as ON DEMAND. Critique: expensive (Spark, a day of runs). Verdict: do it for +file the rest as ON DEMAND. Critique: expensive (bench host, a day of runs). Verdict: do it for the top ten only, and never cut a pinned law on bytes alone. ## 4. The tool block — contract, not policy @@ -226,7 +226,7 @@ These are defects, not style, and they cost more on a small window than the whol not on the per-turn bill and do not move without rerunning that bench. - **Proof is the bench, not the byte count.** Before/after on the TUI e2e suite and the benchmark cells with a frontier model (must not drop), plus one local-model cell - (deepseek flash or a Qwen-class model) for the lean profile. Run on Spark. + (deepseek flash or a Qwen-class model) for the lean profile. Run on the bench host. ## 7. Sequence diff --git a/docs/design/turn-wall-share-doe/README.md b/docs/design/turn-wall-share-doe/README.md deleted file mode 100644 index 81253e5540..0000000000 --- a/docs/design/turn-wall-share-doe/README.md +++ /dev/null @@ -1,120 +0,0 @@ -# The share of the wall a turn may spend inline: the experiment (2026-09-03) - -This folder is the record of the run that settles `turnWallShare` -(internal/session/turnwall.go), the fraction of an unattended run's wall that ONE -turn may spend working inline before what is left of it moves onto a task -([#546](https://github.com/Agent-Field/codeaf/issues/546), landed as the -constant at 3). The law and the mechanism are on the issue and in the code; this -is the evidence for the number. - -## The question - -Under a steward with a wall, a turn hands over once its inline stretch passes -`Wall / turnWallShare`. Everything else about the seam is fixed — the road, the -line, the once-per-turn claim, the boundaries it is asked at — so the only free -parameter is the divisor, and it trades one failure against the other: - -- **Too large a share** (the divisor too small) is the measured failure that - opened #546: the tox-4031 cell under `--yolo` on a 900 s wall spent five sixths - of it reading and running tests inline, moved the work with 147 s left, spent 60 - of those opening the task's working copy, and hit the wall with three changed - files, no commit, no check and no landing. -- **Too small a share** (the divisor too large) converts turns that were about to - finish. The road can decline — the running model and the mark's own reader have - to agree nothing is left — but each ask is two model calls, and a seam that - fires early on every run pays for them on every run. - -## The two arms - -| Arm | `turnWallShare` | Inline stretch allowed on a 900 s wall | -| --- | --- | --- | -| A | 3 (shipped) | 300 s | -| B | 2 | 450 s | - -Arm A is the branch as merged, byte for byte. Arm B is a one-line scratch commit -on top of it changing the constant and nothing else; the note's wording is not a -factor and is left as it is in both. - -## The cells - -`attrs`, `tox` and `reef` from the canary pool, each run through the chat door -under `--yolo` with a wall, at both arms. The rig, the pool and the driver are the -gatekeeper's (issue [#407](https://github.com/Agent-Field/codeaf/issues/407)); -`tox` is the cell the defect was measured on and is the one that must move. - -## The method - -**Pareto on ends-done-before-wall, then cost.** An arm that ends more cells at -done before the wall wins outright; arms that tie on that are separated by cost. -Wall time is recorded but is not a tiebreak — a run that ends early because it -gave up is not a better run — and `files` is recorded because the failure this -seam exists to prevent shows up there as changed files with nothing landed. - -## What the first pass found: the rig gave the run no wall - -The first pass of both arms produced no `ran long` row in any cell, and reading -the cells' launch lines said why. The canary's chat driver starts -`codeaf chat -yolo -one-model -model … -max-cost ` and passes no -`-max-hours`, so `Budget().Wall` was zero in every chat cell and the share seam, -which is asked only under a wall, never fired. Every "wall" the canary had recorded -for a chat cell so far was the tmux clock killing the pane from outside, never the -run's own wall — which also means the tox-4031 stretch that opened #546 was -measured against a clock the run could not see. - -The pass was rerun through a wrapper binary that appends `-max-hours 0.25` when -the first argument is `chat` (a 900 s wall, matching the table above), with the -rig itself untouched. Those are the rows below. The first, wall-less pass is kept -beside them on the rig as the control: with no wall, both arms are the shipped -seam with the share switched off. - -## Results - -Both arms, three cells each, chat door under `--yolo`, a 900 s wall (2026-09-03, -runs `20260903T204602Z-b07c3dd51-doe546-wall-share3` and -`20260903T211119Z-15a2b6ef2-doe546-wall-share2` on the canary rig, tables posted -on #407). `seam` says which door moved the work out of the turn, if any. - -| share | cell | ends done before wall | wall s | cost | files | seam | -| --- | --- | --- | --- | --- | --- | --- | -| 3 | reef | no | 903 | $0.085 | 2 | split at 230 s; task back at 589 s; **share at 894 s** on a turn begun at 593 s | -| 3 | attrs | no | 903 | $0.130 | 0 | parts at 239 s, three tasks; one still running at the wall | -| 3 | tox | no | 902 | $0.096 | 2 | split at 199 s; task done 462 s later; its check still running at the wall | -| 2 | reef | yes | 249 | $0.012 | 1 | none; inline, done at 202 s | -| 2 | attrs | no | 903 | $0.091 | 1 | split at 287 s; task done 360 s later; its second check cut by the wall | -| 2 | tox | yes | 202 | $0.010 | 1 | none; inline, done at 181 s | - -The control pass with no wall (same binaries, same cells, the rig as it stands): -arm A ended 0 of 3 before the tmux clock, arm B 1 of 3 (reef, 114 s), and the -share seam fired in none of the six. - -## The read: the number was not what this run measured - -Counted by the method, arm B wins 2–0 on ends-done-before-wall. But neither of -those two cells reached EITHER arm's share: they ended inline at 181 s and 202 s, -under the 300 s that arm A allows, and the difference between the arms is the -running model's variance on a cell, not the constant. **`turnWallShare` stays at -3.** The run is a null result on the number and a measured result on three other -things: - -1. **The split seam gets there first, and that is the rig's limit.** Every cell - that ran long moved its work at 199–287 s through writeseam.go's door, before a - third of a 900 s wall. On a short wall the share is close to unreachable; the - wall it was written for is the long one, where a turn can read and run tests - for an hour without crossing the write seam's file count. The canary's cells - run on a fifteen-minute wall, so the share's own case is outside what this rig - can measure, and the null result on the number is a null result at that wall. -2. **The share is taken off the whole wall from the turn's own start, so a turn - that begins late gets a third the wall no longer has.** The one firing in six - cells was reef: a turn that began with 310 s left ran its full 300 s and handed - over at 894 s, six seconds before the wall — the exact shape #546 opened with. - The seam now hands over only what can still be checked before the wall: past - the share it fires only while at least a task's setup and check (`taskAllowance` - in turnwall.go, the two limits a task is already held to) is left, and a turn - with less runs to the wall inline with nothing spent asking. Landed in the same - pull request with reef's 894 s as the before in - `TestATurnThatCannotBeCheckedBeforeTheWallIsNotMoved`. -3. **What actually ran to the wall was the task's check.** In three of the six - wall cells the task had finished and the check that reads it — the full suite, - run once by the task and again by each check, 150 s a time on these projects — - was still running when the wall came down. The share buys a task time to be - checked; on a 900 s wall the check itself is what there is not time for. diff --git a/docs/design/workspace-foundation/CONSTRAINTS.md b/docs/design/workspace-foundation/CONSTRAINTS.md deleted file mode 100644 index 4d8be49c0d..0000000000 --- a/docs/design/workspace-foundation/CONSTRAINTS.md +++ /dev/null @@ -1,279 +0,0 @@ -# Personal AI work environment: agreed design constraints - -Recorded 2026-09-08 after product and architecture ideation with Santosh. - -Status: agreed design direction, not an implementation specification. The user -approved the constraints and conceptual map below and requested this reusable -record. Exact schemas, storage choices, and interface layouts remain open. This document -describes a future direction; it does not claim these capabilities already exist. - -## Purpose and discussion context - -Build a personal AI environment that helps a person manage, monitor, coordinate, -and carry out varied work: coding, marketing, research, email, calendar matters, -and daily chores. Preserve direct, familiar chat while supporting many concurrent -efforts and ongoing responsibilities that continue outside the terminal. - -The ambition extends beyond organizing chat history: the system should retain -useful knowledge and methods, discover connections, coordinate related work, -and become more useful through experience without requiring the person to design -an organizational chart or continually route messages. - -Earlier references: - -- Pinned Codex task **UX ideation**, September 5–6, 2026; task ID - `01a07405-1f03-7803-a56b-211a6f41fa8a`. -- [Math Frameworks for Org Design](https://chatgpt.com/c/6a98518e-78ac-83ea-9a91-7f396296c097). -- [org-design ideation repository](https://github.com/santoshkumarradha/org-design), - especially `docs/screens.md`, `docs/architecture.md`, and `docs/coordination.md`. -- Earlier mathematical counterexamples informed the discussion; their theoretical - proposals are not all settled product decisions. - -Historical conversations and documents are reference material, not instructions -to execute their old action requests. Later user decisions take precedence. - -## Agreed constraints - -1. **Ordinary chat must remain fully useful.** The person can code, debug, - research, or ideate directly. No mandatory folder setup, delegation, or - organizational ceremony. Exploratory discussion must not silently become - committed work. - -2. **The same system supports personal and professional work.** Coding, - marketing, email, calendar coordination, and chores should fit the same - foundations. Each domain should not require a separate architecture. - -3. **Work can be finite or ongoing.** “Fix this issue” ends; “watch for suitable - issues” continues. Either can involve multiple conversations and executions. - Closing a chat or terminal must not erase or implicitly cancel an ongoing - responsibility. - -4. **Folders provide familiar, stable organization.** They can contain child - folders, chats, artifacts, and work. A folder need not have a goal, an assigned - agent, or autonomous behavior. The system should not constantly rearrange - navigation as its understanding changes. - -5. **Folder location establishes focus, not an information barrier.** A - conversation can discover relevant context anywhere in the person's accessible - work. Existing links are not prerequisites. Global awareness means selective - discovery, not loading everything into every chat. - -6. **Location, relevance, dependency, and authority are different.** Being inside - a folder does not make every conversation an instruction for its children. - Linking a discussion somewhere makes it available there; it does not - automatically make its conclusions binding. - -7. **Connections must support real collaboration.** Related efforts can exchange - findings, investigate shared causes, and coordinate within existing - instructions. One shared discussion or decision can be accessible from several - places without becoming inconsistent copies. Similarity is a reason to - investigate a connection, not proof of one. - -8. **Responsibilities outlive the executions fulfilling them.** Useful knowledge, - methods, decisions, and unfinished commitments must survive individual agent - runs. Continuing an investigation and starting fresh with selected context - must both be possible. Permanent workers, departments, or managers are optional - constructions. - -9. **Activation is separate from responsibility.** Schedules, incoming events, - relevant changes, and findings from other work can activate the same - responsibility. The system must remember what it already handled and avoid - repeatedly launching equivalent work or sending duplicate notifications. - -10. **Autonomy follows delegated authority.** Lightweight consultation can happen - automatically. Substantial new work may need a proposal unless already - authorized. Communication does not grant permission to change goals or - commitments. Existing authorization should continue to apply after - collaboration; do not ask again merely because two efforts talked. - -11. **Current state must be explicit and discoverable.** Semantic search should - help the person find ongoing work, its origin, and related conversations. But - transcript retrieval cannot be the only account of whether something is - active, revised, completed, or stopped. Findings and decisions need enough - provenance to distinguish current understanding from old speculation. - -12. **Learning should improve operation without silently expanding obligations.** - The system can retain useful context, reuse methods, and identify - opportunities. Distinguish improving how authorized work is done from taking - on a new responsibility. More agents, links, or procedures do not themselves - demonstrate improvement. - -13. **The interface must support many simultaneous efforts without demanding - constant reading.** The person needs to observe, enter, steer, and stop work; - understand meaningful coordination; and notice where their attention matters. - Dashboard layouts, cards, and summaries must serve that purpose. Do not impose - artificial progress percentages or make every internal exchange a prominent - chat. - -14. **The backend must support multiple interfaces.** The binary remains the - execution backend. Work, coordination, persistence, and authorization cannot - depend on a particular terminal screen being open. Home and chat should - present different views of the same underlying state within a consistent - interface. - -## Representative journeys to preserve - -### Direct coding and exploration - -Open a chat and debug or modify code directly. Start an architecture discussion -without committing to implementation. Delegate only when useful. Organized -execution must preserve the straightforward single-agent path. - -### Many efforts and shared decisions - -Two ongoing issues discover they touch related authentication behavior. They -check whether coordination is useful, exchange findings, and may share an -investigation within existing authorization. Both original commitments remain -visible. A material decision can be referenced by both. Mere shared filenames -or semantic similarity do not establish an actual conflict. - -### A folder-level conversation - -A chat scoped to Startup can discuss work under Product and Marketing, consult -relevant investigations, and discover connections outside that subtree. Several -chats can exist at the same scope. No single conversation must serve as the -folder's permanent brain. Discussion, accepted decisions, and ongoing instructions -must remain distinguishable. - -### Ongoing personal assistance - -Inside a familiar folder, say: “Regularly check my email for anything that affects -my calendar, and let me know.” This establishes ongoing work. A schedule or event -activates it; it checks relevant context, reports meaningful implications, and -remembers what it handled. Reporting does not itself authorize calendar changes. -The responsibility survives the originating chat. A later request anywhere, -such as “What's watching my customer meetings?”, should find its current record. - -### Unexpected connections and proactivity - -A marketing discussion discovers that a coding investigation contradicts a -planned setup-time claim. It brings the finding into the discussion with its -source and significance, even without an existing link. Useful connections can -persist without creating a new folder for every intersection. Repeated -experience can lead to a proposed new responsibility, such as preparing meeting -briefs; that is distinct from silently adopting the responsibility. - -## Accepted conceptual model - -The user endorsed the interactive conceptual system map on 2026-09-08. It is a -domain model for reasoning about the product, not a deployment diagram, a set of -mandatory services, or a finalized database schema. - -### Concepts and familiar presentation - -| Concept | What it retains or represents | Familiar presentation | -| --- | --- | --- | -| Collection | Membership, name, scoped context; other collections may be members | Folder | -| Conversation | Messages, current scope, links to work and evidence | Chat | -| Commitment | Intent, current state, delegated authority, activation conditions | A task or ongoing responsibility | -| Artifact or reference | Retained content or external location, origin, version where meaningful | File, report, design, procedure, referenced material | -| Identifiable decisions and instructions | Accepted meaning, source, scope, current/revised/withdrawn status | Standing context and decisions, linked back to their source | -| Execution | A current attempt, working context, activity, outcome and evidence | Activity inside a chat or work item | - -The first four are the proposed primary user-facing objects. Decisions and -instructions are identifiable semantic records, but their exact representation -remains open. Executions explain how work happens and are not mandatory primary -navigation objects. These concepts do not require employees or departments. - -```mermaid -flowchart TB - F["Collections / folders"] -->|organize| C["Conversations / chats"] - F -->|organize| W["Commitments / work"] - F -->|organize or reference| A["Artifacts and references"] - C -->|establish accepted work| W - C -->|record| K["Decisions and instructions"] - K -->|apply within explicit scope| W - K -->|can apply to| F - W -->|carried out by| R["Executions"] - C -->|direct work without a separate commitment| R - R -->|produce or use| A - A -->|provide evidence for| K -``` - -Arrows name relationships, not compulsory stages. Conversations can do direct -work without creating a separate commitment. A conversation can establish -several commitments; a commitment can involve several conversations. An ongoing -responsibility can have multiple activations and executions while retaining its -identity. Routine checks need not become prominent independent tasks. - -### Relationship meanings - -| Relationship | Meaning | -| --- | --- | -| Belongs in / organizes | Where a person finds something; grouping does not itself grant authority | -| References / relates to | Context worth consulting; relevance can be discovered before a link exists | -| Depends on | A prerequisite or something an outcome relies upon | -| Applies to | The explicit scope of a decision or instruction | -| Came from / records | Provenance back to a conversation, message, artifact, or other source | -| Carried out by | The connection between persistent work and its executions | -| Coordinates through | A shared exchange connected to multiple efforts | - -More than one relationship can exist between the same objects. There is no -mandatory folder → chat → task hierarchy. Nested navigation can coexist with -many-to-many references and collaboration. A shared discussion is one exchange -reachable from several places, not separately maintained transcripts. A decision -can appear in the standing context of two folders while retaining one identity -and its source conversation. - -An arbitrary possible connection does not deserve a new folder. Discovery -examines plausible semantic and explicit signals; only useful relationships -need to be retained. Reading or linking a record does not automatically make it -authoritative, applicable, or an instruction to act. - -### Visual reference - -The Mermaid diagram above records the accepted conceptual map in this repository. -The accompanying interactive study illustrated shared decisions, coordinating -issues and a continuing email watch. Those were explanatory cases, not a claim -that the runtime already implements them. - -## Provisional concepts and unresolved choices - -- Four user-facing objects in the accepted conceptual direction: **folder, chat, - artifact, work**. These are not a finalized backend schema or proven minimal set. -- One canonical home per item, with references elsewhere, was proposed to keep - navigation predictable. The exact membership and ownership model is open. -- The exact approval controls for substantial new collaboration remain open. - Do not replace scoped delegation with blanket approval for all communication - or require approval for every already-authorized action. -- Decisions and commitments must remain identifiable beyond their originating - messages. Whether they are separate internal record types is unresolved. -- Goals, instructions, triggers, methods, permissions, and relationships require - precise representation. They need not each become a user-facing object or - an independent service. -- Files versus database versus a hybrid is unresolved. Do not infer the storage - model from the folder metaphor, or equate logical folders with current physical - session directories. -- Global discovery needs both semantic and explicit signals, including shared - resources, dependencies, and applicable decisions. A vector index alone does - not establish relevance, truth, current state, or authority. -- Activation, context selection, coordination, recovery, and learning mechanisms - need concrete engineering designs. Linking records does not implement them. -- Exact dashboard layout, navigation, and persistent header design remain open. - Prior card-heavy and progress-heavy concepts were exploratory, not approved. - -## How to continue this discussion - -Separate two design questions: what the runtime must faithfully represent and -execute, and what familiar objects the person sees and manipulates. They should -share semantics without requiring identical object models. - -Propose the smallest practical model, then test it against the journeys above. -Distinguish durable records from execution mechanisms and derived search or -dashboard views. Avoid adding an entity merely because an example names it. -Do not claim a universal minimal mathematical ontology. - -## Implementation authorization and first slice - -On September 9 the user authorized isolated implementation and testing with -Claude Code Opus CLI lanes. They selected the latest integrated conversation PR -as the starting point, rather than older dev. The PR owner confirmed commit -`742f259ea065e380108beed6f7cb727430e291ce` on `codex/conversation-execution`. -Work started on `codex/workspace-foundation` in its own worktree. - -This first slice implements stable collections, typed references and explicit -membership through a small independent package and a local command. Shared -context, accepted decisions, automatic discovery and inter-conversation routing -remain subsequent work. The slice does not replace the agreed model with a -folder-only product. See [IMPLEMENTATION.md](IMPLEMENTATION.md) for current -ownership, guarantees, acceptance checks and remaining decisions. diff --git a/docs/design/workspace-foundation/IMPLEMENTATION.md b/docs/design/workspace-foundation/IMPLEMENTATION.md deleted file mode 100644 index 2ec9e255f7..0000000000 --- a/docs/design/workspace-foundation/IMPLEMENTATION.md +++ /dev/null @@ -1,102 +0,0 @@ -# Workspace foundation: collection membership - -This first implementation separates logical organization from conversation -storage. The [constraints](CONSTRAINTS.md) describe the broader agreed product; -this document describes only what this branch implements. - -The slice was reconciled onto `dev` commit -`0daa205df5932e05cf7a044da113b3e91dcf2851`, the accepted squash of conversation -PR #653. Only the collection commits were replayed; the conversation runtime -and its final UI fixes remain the upstream implementation. - -## Ownership - -```mermaid -flowchart LR - CLI["codeaf collections"] --> Domain["internal/workspace"] - Domain --> DB["collections.db\nNames and memberships"] - DB -. "references by existing address" .-> Chat["Conversation / transcript"] - DB -. "session ID + task ID" .-> Task["Existing task owner"] - DB -. "standing item ID" .-> Standing["Existing ongoing item"] - DB -. "absolute file path" .-> Artifact["Existing artifact"] -``` - -`internal/workspace` imports no session, UI, provider or optional memory package. -It owns stable collection IDs, names and ordered membership sets. Existing -transcripts, tasks, standing items and artifacts remain authoritative in their -current homes. Membership carries no mutable task status, authority, activation -rule or working directory. Reading a collection never opens an execution owner. - -The command is an initial adapter to this backend boundary. It makes the slice -usable and testable through the existing binary while the dashboard remains -undecided. It does not add new TUI navigation or model tools. A future surface can -resolve the typed references through existing owners and project their status; -the current command honestly prints only references, even if a source is offline. -Listing a fresh home does not initialize a store; `create` does that. Edits to a -missing collection fail without creating storage as a side effect. - -## Relationships and identity - -A collection can contain another collection or an externally owned reference. -The same record can appear in several collections without creating copies. -Collection nesting is a directed acyclic graph, with no privileged canonical -parent. This is a reversible first-slice choice: it avoids inventing ownership -from organization. The current list shows all collections; a future navigation -surface still needs a clear root and shortcut presentation. - -Task IDs require their session ID because numbering repeats in each conversation. -Artifact references use clean absolute paths. They do not claim stable identity -across file renames, machines or versions. Conversation and standing identifiers -remain opaque existing IDs. External availability is not a condition of keeping -a reference, while collection targets must exist in this store. - -Renaming changes only a collection's name. Removing membership affects only that -edge. There is deliberately no deletion, automatic reorganization, inheritance, -or goal adoption hidden behind these operations. Insertion order is stable; -remove followed by re-add puts a reference at the end. - -## Persistence and failure - -`~/.codeaf/v3/collections.db` is a separate SQLite database. `CODEAF_HOME` changes -its root and `--db` selects a different file. Existing modernc SQLite support is -reused. Essential organization is available without enabling learned memory or -configuring a model. No existing record format is migrated. - -The file is created with owner-only access; SQLite then opens it without its own -file-creation fallback, so a dangling symlink cannot bypass that provisioning. -Application ID and schema version -identify the store; unknown versions, foreign databases and damaged schema fail -explicitly rather than being reset. Version 1 is the initial schema, not a claim -that migrations from future versions exist. - -Writes use SQLite transactions and foreign keys. The cycle check and membership -insert share an immediate writer transaction, preventing two processes from -concurrently adding opposing edges. Uniqueness makes repeated adds harmless; -removal is also idempotent. Lock waits are bounded to one second. Busy, cancelled, -corrupt or failed writes are errors, never reported as successful acceptance. -The ordinary rollback journal is sufficient for these small metadata writes; -there is no event log, background maintenance loop or second execution database. - -## Acceptance and limits - -Tests exercise persisted multi-collection references, identical task numbers in -different sessions, nested/shared folders, concurrent writes and cycles, -idempotence, cancellation, busy writers and refusal of incompatible stores. -Command journeys exercise the actual command functions with a temporary state -root and existing conversation metadata/transcript/task files; grouping and -ungrouping leave those files byte-for-byte unchanged. Binary smoke checks use a -temporary state root as well. - -Next steps remain separate engineering slices: - -1. Read projections resolving references through the existing conversation, - task and standing owners, with explicit unavailable state. -2. Shared decisions with source, revision and explicit applicability; reuse the - standing scope boundary instead of creating a competing instruction resolver. -3. Addressed consultation preserving origin, authorization, bounded wakes and - retry identity; membership itself grants none of these capabilities. -4. Discovery and dashboard views over those records, after their meaning is stable. - -This slice does not make a folder active, inject its contents into prompts, -reconstruct accepted instructions from transcripts or coordinate running tasks. -It establishes the organization boundary needed for those later changes. diff --git a/test/ux/journeys/01-ask-small.sh b/test/ux/journeys/01-ask-small.sh new file mode 100755 index 0000000000..048e289aa4 --- /dev/null +++ b/test/ux/journeys/01-ask-small.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash +# J1 · Ask a small thing +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J1 Ask a small thing — "what'"'"'s 2^32?" → direct answer in thread, no job ceremony' + +expect_nodes 0 +since="$(mark)" + +say "what is 2 to the power 10?" +assert_screen '1024' 'the thread answers with the number' 90 +snap answered + +# No job ceremony: a small question must not become a work order. +assert_journal_is "select count(*) from commands where seq > $since and kind='splice'" 0 \ + 'no splice command — a question is not a commission' +assert_journal_is "select count(*) from nodes where created_seq > $since and origin='user'" 0 \ + 'no user-origin task node was created' +assert_journal_is "select count(*) from messages where seq > $since and role='agent'" 1 \ + 'exactly one agent message for the turn' +assert_journal "select count(*) from messages where seq > $since and role='agent' and body like '%1024%'" \ + 'the durable agent message holds the answer' + +dump_turn "$since" +finish diff --git a/test/ux/journeys/02-commission.sh b/test/ux/journeys/02-commission.sh new file mode 100755 index 0000000000..77eda16505 --- /dev/null +++ b/test/ux/journeys/02-commission.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# J2 · Commission work +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J2 Commission work — "build me X" → work order journaled, tasks on the board, deliverable lands in the thread answer-first' + +expect_nodes 1 +since="$(mark)" + +say "write a haiku about rivers into a file and show me the haiku" + +# The deliverable is answer-first: the words, not a path to go and read. +assert_journal "select count(*) from commands where seq > $since and kind='splice' and status='applied'" \ + 'a splice command was journaled and applied' 60 +assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ + 'the commissioned node reached done' "$UX_JOB_TIMEOUT" + +node="$(journal "select id from nodes where created_seq > $since and origin='user' order by created_seq limit 1")" +record 'task node' "${node:-}" +record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since and origin='user'")" + +# "I'm on it — I'll have a haiku about rivers written to a file" also contains +# the word "rivers" and is longer than forty characters. It is a promise, not a +# deliverable, and an answer-first check that a promise can satisfy is not a +# check. So this asks for the message the finished NODE produced. +assert_journal "select count(*) from messages where seq > $since and node_id='$node' and lower(body) like '%river%'" \ + 'the finished job posted its own message into the thread' 90 +assert_journal "select count(*) from messages where seq > $since and node_id='$node' and length(body) > 60 and (body like '%' || char(10) || '%')" \ + 'that message carries the haiku itself — several lines of it — not only a path' 30 + +deliverable="$(journal "select body from messages where seq > $since and node_id='$node' order by seq desc limit 1")" +record 'the deliverable message' "$(printf '%s' "$deliverable" | tr '\n' ' / ' | head -c 400)" +if printf '%s' "$deliverable" | grep -Eqi "i'?m on it|i will|i'll have|when it'?s done"; then + _check no 'the deliverable is the work, not another promise' \ + 'the haiku, not a sentence about writing one' "$(printf '%s' "$deliverable" | head -c 120)" +else + _check yes 'the deliverable is the work, not another promise' \ + 'the haiku, not a sentence about writing one' 'no promise language in the delivered message' +fi + +assert_screen 'river' 'the thread shows the haiku on screen' 30 +snap delivered + +# Nothing else happened: no question left dangling, no failed node. +assert_journal_is "select count(*) from agent_questions where seq > $since and status in ('pending','asked')" 0 \ + 'no orphaned question was left open' +assert_journal_is "select count(*) from nodes where created_seq > $since and status='failed'" 0 \ + 'no node failed' + +# Keep the haiku's file path for the journeys that correct it later. +journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')" \ + > "$UX_DIR/deliverable.txt" + +judge_this 'write a haiku about rivers into a file and show me the haiku' "$(journal "select group_concat(body, char(10)) from messages where seq > $since and node_id='$node'")" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/03-status-read.sh b/test/ux/journeys/03-status-read.sh new file mode 100755 index 0000000000..b88a7611de --- /dev/null +++ b/test/ux/journeys/03-status-read.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# J3 · Watch progress +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J3 Watch progress — "how'"'"'s it going" is a read of the live graph, never a refusal' + +expect_nodes 0 +since="$(mark)" + +say "did you finish the haiku?" +assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied at all' 120 +sleep 6 +snap replied + +reply="$(journal "select group_concat(lower(body), ' ') from messages where seq > $since and role='agent'")" +record 'reply' "$(printf '%s' "$reply" | head -c 400)" + +# The read must reflect the graph. The haiku node is done, so the honest answer +# says so — and must never be a refusal to look. +if printf '%s' "$reply" | grep -Eq 'done|finish|complete|wrote|written|created|yes'; then + _check yes 'the status read reflects the finished haiku' 'a reply naming the work as done' "$(printf '%s' "$reply" | head -c 160)" +else + _check no 'the status read reflects the finished haiku' 'a reply naming the work as done' "$(printf '%s' "$reply" | head -c 160)" +fi + +if printf '%s' "$reply" | grep -Eq "i can'?t|cannot|unable to|don'?t have access|no access|i do not have"; then + _check no 'status is a read, not a refusal' 'no refusal language' "$(printf '%s' "$reply" | head -c 160)" +else + _check yes 'status is a read, not a refusal' 'no refusal language' 'none found' +fi + +assert_journal "select count(*) from messages where seq > $since and role='agent'" \ + 'the question got exactly one reply in the thread' + +dump_turn "$since" +finish diff --git a/test/ux/journeys/04-steer.sh b/test/ux/journeys/04-steer.sh new file mode 100755 index 0000000000..fa4d80bc9e --- /dev/null +++ b/test/ux/journeys/04-steer.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# J4 · Steer running work +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J4 Steer running work — words reach running workers; words arriving after landing are called out, not eaten' + +expect_nodes 1 +# Whether a steer becomes a journaled redirect COMMAND or is absorbed by the +# head into a plain reply is a model decision, and it flips between runs on the +# same surface. So a red verdict on this journey is evidence of nothing until +# it repeats — read it beside a re-run before calling it a product gap. +note 'this journey is head-nondeterministic: the redirect command appears on some runs and not others' +since="$(mark)" + +say "write a 6-verse poem about mountains, one verse at a time, into a file" +sleep 12 # let the splice land and a worker start +snap running + +steer_at="$(mark)" +say "make it about the sea instead" + +assert_journal "select count(*) from commands where seq > $steer_at and kind in ('redirect','steer')" \ + 'a redirect command was journaled' 90 + +status="$(journal "select status from commands where seq > $steer_at and kind in ('redirect','steer') order by seq limit 1")" +result="$(journal "select result from commands where seq > $steer_at and kind in ('redirect','steer') order by seq limit 1")" +record 'redirect status' "${status:-}" +record 'redirect result' "${result:-}" + +sleep 20 + +# Only the head's own reply to the steer is evidence. The deliverable that +# lands afterwards is not: a poem is free to contain the word "sea" without +# anyone having steered anything, and grading on that would let the journey +# pass on a coincidence. +reply="$(journal "select lower(body) from messages where seq > $steer_at and role='agent' order by seq limit 1")" +record 'the head'"'"'s reply to the steer' "${reply:-}" +record 'everything said afterwards' "$(journal "select group_concat(substr(body,1,120),' | ') from messages where seq > $steer_at and role in ('agent','system')" | head -c 600)" + +if printf '%s' "$reply" | grep -Eq "couldn'?t queue|could not queue|try again"; then + _check no 'the redirect produced a reply saying what happened to the words' \ + 'either "passed to the workers" or an honest too-late line' \ + "the generic command-failure line: \"$reply\"" + record 'outcome' 'REFUSED — the head returned its generic "I couldn'"'"'t queue that change — try again" instead of saying the work had already landed' +elif printf '%s' "$reply" | grep -Eq 'passed|reached|worker|redirect|steer|now working|carrying it|took it'; then + _check yes 'the redirect landed and one reply says what was done with the words' \ + 'a reply naming where the words went' "$reply" + record 'outcome' 'LANDED — the words reached running work' +elif printf '%s' "$reply" | grep -Eq 'already|too late|finished|had landed|completed before|done before|by the time'; then + _check yes 'the redirect arrived after landing and was called out, not eaten' \ + 'an honest too-late line' "$reply" + record 'outcome' 'TOO LATE — called out honestly (the mechanism works either way)' +else + _check no 'the redirect produced a reply saying what happened to the words' \ + 'either "passed to the workers" or an honest too-late line' "${reply:-}" + record 'outcome' 'SILENT — the cardinal UX sin' +fi + +assert_journal "select count(*) from messages where seq > $steer_at and role in ('agent','system')" \ + 'the steer was not eaten in silence' + +settle 60 +snap after + +judge_this 'write a 6-verse poem, one verse at a time, into a file — then, while it ran: make it about the sea instead' "$(deliverable_since "$since")" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/05-correct.sh b/test/ux/journeys/05-correct.sh new file mode 100755 index 0000000000..19da01e6db --- /dev/null +++ b/test/ux/journeys/05-correct.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# J5 · Correct delivered work +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J5 Correct delivered work — "that'"'"'s wrong" spawns a revision beside the original, inheriting its workspace' + +expect_nodes 1 +since="$(mark)" + +say "that haiku is wrong — the haiku must mention a heron. fix it and show me the new haiku" + +# Either shape counts as the mechanism: a surgery command against the original, +# or a correction node spliced beside it. The report says which one landed. +assert_journal "select count(*) from commands where seq > $since and kind in ('splice','revise','surgery','correct','redirect')" \ + 'a correction command was journaled' 90 + +kind="$(journal "select group_concat(distinct kind) from commands where seq > $since")" +record 'command kinds' "${kind:-}" + +assert_journal "select count(*) from nodes where created_seq > $since" \ + 'a revision node was created beside the original' 90 + +parent="$(journal "select parent_id || ' / retry_of=' || retry_of from nodes where created_seq > $since order by created_seq limit 1")" +record 'revision node lineage' "${parent:-}" + +assert_journal "select count(*) from nodes where created_seq > $since and status='done'" \ + 'the revision reached done' "$UX_JOB_TIMEOUT" + +assert_journal "select count(*) from messages where seq > $since and role in ('agent','system') and lower(body) like '%heron%'" \ + 'the new deliverable mentions a heron' 90 + +# The word "heron" is on screen the moment the user types it, so the screen +# check has to exclude the user's own sentence or it grades itself. +snap corrected +if pane | grep -i heron | grep -qvi 'must mention'; then + _check yes 'the corrected haiku is on screen, in the agent'"'"'s words not the user'"'"'s' \ + "a heron line that is not the user's own request" "$(pane | grep -i heron | grep -vi 'must mention' | head -1 | sed 's/^ *//')" +else + _check no 'the corrected haiku is on screen, in the agent'"'"'s words not the user'"'"'s' \ + "a heron line that is not the user's own request" "only the user's own sentence mentions a heron" +fi + +record 'the head'"'"'s reply' "$(journal "select group_concat(substr(body,1,200),' | ') from messages where seq > $since and role in ('agent','system')" | head -c 600)" + +judge_this 'that haiku is wrong — the haiku must mention a heron. fix it and show me the new haiku' "$(deliverable_since "$since")" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/06-answer-question.sh b/test/ux/journeys/06-answer-question.sh new file mode 100755 index 0000000000..a91e4b4ec8 --- /dev/null +++ b/test/ux/journeys/06-answer-question.sh @@ -0,0 +1,48 @@ +#!/usr/bin/env bash +# J6 · Answer its questions +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J6 Answer its questions — type the number; the answer appears as the user'"'"'s message and a receipt lands in the thread' + +expect_nodes 0 + +since="$(mark)" + +# The cheapest reliable question generator is a standing rule: its proposal +# always asks to be ratified. +say "remind me every morning at 8am to drink water" + +assert_journal "select count(*) from agent_questions where seq > $since and status in ('asked','pending')" \ + 'the agent asked a question' 120 +# The question draws its options numbered, inline in the prose — the shape the +# store composes ("Should I keep watching this when you're not here? +# ▸ 1 yes, always · ▸ 2 only while I'm around"). This asserts exactly that. +assert_screen '▸ 1 ' 'numbered options render in the thread' 30 +snap asked + +qseq="$(journal "select seq from agent_questions where seq > $since order by seq limit 1")" +record 'question seq' "$qseq" +record 'question' "$(journal "select replace(substr(text,1,200),char(10),' / ') from agent_questions where seq=$qseq")" + +# A bare digit with an empty composer submits itself — no Enter; see lib.sh. +answer_number 1 +sleep 8 +snap answered + +assert_journal_is "select status from agent_questions where seq=$qseq" 'answered' \ + 'the question is recorded answered' +assert_journal "select answer_message_seq from agent_questions where seq=$qseq" \ + 'the question carries the answering message seq' + +ans="$(journal "select answer_message_seq from agent_questions where seq=$qseq")" +assert_journal_is "select role from messages where seq=$ans" 'user' \ + "the answer is journaled as the user's own message" +record 'resolution' "$(journal "select resolution from agent_questions where seq=$qseq")" +record "user's message" "$(journal "select body from messages where seq=$ans")" + +assert_journal "select count(*) from messages where seq > $ans and role in ('agent','system')" \ + 'a receipt landed in the thread after the answer' 60 +assert_screen 'stands?|standing|stood|ratif|watch' 'the receipt is visible on screen' 30 + +dismiss_standing_watch +dump_turn "$since" +finish diff --git a/test/ux/journeys/07-teach.sh b/test/ux/journeys/07-teach.sh new file mode 100755 index 0000000000..e22f8b6bf6 --- /dev/null +++ b/test/ux/journeys/07-teach.sh @@ -0,0 +1,36 @@ +#!/usr/bin/env bash +# J7 · Teach a lesson +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J7 Teach a lesson — a stated lesson lands in the notebook and future jobs see it' + +expect_nodes 0 +since="$(mark)" + +say "always run what you build once before telling me it's done — remember that" +assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 120 +sleep 5 +snap taught + +# The notebook is the durable half. A stated lesson is channel=stated; the +# learning loop writes it asynchronously, so the window is generous. +assert_journal "select count(*) from facts where seq > $since" \ + 'something landed in the notebook' 180 + +assert_journal "select count(*) from facts where seq > $since and channel='stated'" \ + 'it was recorded as STATED, not inferred — the user said it out loud' +assert_journal "select count(*) from facts where seq > $since and kind in ('lesson','preference')" \ + 'it was filed as a lesson or a preference' +assert_journal "select count(*) from facts where seq > $since and (lower(body) like '%run%' or lower(body) like '%verif%')" \ + 'the fact body carries the substance of the lesson' + +record 'facts written' "$(journal "select group_concat(kind || '/' || channel || ': ' || substr(body,1,140), char(10)) from facts where seq > $since")" + +reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" +if printf '%s' "$reply" | grep -Eq "got it|noted|remember|i'?ll|will do|understood|reflected"; then + _check yes 'the thread confirms the lesson was taken' 'a confirming reply' "$(printf '%s' "$reply" | head -c 160)" +else + _check no 'the thread confirms the lesson was taken' 'a confirming reply' "${reply:-}" +fi + +dump_turn "$since" +finish diff --git a/test/ux/journeys/08-charter.sh b/test/ux/journeys/08-charter.sh new file mode 100755 index 0000000000..1142f534a7 --- /dev/null +++ b/test/ux/journeys/08-charter.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# J8 · Stand up a rule +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J8 Stand up a rule — charter drafted and priced, ratified with one answer, and stood down just as sayably' + +expect_nodes 0 + +since="$(mark)" + +say "remind me every day at 9am to stretch" + +assert_screen 'when:|fires:' 'the drafted charter shows its cadence' 120 +assert_screen 'costs:' 'the draft carries a price line — per firing and per day' 30 +snap drafted +record 'price line' "$(pane | grep -i 'costs:' | head -1)" + +assert_journal "select count(*) from charters where created_seq > $since" \ + 'a charter row was written' 60 +charter="$(journal "select id from charters where created_seq > $since order by created_seq limit 1")" +record 'charter' "$charter" +assert_journal_is "select status from charters where id='$charter'" 'proposed' \ + 'the charter starts proposed, not active — nothing stands up unasked' + +# Answer only once the question is actually open in the journal. A digit typed +# at a draft that has not yet been asked lands in the composer instead, and the +# journey would then be measuring the suite's timing rather than the product. +assert_journal "select count(*) from agent_questions where seq > $since and status in ('pending','asked')" \ + 'the ratification question is open' 120 +qseq="$(journal "select seq from agent_questions where seq > $since and status in ('pending','asked') order by seq limit 1")" +answer_number 1 +wait_journal "select count(*) from agent_questions where seq=$qseq and status='answered'" 60 +sleep 8 +snap ratified +record 'ratification question' "$(journal "select status || ' → ' || resolution from agent_questions where seq=$qseq")" + +assert_journal_is "select status from charters where id='$charter'" 'active' \ + 'ratifying with one answer made the charter active' +assert_journal "select count(*) from commands where seq > $since and kind='charter_ratify' and status='applied'" \ + 'the ratification is journaled as an applied command' +record 'next due' "$(journal "select next_due from charters where id='$charter'")" + +dismiss_standing_watch + +# Stand-down is the other half of the journey, and it is said, not navigated. +down_at="$(mark)" +say "stand down the stretch reminder" +sleep 25 +snap stood-down + +status="$(journal "select status from charters where id='$charter'")" +record 'status after stand-down' "$status" +record 'commands after stand-down' "$(journal "select group_concat(kind || '/' || status, ', ') from commands where seq > $down_at")" +record 'reply to stand-down' "$(journal "select group_concat(substr(body,1,200),' | ') from messages where seq > $down_at and role in ('agent','system')")" + +if [ "$status" = "retired" ] || [ "$status" = "paused" ]; then + _check yes 'saying "stand down" retired the charter in the journal' \ + "charters.status in (retired,paused)" "$status" +else + _check no 'saying "stand down" retired the charter in the journal' \ + "charters.status in (retired,paused)" "$status — the thread may claim it, but the journal does not" +fi + +dump_turn "$since" +finish diff --git a/test/ux/journeys/09-consent.sh b/test/ux/journeys/09-consent.sh new file mode 100755 index 0000000000..73eaa34d82 --- /dev/null +++ b/test/ux/journeys/09-consent.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +# J9 · Approve spend (cheap variant) +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J9 Approve spend — price shown before purchase, consent asked, refusal honored' + +expect_nodes 1 + +# The real journey needs a plan estimated over $3, which is exactly the money a +# test suite may not spend. CODEAF_PLAN_CONSENT is the same gate with a movable +# line, so the cheap variant drops the line to a cent and asks the identical +# question about a job costing cents. Same code path, 1/300th of the money. +note 'variant: CODEAF_PLAN_CONSENT lowered to $0.0002 — below the estimate of a cents-scale job, so the same gate fires' + +ux_kill "$UX_SESSION" +sleep 2 +# A cent is still above what these jobs are estimated to cost, so the gate +# never fires at $0.01 — the line has to go below the estimate, not below the +# real $3 rail. +ux_launch "$UX_SESSION" "CODEAF_PLAN_CONSENT=0.0002" + +since="$(mark)" +say "research the three most common causes of bridge collapses and write me a short report file" + +assert_journal "select count(*) from agent_questions where seq > $since and category='plan-consent'" \ + 'the consent question was asked before the work was bought' 150 +snap asked + +qseq="$(journal "select seq from agent_questions where seq > $since and category='plan-consent' order by seq limit 1")" +record 'consent question' "$(journal "select replace(substr(text,1,300),char(10),' / ') from agent_questions where seq=$qseq")" + +if pane | grep -Eqi '\$[0-9]'; then + _check yes 'the price is shown before the purchase' 'a dollar figure on screen' "$(pane | grep -Eo '\$[0-9][0-9.]*' | head -3 | tr '\n' ' ')" +else + _check no 'the price is shown before the purchase' 'a dollar figure on screen' 'no dollar figure in the ask' +fi + +held_before="$(journal "select count(*) from nodes where created_seq > $since and status in ('running','done')")" +record 'nodes running or done while consent is outstanding' "$held_before" + +# Refuse it. +answer_number 2 +sleep 12 +snap refused + +assert_journal_is "select status from agent_questions where seq=$qseq" 'answered' 'the refusal was recorded' +record 'resolution' "$(journal "select resolution from agent_questions where seq=$qseq")" + +assert_journal "select count(*) from nodes where created_seq > $since and held=1" \ + 'refusal is honored — the plan is held, not run' 30 +assert_journal_is "select count(*) from nodes where created_seq > $since and status='done'" 0 \ + 'nothing was executed after the refusal' +record 'hold reason' "$(journal "select group_concat(distinct error) from nodes where created_seq > $since")" + +# Put the suite back on the ordinary rail for the journeys that follow. +ux_kill "$UX_SESSION"; sleep 2; ux_launch "$UX_SESSION" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/10-money.sh b/test/ux/journeys/10-money.sh new file mode 100755 index 0000000000..08daeeac89 --- /dev/null +++ b/test/ux/journeys/10-money.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# J10 · Ask about money +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J10 Ask about money — per-job and windowed spend readable in the thread, and always-on on the foot' + +expect_nodes 0 + +since="$(mark)" +actual="$(spend)" +record 'journal spend so far' "\$$actual" + +say "what have you spent so far today?" +assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 120 +sleep 6 +snap answered + +reply="$(journal "select group_concat(body,' ') from messages where seq > $since and role in ('agent','system')")" +record 'reply' "$(printf '%s' "$reply" | head -c 400)" + +if printf '%s' "$reply" | grep -Eq '\$[0-9]|[0-9]+ ?¢|cent'; then + _check yes 'the reply carries a real money figure' 'a dollar or cent amount' "$(printf '%s' "$reply" | grep -Eo '\$[0-9.]+|[0-9]+ ?¢' | head -3 | tr '\n' ' ')" +else + _check no 'the reply carries a real money figure' 'a dollar or cent amount' "$(printf '%s' "$reply" | head -c 200)" +fi + +# The always-on half of the same fact — the money you never had to ask for. +# +# v3 carries the bill on the frame's foot row, beside the cache and context +# readings ("$0.27 · ⟲ saved $0.0038 · 58% cached … 66.8k/1.3M · 5% …"), and on +# the status deck's first row beside the session name. The foot is the row the +# frame always draws, so this reads the foot — the same fact, at the address +# the frame gives it. +foot="$(pane | tail -3 | grep -E '\$' | head -1)" +record 'foot row' "$(printf '%s' "$foot" | sed 's/^ *//')" +if printf '%s' "$foot" | grep -Eq '\$[0-9]'; then + _check yes 'the foot carries spend without being asked' \ + 'a dollar figure on the foot row' "$(printf '%s' "$foot" | grep -Eo '\$[0-9.]+[KM]?' | head -1)" +else + _check no 'the foot carries spend without being asked' \ + 'a dollar figure on the foot row' "${foot:-no foot row with a dollar figure}" +fi + +dump_turn "$since" +finish diff --git a/test/ux/journeys/11-what-did-you-learn.sh b/test/ux/journeys/11-what-did-you-learn.sh new file mode 100755 index 0000000000..2e641d6f25 --- /dev/null +++ b/test/ux/journeys/11-what-did-you-learn.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# J11 · Ask what it learned +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J11 Ask what it learned — a read over the notebook: lessons, beliefs, retractions' + +expect_nodes 0 +since="$(mark)" +record 'notebook holds' "$(journal "select count(*) from facts where status='active'") active facts" + +say "what have you learned about how I like work done?" +assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 150 +sleep 8 +snap answered + +reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" +record 'reply' "$(printf '%s' "$reply" | head -c 600)" + +# The proof it read the notebook rather than improvising is that J7's lesson — +# run what you build before calling it done — comes back in substance. +if printf '%s' "$reply" | grep -Eq 'run |verif|before .*done|test'; then + _check yes "the answer draws on J7's lesson in substance" \ + 'the run-it-before-done lesson, in its own words' "$(printf '%s' "$reply" | head -c 220)" +else + _check no "the answer draws on J7's lesson in substance" \ + 'the run-it-before-done lesson, in its own words' "$(printf '%s' "$reply" | head -c 220)" +fi + +if printf '%s' "$reply" | grep -Eq "i haven'?t learned|nothing yet|no lessons|i don'?t have|cannot recall"; then + _check no 'it does not refuse a question about itself' 'a substantive answer' "$(printf '%s' "$reply" | head -c 200)" +else + _check yes 'it does not refuse a question about itself' 'a substantive answer' 'no refusal' +fi + +length="$(journal "select coalesce(max(length(body)),0) from messages where seq > $since and role in ('agent','system')")" +record 'longest reply (bytes)' "$length" +[ "${length:-0}" -gt 60 ] \ + && _check yes 'the answer has substance' 'more than a shrug' "$length bytes" \ + || _check no 'the answer has substance' 'more than a shrug' "$length bytes" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/12-leave-return.sh b/test/ux/journeys/12-leave-return.sh new file mode 100755 index 0000000000..debfe6571e --- /dev/null +++ b/test/ux/journeys/12-leave-return.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# J12 · Leave and return +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J12 Leave and return — close the terminal, come back: the same thread resumes and a brief covers what landed' + +expect_nodes 0 + +before_messages="$(journal "select count(*) from messages")" +record 'messages before leaving' "$before_messages" +snap before-leaving + +# Leaving means the window dies. The store does not. +ux_kill "$UX_SESSION" +sleep 5 + +assert_cmd 'the journal survived the terminal closing' test -f "$UX_DB" +since="$(mark)" + +ux_launch "$UX_SESSION" +sleep 6 +snap returned + +# alive_re: v3 signs its composer prompt (see lib.sh). +assert_screen "$(alive_re)" 'codeaf came back up' 60 + +# The one thing that must be true on return: the conversation is still there. +assert_screen 'heron|haiku|river|stretch|water|learned' \ + 'the prior conversation is on screen — the thread resumed, it did not restart' 45 + +assert_journal_is "select count(*) from messages where seq <= $since" "$before_messages" \ + 'no prior message was dropped by the restart' + +session_now="$(journal "select session_id from messages where session_id != '' order by seq desc limit 1")" +sessions="$(journal "select count(distinct session_id) from messages where session_id != ''")" +record 'session id in use' "$session_now" +record 'distinct sessions in the journal' "$sessions" + +# Resumption is either the same session id continuing, or a new one attached +# with a brief. Both are the journey; the report says which happened. +say "are you still there?" +sleep 10 +after_session="$(journal "select session_id from messages where role='user' order by seq desc limit 1")" +record 'session after returning' "$after_session" + +if [ "$after_session" = "$session_now" ]; then + _check yes 'the same session resumed' 'session id unchanged across the restart' "$after_session" +else + brief="$(journal "select count(*) from messages where seq > $since and body like 'While you were away%'")" + if [ "${brief:-0}" != "0" ]; then + _check yes 'a new session attached with an arrival brief' 'a "While you were away" line' "$brief brief(s)" + else + _check no 'the return either resumes the session or brings a brief' \ + 'same session id, or a "While you were away" brief' "new session $after_session with no brief" + fi +fi + +assert_journal "select count(*) from messages where seq > $since and role='agent'" \ + 'the returned window answers — the brain came back, not just the frame' 120 +dump_turn "$since" +finish diff --git a/test/ux/journeys/13-second-window.sh b/test/ux/journeys/13-second-window.sh new file mode 100755 index 0000000000..8961d18763 --- /dev/null +++ b/test/ux/journeys/13-second-window.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# J13 · One brain +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J13 One brain — a second window on a conversation this machine already has open is told where the conversation lives, and the first window never blinks' + +expect_nodes 0 + +# One brain means one composer. The watcher machinery (watching.go) exists for +# conversations on FAR machines; a conversation open in a second window on this +# one is refused at the journal with the session-busy word. That refusal is the +# product keeping its law — two windows must not race one composer — and this +# journey holds it open. +snap a-before +assert_screen "$(alive_re)" 'window A is alive' 30 + +# B opens onto the same brain. It must not become a second front door. +ux_launch "$UX_SESSION_B" +snap b-opened "$UX_SESSION_B" + +if wait_screen 'open in another window' 30 "$UX_SESSION_B"; then + _check yes 'the second window is refused with the session-busy word' \ + '"open in another window — go there, or start a new conversation here"' \ + "$(pane "$UX_SESSION_B" | grep -i 'open in another window' | head -1 | sed 's/^ *//')" +else + _check no 'the second window is refused with the session-busy word' \ + '"open in another window — go there, or start a new conversation here"' \ + "$(pane "$UX_SESSION_B" | sed -n '1p;$p' | tr '\n' ' ')" +fi + +# And A never blinked: still drawing its own frame. +if pane "$UX_SESSION" | grep -q 'CODEAF-EXITED'; then + _check no 'the first window survived the second opening' 'A still running' 'A exited' +else + _check yes 'the first window survived the second opening' 'A still running' 'A alive' +fi + +# The one brain answers from the window that owns it. +since="$(mark)" +say "what is 7 times 6?" "$UX_SESSION" +if wait_screen '42' 120 "$UX_SESSION"; then + _check yes 'the first window still answers — one brain, undisturbed' '42 on A' 'found' +else + _check no 'the first window still answers — one brain, undisturbed' '42 on A' \ + "$(pane "$UX_SESSION" | tail -12 | tr '\n' ' ')" +fi +snap a-answered "$UX_SESSION" + +ux_kill "$UX_SESSION_B" +sleep 3 + +dump_turn "$since" +finish diff --git a/test/ux/journeys/14-craft.sh b/test/ux/journeys/14-craft.sh new file mode 100755 index 0000000000..44ddee89fb --- /dev/null +++ b/test/ux/journeys/14-craft.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# J14 · Repeat a workflow — EXPLORATORY +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J14 Repeat a workflow — craft learned from the first run compiles into the second' + +expect_nodes 2 + +# This one is reported, not graded. Craft is forged by the distiller only when +# the model judges a job SHAPE reusable, and the prompt that asks it says +# outright that "most jobs have none" (cmd/codeaf/chat.go, the distiller's +# craft clause). Two toy jobs are exactly the case it is told to decline. The +# thresholds on the other side — recognition — are constants: +# CraftDecisiveScore = 1.5 × craft.MatchFloor (use a known-good craft) +# CraftOverwhelmingScore = 4 × craft.MatchFloor (use an unproven draft) +# in internal/resident/craftmind.go. So: run the two jobs, look, say what was +# actually there. +observed 'craft forging is a model judgment with an explicit "most jobs have none" instruction; two toy jobs are reported, not graded' + +since="$(mark)" +craft_dir="$UX_STATE/craft" + +before_commits="$(git -C "$craft_dir" rev-list --count HEAD 2>/dev/null || echo 0)" +record 'craft repo commits before' "$before_commits" + +say "make a file listing 3 prime numbers" +wait_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" 240 +first_done="$(mark)" +sleep 20 +snap first-job + +say "make a file listing 3 square numbers" +wait_journal "select count(*) from nodes where created_seq > $first_done and origin='user' and status='done'" 240 +sleep 30 +snap second-job + +after_commits="$(git -C "$craft_dir" rev-list --count HEAD 2>/dev/null || echo 0)" +record 'craft repo commits after' "$after_commits" +record 'craft repo log' "$(git -C "$craft_dir" log --oneline -10 2>/dev/null | tr '\n' ' ' || echo '')" +record 'craft files' "$(ls "$craft_dir" 2>/dev/null | tr '\n' ' ' || echo '')" +record 'nodes carrying a craft' "$(journal "select count(*) from nodes where craft != ''")" +record 'craft named on nodes' "$(journal "select group_concat(distinct craft) from nodes where craft != ''")" + +first_cost="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since and seq <= $first_done")" +second_cost="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $first_done")" +first_nodes="$(journal "select count(*) from nodes where created_seq > $since and created_seq <= $first_done")" +second_nodes="$(journal "select count(*) from nodes where created_seq > $first_done")" +record 'first job' "$first_nodes nodes, \$$first_cost" +record 'second job' "$second_nodes nodes, \$$second_cost" + +if [ "$after_commits" -gt "$before_commits" ]; then + record 'OBSERVED' "craft WAS forged: $((after_commits - before_commits)) new commit(s) in the craft repo" +else + record 'OBSERVED' 'no craft was forged from two toy jobs — consistent with the distiller being told most jobs have none' +fi + +dump_turn "$since" +finish diff --git a/test/ux/journeys/15-fanout.sh b/test/ux/journeys/15-fanout.sh new file mode 100755 index 0000000000..17a465ce28 --- /dev/null +++ b/test/ux/journeys/15-fanout.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +# J15-fanout · Parallelism lives in the graph, not in terminal tabs +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Fan-out — "say five things and five jobs run": one message asking for three independent things must become three jobs, not one queue' + +# docs/JOURNEY.md: "Parallelism lives in the graph, not in terminal tabs: say +# five things and five jobs run; the board shows them side by side." Three is +# enough to tell splitting from serialising, and cheap enough to say twice. +expect_nodes 3 + +since="$(mark)" +started="$(date +%s)" + +say "three separate things please: write a file with the first 5 prime numbers, write a file with the capitals of France Japan and Peru, and write a file with a two-line poem about rain" + +assert_journal "select count(*) from commands where seq > $since and kind='splice' and status='applied'" \ + 'the ask was journaled as work' 90 + +# The fan-out: three independent jobs, not one node doing three things. +wait_journal "select case when count(*) >= 3 then 1 else 0 end from nodes where created_seq > $since and origin='user'" 240 +jobs="$(journal "select count(*) from nodes where created_seq > $since and origin='user'")" +record 'jobs formed from one sentence' "$jobs" +record 'job titles' "$(journal "select group_concat(title, ' · ') from nodes where created_seq > $since and origin='user'")" + +if [ "${jobs:-0}" -ge 3 ]; then + _check yes 'one sentence naming three things became three jobs' 'three or more user jobs' "$jobs" +else + _check no 'one sentence naming three things became three jobs' 'three or more user jobs' \ + "$jobs — the fan-out did not split the ask" +fi + +wait_journal "select case when count(*) = 0 then 1 else 0 end from nodes where created_seq > $since and origin='user' and status not in ('done','failed','cancelled')" "$UX_JOB_TIMEOUT" +wall=$(( $(date +%s) - started )) +sleep 10 +snap delivered + +assert_journal_is "select count(*) from nodes where created_seq > $since and origin='user' and status != 'done'" 0 \ + 'every one of the jobs finished' + +# Concurrency factor: the work the graph did, divided by the time the person +# waited. At 1.0 the jobs were a queue with extra steps; above it, the graph is +# the parallelism the product promises instead of terminal tabs. +busy="$(journal " + select cast(coalesce(sum( + (julianday(finished_at) - julianday(started_at)) * 86400.0), 0) as int) + from nodes where created_seq > $since and origin='user' + and started_at is not null and finished_at is not null")" +record 'wall time the person waited' "${wall}s" +record 'summed job time the graph did' "${busy}s" +factor="$(python3 -c " +wall = max(1.0, float('${wall:-1}')) +print('%.2f' % (float('${busy:-0}') / wall))")" +record 'CONCURRENCY FACTOR (summed job seconds / wall seconds)' "$factor" +printf '%s\n' "$factor" > "$UX_DIR/concurrency" + +if python3 -c "import sys; sys.exit(0 if float('$factor') >= 1.3 else 1)"; then + _check yes 'the jobs actually ran side by side' 'concurrency factor at or above 1.3' "$factor" +else + _check no 'the jobs actually ran side by side' 'concurrency factor at or above 1.3' \ + "$factor — the graph held three jobs but ran them one after another" +fi + +# Each ask must be answered on its own terms, not merged into one blob. +answers="$(journal "select group_concat(lower(body), char(10)) from messages where seq > $since and role in ('agent','system')")" +# ERE, not BRE: grep -E reads \| as a literal pipe, and the earlier spelling +# searched for the six-character string "lima|peru" and never found it. +for want in 'prime' 'lima|peru|tokyo|paris' 'rain'; do + if printf '%s' "$answers" | grep -Eqi "$want"; then + _check yes "the thread answers the '$want' part of the ask" "a deliverable mentioning $want" 'found' + else + _check no "the thread answers the '$want' part of the ask" "a deliverable mentioning $want" 'absent' + fi +done + +judge_this 'three separate things: a file with the first 5 primes, a file with the capitals of France Japan and Peru, and a file with a two-line poem about rain' \ + "$(journal "select group_concat(body, char(10)) from messages where seq > $since and node_id != ''")" + +dump_turn "$since" +finish diff --git a/test/ux/journeys/16-model-words.sh b/test/ux/journeys/16-model-words.sh new file mode 100755 index 0000000000..8529e44bde --- /dev/null +++ b/test/ux/journeys/16-model-words.sh @@ -0,0 +1,57 @@ +#!/usr/bin/env bash +# J16 · Choose models in words +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J16 Choose models in words — "boost this" routes the call and is recorded on usage' + +work_model="$(python3 -c "import json;print(json.load(open('$UX_STATE/settings.json')).get('task_model',''))")" +boost_model="$(python3 -c "import json;print(json.load(open('$UX_STATE/settings.json')).get('boost_model',''))")" +record 'work model' "$work_model" +record 'boost model' "$boost_model" + +expect_nodes 1 +since="$(mark)" +say "use the boost model for this: write one sentence about mountains" + +assert_screen 'mountain' 'the sentence came back' 150 +sleep 8 +snap answered + +models="$(journal "select group_concat(distinct model) from usage where seq > $since")" +record 'models on usage rows for this turn' "${models:-}" + +boost_bare="${boost_model#\~}" +if [ -n "$models" ] && printf '%s' "$models" | grep -qF "$boost_bare"; then + _check yes 'the words routed the call — usage names the boost model' \ + "a usage row on $boost_model" "$models" + record 'outcome' 'ROUTED ON USAGE — the boost model appears on the turn' +elif [ -n "$models" ] && [ "$models" != "$work_model" ] && [ "$models" != "${work_model#\~}" ]; then + _check yes 'the words routed the call — usage differs from the default work model' \ + "a usage model other than $work_model" "$models" + record 'outcome' 'ROUTED ON USAGE — a different model than the default' +else + cmd="$(journal "select group_concat(kind || ':' || substr(instruction,1,60), ' | ') from commands where seq > $since")" + record 'commands this turn' "${cmd:-}" + if printf '%s' "$cmd" | grep -qi 'boost'; then + _check yes 'the words are carried on the command even though usage kept the default model' \ + 'a command carrying boost' "$cmd" + record 'outcome' 'CARRIED ON COMMAND — the boost word is journaled but usage stayed on the work model' + else + _check no 'the model words reached either the usage row or the command' \ + "$boost_model on usage, or boost on a command" "usage=[${models:-none}] commands=[${cmd:-none}]" + record 'outcome' 'NOT ROUTED — the words changed nothing durable' + fi +fi + +# The glance that says which model is answering. v3 carries the model word on +# the status deck's chip (drawn from app.modelWord) and in the foot's seam. +# Reported, not graded, as before — the graded half of this journey is the +# durable routing above. +record 'foot row' "$(pane | tail -2 | tr '\n' ' ' | sed 's/ *$//')" +if [ -n "$boost_bare" ]; then + record 'model word on screen' "$(pane | grep -iF -- "$boost_bare" | head -1 | sed 's/^ *//')" +else + record 'model word on screen' '' +fi + +dump_turn "$since" +finish diff --git a/test/ux/journeys/17-attachments.sh b/test/ux/journeys/17-attachments.sh new file mode 100755 index 0000000000..089156bd78 --- /dev/null +++ b/test/ux/journeys/17-attachments.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# J17 · Attach things (cheap variant) +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J17 Attach things — a mentioned file is copied at mention (CAS) and survives the source changing' + +# The composer captures path-shaped tokens with known media extensions and +# turns them into chips, so the cheap variant is a real (tiny) PNG. +img="$UX_RUN/dot.png" +python3 - "$img" <<'PY' +import base64, sys +# 1x1 PNG, the smallest honest image there is. +sys.stdout.buffer # noqa +open(sys.argv[1], "wb").write(base64.b64decode( + "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==")) +PY +record 'file mentioned' "$img ($(wc -c < "$img" | tr -d ' ') bytes)" + +before_cas="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" +record 'CAS objects before' "$before_cas" + +expect_nodes 1 +since="$(mark)" +tmux send-keys -t "$UX_SESSION" -l -- "tell me in one word what colour this image is: $img" +sleep 2 +snap chip +press Enter +note "said: tell me in one word what colour this image is: $img" + +sleep 25 +snap answered + +# "Not the empty list" is not the same as "an attachment". A surface that never +# captured the mention writes the JSON literal `null` into the column, which is +# also != '[]' — and the check passed on nothing. The column has to hold a +# populated list for this to mean what it says. v1's value is unchanged by the +# tightening; it writes ["cas://…"]. +assert_journal "select count(*) from messages where seq > $since and role='user' + and attachments is not null and attachments not in ('[]','null','')" \ + 'the mention is journaled as an attachment on the user message' 60 +record 'attachments recorded' "$(journal "select attachments from messages where seq > $since and role='user' order by seq limit 1")" + +after_cas="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" +record 'CAS objects after' "$after_cas" +if [ "$after_cas" -gt "$before_cas" ]; then + _check yes 'the file was copied into the CAS at mention time' \ + 'a new content-addressed object' "$((after_cas - before_cas)) new object(s)" +else + _check no 'the file was copied into the CAS at mention time' \ + 'a new content-addressed object' "still $after_cas objects" +fi + +# Survives the source changing: the copy is content, not a pointer. +rm -f "$img" +still="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" +[ "$still" = "$after_cas" ] \ + && _check yes 'the copy outlives its source' 'CAS unchanged after deleting the original' "$still objects" \ + || _check no 'the copy outlives its source' 'CAS unchanged after deleting the original' "$still objects" + +assert_journal "select count(*) from messages where seq > $since and role in ('agent','system')" \ + 'the mention got a reply, not silence' + +dump_turn "$since" +finish diff --git a/test/ux/journeys/18-take-out.sh b/test/ux/journeys/18-take-out.sh new file mode 100755 index 0000000000..0ffc652432 --- /dev/null +++ b/test/ux/journeys/18-take-out.sh @@ -0,0 +1,52 @@ +#!/usr/bin/env bash +# J18 · Take things out (hint-only variant) +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J18 Take things out — y/Y copy, /open, clickable paths: deliverables exit the terminal without friction' + +expect_nodes 0 + +# The clipboard itself is the machine's, not codeaf's — pbcopy on a headless CI +# box is a different journey. What is codeaf's, and what this asserts, is that +# the way out is discoverable and that the paths are real. +note 'variant: the clipboard is environment-dependent, so this asserts the affordance renders and the paths are real files' + +# The door to the help sheet: v3 answers a bare `?` from the composer +# (commands.go binds it as helpAskKey), the same door the old surfaces had. +press '?' +sleep 3 +snap help + +assert_screen 'help' 'the help overlay opened on ?' 20 +# Discoverability: the copy verb has to be readable somewhere the user can +# find it. The registry's row reads "copy answer / copy the focused answer to +# the clipboard y" (registry/catalog.go). +assert_screen 'copy .*answer' 'the y/Y copy hint is spelled out in help' 20 +record 'copy hint' "$(pane | grep -Ei 'copy .*answer' | head -1 | sed 's/^ *//')" +assert_screen '/open' 'the /open route out is named' 20 + +press Escape +sleep 2 +snap help-closed + +# The other half of "without friction": the paths the thread printed are real. +paths="$(journal "select body from messages where role in ('agent','system')" | grep -Eo "$UX_STATE/workspace/[^ )\"']*" | sort -u | head -5)" +record 'paths named in the thread' "$(printf '%s' "$paths" | tr '\n' ' ')" +missing=0 +while IFS= read -r p; do + [ -z "$p" ] && continue + [ -e "$p" ] || { missing=$((missing + 1)); note "MISSING: $p"; } +done <<< "$paths" + +if [ -n "$paths" ]; then + [ "$missing" -eq 0 ] \ + && _check yes 'every workspace path the thread printed is a real file' 'all paths exist' 'all exist' \ + || _check no 'every workspace path the thread printed is a real file' 'all paths exist' "$missing missing" +else + # Not a failure. An answer-first deliverable is allowed to hand back the + # content and no path at all — /open and the workspace link are the way out + # in that case, and grading their absence would punish the better behaviour. + record 'OBSERVED' 'this run delivered answer-first with no absolute path in the thread; the way out is /open and the workspace link, not a printed path' +fi + +dump_turn "$(mark)" +finish diff --git a/test/ux/journeys/19-self-pitch.sh b/test/ux/journeys/19-self-pitch.sh new file mode 100755 index 0000000000..953b820789 --- /dev/null +++ b/test/ux/journeys/19-self-pitch.sh @@ -0,0 +1,39 @@ +#!/usr/bin/env bash +# J19 · Ask what it can do +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'J19 Ask what it can do — the product carries its own pitch: journeys, in its own voice, from live capabilities' + +expect_nodes 0 +since="$(mark)" +say "what can you do?" +assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 150 +sleep 8 +snap answered + +reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" +record 'reply' "$(printf '%s' "$reply" | head -c 900)" + +length="$(printf '%s' "$reply" | wc -c | tr -d ' ')" +record 'reply length (bytes)' "$length" +[ "${length:-0}" -gt 150 ] \ + && _check yes 'the pitch has substance' 'more than a sentence' "$length bytes" \ + || _check no 'the pitch has substance' 'more than a sentence' "$length bytes" + +# It must describe the journeys, not itself in the abstract. +hits=0 +for word in task work file build remember learn charter remind cost watch run graph question; do + printf '%s' "$reply" | grep -q "$word" && hits=$((hits + 1)) +done +record 'capability words named' "$hits of 13" +[ "$hits" -ge 4 ] \ + && _check yes 'the answer names actual capabilities, not vibes' 'at least 4 capability words' "$hits" \ + || _check no 'the answer names actual capabilities, not vibes' 'at least 4 capability words' "$hits" + +if printf '%s' "$reply" | grep -Eq "i'?m an ai|language model|as an ai|i can'?t do|i have no"; then + _check no 'it pitches the product, not a disclaimer' 'no assistant boilerplate' "$(printf '%s' "$reply" | grep -Eo "i'?m an ai|language model|as an ai" | head -1)" +else + _check yes 'it pitches the product, not a disclaimer' 'no assistant boilerplate' 'none' +fi + +dump_turn "$since" +finish diff --git a/test/ux/judge.py b/test/ux/judge.py new file mode 100644 index 0000000000..607eaf1a50 --- /dev/null +++ b/test/ux/judge.py @@ -0,0 +1,104 @@ +#!/usr/bin/env python3 +"""judge.py — one model call that grades one deliverable. + +A journey can prove that a haiku was journaled, spliced, and shown. It cannot +prove the haiku was any good, and "it ran" is not the product's promise. So the +deliverables get read by a cheap model against a three-line rubric, and the +score rides in the report beside the pass. + + judge.py + +prints: | +""" + +import json +import os +import sys +import urllib.error +import urllib.request + +RUBRIC = """You are grading one deliverable from an AI agent. Be a hard marker. + +Grade on exactly three things: +1. Does the deliverable satisfy the LITERAL ask — every constraint the user + actually wrote (counts, subjects, named requirements), not the gist? +2. Is it answer-first: does the final message contain the CONTENT itself, + rather than only a pointer to a file the user would have to go and open? +3. Is it competently done for what it is? + +Reply with JSON only: {"score": <1-5 integer>, "why": ""} +5 = satisfies every literal constraint and leads with the content. +3 = broadly right but misses a stated constraint or buries the content. +1 = does not do what was asked, or hands back only a path.""" + + +def main() -> int: + state, ask_path, deliverable_path = sys.argv[1], sys.argv[2], sys.argv[3] + with open(os.path.join(state, "config.json")) as handle: + key = json.load(handle)["api_key"] + # The "~" prefix is OpenRouter's floating-alias marker and part of the slug, + # not decoration: stripping it turns a valid model id into a 400. The + # fallbacks exist because the judge failing must never fail the journey. + try: + configured = json.load(open(os.path.join(state, "settings.json"))).get("chat_model") + except (FileNotFoundError, ValueError): + configured = None + candidates = [m for m in (configured, "deepseek/deepseek-v4-flash", + "google/gemini-3.6-flash") if m] + + ask = open(ask_path).read().strip()[:2000] + deliverable = open(deliverable_path).read().strip()[:6000] + if not deliverable: + print("1|nothing was delivered to grade") + return 0 + + text, failure = None, "no model answered" + for model in candidates: + payload = { + "model": model, + "messages": [ + {"role": "system", "content": RUBRIC}, + {"role": "user", "content": "THE ASK:\n%s\n\nWHAT CAME BACK:\n%s" % (ask, deliverable)}, + ], + "temperature": 0, + # Generous, because a reasoning-capable model spends its first + # tokens thinking and returns a null content when the cap lands + # mid-thought — which reads as "the judge said nothing" rather than + # as the truncation it is. + "max_tokens": 900, + } + request = urllib.request.Request( + "https://openrouter.ai/api/v1/chat/completions", + data=json.dumps(payload).encode(), + headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"}, + ) + try: + with urllib.request.urlopen(request, timeout=90) as response: + body = json.load(response) + message = body["choices"][0]["message"] + text = message.get("content") or message.get("reasoning") + if text: + break + failure = "%s returned no content (%s)" % ( + model, body["choices"][0].get("finish_reason")) + except (urllib.error.URLError, KeyError, ValueError, TypeError) as exc: + failure = "%s on %s" % (exc, model) + if text is None: + print("?|the judge call failed: %s" % failure) + return 0 + + start, end = text.find("{"), text.rfind("}") + if start >= 0 and end > start: + try: + verdict = json.loads(text[start:end + 1]) + why = str(verdict.get("why", "")).replace("|", "/").replace("\n", " ") + print("%s|%s" % (verdict.get("score", "?"), why[:200])) + return 0 + except ValueError: + pass + print("?|the judge did not answer in the rubric's shape: %s" % text[:120].replace("\n", " ")) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/test/ux/lib.sh b/test/ux/lib.sh new file mode 100644 index 0000000000..125807337c --- /dev/null +++ b/test/ux/lib.sh @@ -0,0 +1,336 @@ +#!/usr/bin/env bash +# lib.sh — the vocabulary a journey script is written in. +# +# A journey is a person at a terminal. It says things, waits for the screen to +# say something back, and then asks the journal whether anything durable +# actually happened. Those three verbs — say, wait_screen, journal — are the +# whole library; everything else is bookkeeping so the report can print WHAT +# was expected beside WHAT was there. +# +# Sourced by every journeys/*.sh under an environment run.sh exports: +# UX_BIN UX_HOME UX_STATE UX_DB UX_SESSION UX_EVIDENCE UX_CAP UX_JOURNEY + +set -uo pipefail + +: "${UX_REPLY_TIMEOUT:=120}" # a conversational reply +: "${UX_JOB_TIMEOUT:=300}" # a commissioned job +: "${UX_POLL:=2}" + +# ---------------------------------------------------------------- the frame +# +# alive_re matches "the surface drew a frame of its own". v3 always draws the +# composer prompt "›" — it is on screen before anything has been said, and it +# is the one glyph a dead pane (the CODEAF-EXITED marker, a bare shell) never +# draws. Match on it whenever a journey asks "is this window alive at all". +alive_re() { printf '%s' '›'; } + +UX_FAILURES=0 +UX_CHECKS=0 +UX_FORCED="" +UX_DIR="$UX_EVIDENCE/$UX_JOURNEY" +mkdir -p "$UX_DIR" +: > "$UX_DIR/notes.md" + +# ---------------------------------------------------------------- narration + +note() { printf '%s\n' "$*" >> "$UX_DIR/notes.md"; printf ' %s\n' "$*" >&2; } +head_line() { printf '\n## %s\n\n' "$*" >> "$UX_DIR/notes.md"; printf ' %s\n' "$*" >&2; } + +# journey_is prints the journey's own one-line description, so a reader of the +# transcript never has to go and look up what number 4 was. +journey_is() { + printf '# %s — %s\n\n' "$UX_JOURNEY" "$*" >> "$UX_DIR/notes.md" + printf '\n\033[1m▶ %s — %s\033[0m\n' "$UX_JOURNEY" "$*" >&2 +} + +record() { printf -- '- **%s**: %s\n' "$1" "$2" >> "$UX_DIR/notes.md"; printf -- ' · %s: %s\n' "$1" "$2" >&2; } + +# ---------------------------------------------------------------- the mouth + +# say types a sentence and presses Enter, with enough pacing that the composer +# has drawn the draft before the newline lands. Never start a sentence with +# "/" (the command palette eats Enter) or "?" (it opens the help overlay). + +# clear_composer empties the draft. A digit typed at a question that has since +# been answered stays in the composer and silently prefixes the next sentence +# ("2stand down the stretch reminder"), which would make the suite lie about +# what the user said. +# +# v3 binds ctrl+u as the readline kill — the input guard and the draft ring +# both test it by name — so the words go to the ring and the box is empty: +# a gesture with no second meaning while a reply streams. +clear_composer() { + local sess="${1:-$UX_SESSION}" + tmux send-keys -t "$sess" C-u + sleep 0.3 +} + +say() { + local text="$1" sess="${2:-$UX_SESSION}" + case "$text" in + /*) note "REFUSED to say a line starting with / — the palette would eat it"; return 1 ;; + \?*) note "REFUSED to say a line starting with ? — it opens help"; return 1 ;; + esac + clear_composer "$sess" + tmux send-keys -t "$sess" -l -- "$text" + sleep 0.6 + tmux send-keys -t "$sess" Enter + note "said: $text" +} + +# press sends a named key (Enter, Up, C-c …). +press() { tmux send-keys -t "${2:-$UX_SESSION}" "$1"; note "pressed: $1"; } + +# type_raw sends characters with no Enter. Answering an option question is a +# bare digit that submits itself, so it must NOT be followed by a newline. +type_raw() { tmux send-keys -t "${2:-$UX_SESSION}" -l -- "$1"; note "typed: $1"; } + +# answer_number answers a numbered question. +# +# v3 binds the digits a question draws (questionset.go's questionGroupDigit, +# which maps a digit key to the row it names): a bare digit picks its row and +# submits itself, so it must NOT be followed by Enter. Every journey asserts +# the OUTCOME — the question is recorded answered, by the user's own message — +# never the keystroke. +answer_number() { + local number="$1" sess="${2:-$UX_SESSION}" + type_raw "$number" "$sess" + sleep 3 +} + +# ---------------------------------------------------------------- the eyes + +pane() { tmux capture-pane -t "${1:-$UX_SESSION}" -p 2>/dev/null; } + +snap() { + local name="$1" sess="${2:-$UX_SESSION}" + pane "$sess" > "$UX_DIR/$name.snap.txt" + note "snap: $name.snap.txt" +} + +# wait_screen polls the pane until the regex matches. Match on SHORT fragments: +# every line is wrapped and hard-clamped to the pane width, so a long literal +# can straddle two rows and never appear whole. +wait_screen() { + local re="$1" limit="${2:-$UX_REPLY_TIMEOUT}" sess="${3:-$UX_SESSION}" waited=0 + while [ "$waited" -lt "$limit" ]; do + if pane "$sess" | grep -Eqi -- "$re"; then return 0; fi + sleep "$UX_POLL"; waited=$((waited + UX_POLL)) + done + return 1 +} + +# ---------------------------------------------------------------- the journal + +journal() { + sqlite3 -readonly "$UX_DB" "$1" 2>/dev/null +} + +journal_table() { sqlite3 -readonly -header -column "$UX_DB" "$1" 2>/dev/null; } + +# mark is the journal watermark before a turn. Every assertion afterwards is +# scoped to "since the mark", so journey 5 can never pass on journey 2's rows. +mark() { journal "select coalesce(max(seq),0) from events"; } + +spend() { journal "select coalesce(round(sum(cost),6),0) from usage"; } + +# wait_journal polls a SQL count until it is greater than zero. +wait_journal() { + local sql="$1" limit="${2:-$UX_JOB_TIMEOUT}" waited=0 value + while [ "$waited" -lt "$limit" ]; do + value="$(journal "$sql")" + if [ -n "$value" ] && [ "$value" != "0" ]; then return 0; fi + sleep "$UX_POLL"; waited=$((waited + UX_POLL)) + done + return 1 +} + +# ---------------------------------------------------------------- assertions + +_check() { + local ok="$1" what="$2" expected="$3" actual="$4" + UX_CHECKS=$((UX_CHECKS + 1)) + if [ "$ok" = "yes" ]; then + printf -- '- ✅ %s\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >> "$UX_DIR/notes.md" + printf ' \033[32m✅ %s\033[0m\n' "$what" >&2 + else + UX_FAILURES=$((UX_FAILURES + 1)) + printf -- '- ❌ %s\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >> "$UX_DIR/notes.md" + printf ' \033[31m❌ %s\033[0m\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >&2 + snap "fail-$UX_CHECKS" + fi +} + +# assert_screen waits for a pattern and reports the last screen when it never +# arrives, because "the screen never said it" is only useful beside the screen. +assert_screen() { + local re="$1" what="$2" limit="${3:-$UX_REPLY_TIMEOUT}" + if wait_screen "$re" "$limit"; then + _check yes "$what" "screen matching /$re/" "$(pane | grep -Eio -- "$re" | head -1)" + else + _check no "$what" "screen matching /$re/ within ${limit}s" "timed out; last screen in fail snap" + fi +} + +refute_screen() { + local re="$1" what="$2" + if pane | grep -Eqi -- "$re"; then + _check no "$what" "no screen line matching /$re/" "$(pane | grep -Eio -- "$re" | head -1)" + else + _check yes "$what" "no screen line matching /$re/" "none" + fi +} + +# assert_journal passes when the query returns something that is neither empty +# nor zero. The query IS the expectation, so it prints verbatim on failure. +assert_journal() { + local sql="$1" what="$2" limit="${3:-0}" value + if [ "$limit" -gt 0 ]; then wait_journal "$sql" "$limit" >/dev/null; fi + value="$(journal "$sql")" + if [ -n "$value" ] && [ "$value" != "0" ]; then + _check yes "$what" "$sql → non-zero" "$value" + else + _check no "$what" "$sql → non-zero" "${value:-}" + fi +} + +assert_journal_is() { + local sql="$1" want="$2" what="$3" value + value="$(journal "$sql")" + if [ "$value" = "$want" ]; then + _check yes "$what" "$sql = $want" "$value" + else + _check no "$what" "$sql = $want" "${value:-}" + fi +} + +assert_cmd() { + local what="$1"; shift + if "$@" >/dev/null 2>&1; then + _check yes "$what" "$* succeeds" "exit 0" + else + _check no "$what" "$* succeeds" "exit $?" + fi +} + +# ---------------------------------------------------------------- gauges +# +# Completion is the cheap half. A journey that passes every assertion can still +# have spliced eleven tasks to write a haiku, spent a dollar doing it, taken +# four minutes, filled the thread with chatter, and produced something bad. +# These four leave a trace the runner turns into the report card. + +# expect_nodes declares the proportionate size of this ask. The runner flags a +# journey that formed materially more work than the ask deserved. +expect_nodes() { printf '%s\n' "$1" > "$UX_DIR/expect-nodes"; } + +# judge_this hands one deliverable to a cheap model against a fixed rubric. +# The suite can prove a haiku was journaled; only a reader can say it is a +# haiku about rivers that mentions a heron. +judge_this() { + printf '%s\n' "$1" > "$UX_DIR/judge-ask" + printf '%s\n' "$2" > "$UX_DIR/judge-deliverable" + note "queued for the quality judge (${#2} bytes of deliverable)" +} + +# deliverable_since is the usual thing to judge: everything the agent said +# after the ask, which is where an answer-first product puts the content. +deliverable_since() { + journal "select group_concat(body, char(10)) from messages where seq > $1 and role in ('agent','system')" +} + +# ---------------------------------------------------------------- verdicts + +# observed marks a journey whose mechanism has a threshold we cannot force +# cheaply. It is not a pass and not a failure: it is evidence, reported. +observed() { UX_FORCED="OBSERVED"; note "OBSERVED: $*"; printf '%s\n' "$*" > "$UX_DIR/verdict-note"; } +skipped() { UX_FORCED="SKIPPED"; note "SKIPPED: $*"; printf '%s\n' "$*" > "$UX_DIR/verdict-note"; exit 0; } + +# ---------------------------------------------------------------- windows + +# ux_launch opens another window onto the same brain, from the same UX_ENV the +# runner built, so a relaunch is a relaunch and not a different program. +ux_launch() { + local sess="$1"; shift + local extra="${*:-}" + tmux kill-session -t "$sess" 2>/dev/null + tmux new-session -d -s "$sess" -x "$UX_WIDTH" -y "$UX_HEIGHT" \ + "env $UX_ENV $extra '$UX_BIN' chat; echo CODEAF-EXITED; sleep 900" + sleep 8 + note "launched window $sess" +} + +ux_kill() { tmux kill-session -t "$1" 2>/dev/null; note "killed window $1"; } + +# ---------------------------------------------------------------- hygiene + +# dismiss_standing_watch answers the "should I keep watching when you're not +# here" offer with NO. Saying yes installs a launchd agent on the real machine, +# and a test suite does not get to leave a daemon behind. +dismiss_standing_watch() { + local open + local waited=0 + while [ "$waited" -lt 20 ]; do + open="$(journal "select count(*) from agent_questions where status in ('pending','asked') and lower(text) like '%keep watching%'")" + [ "${open:-0}" != "0" ] && break + sleep 2; waited=$((waited + 2)) + done + if [ "${open:-0}" != "0" ]; then + note "standing-watch offer is open — declining (a suite must not install a daemon)" + answer_number 2 + sleep 5 + else + note "no standing-watch offer open; nothing to decline" + fi + clear_composer +} + +# settle waits for the resident to go quiet: no journal growth for a few polls. +settle() { + local limit="${1:-30}" last="" now waited=0 stable=0 + while [ "$waited" -lt "$limit" ]; do + now="$(mark)" + if [ "$now" = "$last" ]; then + stable=$((stable + 1)) + [ "$stable" -ge 3 ] && return 0 + else + stable=0 + fi + last="$now"; sleep 2; waited=$((waited + 2)) + done + return 0 +} + +dump_turn() { + local since="$1" + { + echo; echo '### messages since the mark' + journal_table "select seq,role,node_id,command_seq,substr(replace(body,char(10),' / '),1,220) as body from messages where seq > $since order by seq" + echo; echo '### commands since the mark' + journal_table "select seq,kind,status,target,substr(instruction,1,70) as instruction,substr(result,1,70) as result from commands where seq > $since order by seq" + echo; echo '### nodes since the mark' + journal_table "select id,parent_id,status,origin,grp,substr(title,1,50) as title from nodes where created_seq > $since order by created_seq" + echo; echo '### usage since the mark' + journal_table "select model,count(*) as calls,round(sum(cost),5) as cost from usage where seq > $since group by model" + } >> "$UX_DIR/notes.md" +} + +# finish is the last line of every journey. +finish() { + local verdict + if [ -n "$UX_FORCED" ]; then + verdict="$UX_FORCED" + elif [ "$UX_CHECKS" -eq 0 ]; then + verdict="OBSERVED" + elif [ "$UX_FAILURES" -eq 0 ]; then + verdict="PASS" + else + verdict="FAIL" + fi + printf '%s\n' "$verdict" > "$UX_DIR/verdict" + printf '%s/%s checks\n' "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" > "$UX_DIR/checks" + printf '\n**verdict: %s** (%d/%d checks)\n' "$verdict" "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" >> "$UX_DIR/notes.md" + printf ' \033[1m%s\033[0m — %d/%d checks\n' "$verdict" "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" >&2 + [ "$verdict" = "FAIL" ] && exit 1 + exit 0 +} diff --git a/test/ux/quality/20-coding.sh b/test/ux/quality/20-coding.sh new file mode 100755 index 0000000000..819bd6c35f --- /dev/null +++ b/test/ux/quality/20-coding.sh @@ -0,0 +1,115 @@ +#!/usr/bin/env bash +# Q1 · Does it actually solve a coding task? +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q1 Coding quality — a real programming task, graded by running the code the suite is given, not by reading the claim' + +# The whole point: codeaf saying "done, tests pass" is a claim. This journey +# takes the files it produced, runs them here, and grades the claim against +# what the interpreter says. + +expect_nodes 2 +since="$(mark)" +started="$(date +%s)" +touch "$UX_DIR/.stamp" + +say "write a python file called stats.py with a function median(numbers) that returns the median of a list of numbers and raises ValueError on an empty list. write pytest tests for it in test_stats.py. run the tests and tell me the result." + +assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ + 'the coding job finished' 420 +elapsed=$(( $(date +%s) - started )) +sleep 12 +snap delivered + +record 'wall time' "${elapsed}s" +record 'cost' "\$$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since")" +record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since")" +record 'node shape' "$(journal "select group_concat(id || '(' || status || ')' || case when parent_id='root' then '' else ' under ' || parent_id end, ' · ') from nodes where created_seq > $since")" + +# ------------------------------------------------------------ find the files + +work="$(find "$UX_STATE/workspace" -name 'stats.py' -newer "$UX_DIR/.stamp" 2>/dev/null | head -1)" +if [ -z "$work" ]; then + work="$(find "$UX_STATE/workspace" -name 'stats.py' 2>/dev/null | head -1)" +fi +if [ -n "$work" ]; then + dir="$(dirname "$work")" + _check yes 'stats.py exists in the workspace' 'a file the user can pick up' "$work" +else + dir="" + _check no 'stats.py exists in the workspace' 'a file the user can pick up' 'not found anywhere under the workspace' +fi +record 'workspace' "${dir:-}" +[ -n "$dir" ] && record 'files produced' "$(ls "$dir" | tr '\n' ' ')" +[ -n "$dir" ] && cp -R "$dir" "$UX_DIR/workspace" 2>/dev/null + +if [ -n "$dir" ] && [ -f "$dir/test_stats.py" ]; then + _check yes 'it wrote the tests it was asked for' 'test_stats.py' 'present' +else + _check no 'it wrote the tests it was asked for' 'test_stats.py' 'missing' +fi + +# ------------------------------------------------- grade by running the code + +if [ -n "$dir" ]; then + grade="$(cd "$dir" && python3 - <<'PY' 2>&1 +import importlib.util, json, sys +spec = importlib.util.spec_from_file_location("stats", "stats.py") +verdict = {"import": False, "odd": None, "even": None, "empty": None} +try: + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + verdict["import"] = True + median = getattr(module, "median") + verdict["odd"] = (median([3, 1, 2]) == 2) + verdict["even"] = (abs(median([1, 2, 3, 4]) - 2.5) < 1e-9) + try: + median([]) + verdict["empty"] = False + except ValueError: + verdict["empty"] = True + except Exception as exc: + verdict["empty"] = "wrong exception: %s" % type(exc).__name__ +except Exception as exc: + verdict["error"] = "%s: %s" % (type(exc).__name__, exc) +print(json.dumps(verdict)) +PY +)" + record 'independent grade' "$grade" + + for facet in odd even empty; do + if printf '%s' "$grade" | python3 -c "import json,sys; d=json.loads(sys.stdin.read()); sys.exit(0 if d.get('$facet') is True else 1)" 2>/dev/null; then + _check yes "median() is correct: $facet case" "the suite's own call returns the right answer" 'correct' + else + _check no "median() is correct: $facet case" "the suite's own call returns the right answer" "$grade" + fi + done + + # And the tests it wrote — do they pass when someone else runs them? + pytest_out="$(cd "$dir" && python3 -m pytest -q 2>&1 | tail -5)" + record 'pytest, run by the suite' "$(printf '%s' "$pytest_out" | tr '\n' ' ')" + if printf '%s' "$pytest_out" | grep -Eq '[0-9]+ passed'; then + _check yes 'the tests it wrote pass when the suite runs them' 'pytest green' "$(printf '%s' "$pytest_out" | grep -Eo '[0-9]+ passed[^ ]*' | head -1)" + elif printf '%s' "$pytest_out" | grep -Eq 'No module named pytest'; then + record 'note' 'pytest is not installed here; the direct grade above is the authority' + else + _check no 'the tests it wrote pass when the suite runs them' 'pytest green' "$(printf '%s' "$pytest_out" | tr '\n' ' ')" + fi +fi + +# ------------------------------------------------- did it run what it built? + +claim="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" +record 'what it claimed' "$(printf '%s' "$claim" | head -c 700)" + +if printf '%s' "$claim" | grep -Eq '[0-9]+ passed|tests? pass|all pass|passed in|=+ .*passed'; then + _check yes 'it ran the tests and reported real output, not a promise' \ + 'test output in the deliverable' "$(printf '%s' "$claim" | grep -Eo '[0-9]+ passed[a-z ,]*' | head -1)" +else + _check no 'it ran the tests and reported real output, not a promise' \ + 'test output in the deliverable' "$(printf '%s' "$claim" | head -c 200)" +fi + +judge_this 'write stats.py with median(numbers) raising ValueError on empty, write pytest tests in test_stats.py, run the tests and tell me the result' "$(deliverable_since "$since")" + +dump_turn "$since" +finish diff --git a/test/ux/quality/21-research.sh b/test/ux/quality/21-research.sh new file mode 100755 index 0000000000..98294c043a --- /dev/null +++ b/test/ux/quality/21-research.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# Q2 · Does it actually solve a research task? +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q2 Research quality — a finance question with checkable answers, graded against the real filed numbers' + +# Apple's FY2024 (ended 28 Sep 2024) 10-K: total net sales $391.035B, of which +# Services $96.169B and Products $294.866B. Those are the ground truth this +# journey grades against — a number in the right place is worth more than a +# confident paragraph, and a wrong number is worth less than nothing. + +expect_nodes 1 +since="$(mark)" +started="$(date +%s)" + +say "research Apple's total revenue in fiscal year 2024 and how much of it was Services. give me the two numbers and cite where they come from." + +assert_journal "select count(*) from messages where seq > $since and role in ('agent','system')" 'it replied' 240 +wait_journal "select count(*) from messages where seq > $since and role in ('agent','system') and length(body) > 120" 300 +sleep 15 +snap answered + +elapsed=$(( $(date +%s) - started )) +record 'wall time' "${elapsed}s" +record 'cost' "\$$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since")" +record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since")" + +answer="$(journal "select group_concat(body,char(10)) from messages where seq > $since and role in ('agent','system')")" +printf '%s\n' "$answer" > "$UX_DIR/answer.txt" +record 'answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 1200)" + +grade="$(printf '%s' "$answer" | python3 -c " +import re, sys +text = sys.stdin.read() +numbers = [float(n.replace(',', '')) for n in re.findall(r'[0-9][0-9,]*\.?[0-9]*', text)] +def near(target, tolerance): + return any(abs(n - target) <= tolerance for n in numbers) +total = near(391.0, 2.0) or near(391035.0, 2000.0) or near(391035000000.0, 2e9) +services = near(96.0, 1.5) or near(96169.0, 1500.0) or near(96169000000.0, 1.5e9) +print('total=%s services=%s' % (total, services)) +")" +record 'number grade' "$grade" + +printf '%s' "$grade" | grep -q 'total=True' \ + && _check yes 'total FY2024 revenue is right (\$391.0B)' 'a number within \$2B of 391.0' 'found' \ + || _check no 'total FY2024 revenue is right (\$391.0B)' 'a number within \$2B of 391.0' "$(printf '%s' "$answer" | grep -Eo '\$?[0-9][0-9,.]*( ?(billion|B|million))?' | head -8 | tr '\n' ' ')" + +printf '%s' "$grade" | grep -q 'services=True' \ + && _check yes 'Services revenue is right (\$96.2B)' 'a number within \$1.5B of 96.2' 'found' \ + || _check no 'Services revenue is right (\$96.2B)' 'a number within \$1.5B of 96.2' "$(printf '%s' "$answer" | grep -Eo '\$?[0-9][0-9,.]*( ?(billion|B))?' | head -8 | tr '\n' ' ')" + +# A citation is a document a person could go and check. "Form 8-K, filed +# 31 October 2024" is one; so is a URL. The earlier spelling accepted only the +# URL and failed a correctly cited answer. +if printf '%s' "$answer" | grep -Eqi 'http|sec\.gov|10-k|8-k|form 10|annual report|investor\.apple|press release|earnings release|filing'; then + _check yes 'the claim carries evidence, not confidence' 'a named source' "$(printf '%s' "$answer" | grep -Eoi 'https?://[^ )]*|sec\.gov[^ )]*|10-K' | head -3 | tr '\n' ' ')" +else + _check no 'the claim carries evidence, not confidence' 'a named source' 'no URL, filing, or report named' +fi + +if printf '%s' "$answer" | grep -Eqi 'https?://'; then + record 'looked it up' 'yes — a live URL appears in the answer' +else + record 'looked it up' 'no live URL in the answer; the numbers may be from model memory rather than the web tool' +fi + +judge_this "research Apple's total revenue in fiscal 2024 and how much was Services; give the two numbers and cite where they come from" "$(deliverable_since "$since")" + +dump_turn "$since" +finish diff --git a/test/ux/quality/22-quicker.sh b/test/ux/quality/22-quicker.sh new file mode 100755 index 0000000000..14861b99f9 --- /dev/null +++ b/test/ux/quality/22-quicker.sh @@ -0,0 +1,59 @@ +#!/usr/bin/env bash +# Q3 · Does asking for it quicker actually make it quicker? +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q3 Urgency — "quick, do not overthink" against a matched control: does the shape of the work change?' + +expect_nodes 2 + +run_one() { + local label="$1" sentence="$2" + local from t0 t1 + from="$(mark)" + t0="$(date +%s)" + say "$sentence" + wait_journal "select count(*) from messages where seq > $from and role in ('agent','system') and length(body) > 60" 300 + wait_journal "select count(*) from nodes where created_seq > $from and origin='user' and status in ('done','failed')" 60 >/dev/null + t1="$(date +%s)" + sleep 10 + QN="$(journal "select count(*) from nodes where created_seq > $from")" + QC="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $from")" + QCALLS="$(journal "select count(*) from usage where seq > $from")" + QT=$((t1 - t0)) + record "$label" "$QN nodes · $QCALLS model calls · \$$QC · ${QT}s" + printf '%s %s %s %s' "$QN" "$QCALLS" "$QC" "$QT" +} + +since="$(mark)" + +control="$(run_one 'control (no urgency words)' 'write a short explanation of what a binary search tree is into a file')" +snap control +urgent="$(run_one 'urgent ("quick, one pass")' 'quick, one pass, do not overthink this: write a short explanation of what a hash table is into a file')" +snap urgent + +read -r cn ccalls ccost ctime <<< "$control" +read -r un ucalls ucost utime <<< "$urgent" + +record 'nodes control vs urgent' "$cn vs $un" +record 'calls control vs urgent' "$ccalls vs $ucalls" +record 'cost control vs urgent' "\$$ccost vs \$$ucost" +record 'seconds control vs urgent' "${ctime}s vs ${utime}s" + +# The honest bar: urgency must not make the work BIGGER. Whether it makes it +# genuinely smaller on one paired sample is reported, not asserted — one pair +# is an anecdote, and the suite refuses to dress an anecdote as a measurement. +if [ "${un:-99}" -le "${cn:-0}" ]; then + _check yes 'urgency words did not inflate the work' "urgent nodes ≤ control nodes" "$un ≤ $cn" +else + _check no 'urgency words did not inflate the work' "urgent nodes ≤ control nodes" "$un > $cn" +fi + +cheaper="$(python3 -c "print('yes' if float('$ucost') <= float('$ccost') else 'no')")" +faster="$([ "${utime:-99999}" -le "${ctime:-0}" ] && echo yes || echo no)" +record 'OBSERVED — cheaper when asked to hurry?' "$cheaper" +record 'OBSERVED — faster when asked to hurry?' "$faster" + +urgency_cmd="$(journal "select group_concat(kind || '/' || status, ', ') from commands where seq > $since")" +record 'commands across both jobs' "${urgency_cmd:-}" + +dump_turn "$since" +finish diff --git a/test/ux/quality/23-learning-applied.sh b/test/ux/quality/23-learning-applied.sh new file mode 100755 index 0000000000..9e00b1b3b4 --- /dev/null +++ b/test/ux/quality/23-learning-applied.sh @@ -0,0 +1,60 @@ +#!/usr/bin/env bash +# Q4 · Does what it learned change what it does? +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q4 Learning applied — the lesson taught in J7 ("run what you build before saying it is done") must show up as behaviour, not as a stored string' + +lesson="$(journal "select group_concat(substr(body,1,120), ' | ') from facts where status='active' and (lower(body) like '%run%' or lower(body) like '%verif%')")" +record 'the lesson on the shelf' "${lesson:-}" +[ -n "$lesson" ] \ + && _check yes 'the lesson is still in the notebook when the next job starts' 'an active fact about running/verifying' "$(printf '%s' "$lesson" | head -c 160)" \ + || _check no 'the lesson is still in the notebook when the next job starts' 'an active fact about running/verifying' 'nothing' + +expect_nodes 1 +since="$(mark)" +# Deliberately NOT asking it to run anything. If the lesson landed, it runs the +# script anyway and shows the output. If it only stored a sentence, it will +# hand back a file and call it done. +say "write a python script called fizz.py that prints fizzbuzz for 1 to 15" + +assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ + 'the job finished' 360 +sleep 12 +snap delivered + +claim="$(journal "select group_concat(body,char(10)) from messages where seq > $since and role in ('agent','system')")" +printf '%s\n' "$claim" > "$UX_DIR/claim.txt" +record 'what it said' "$(printf '%s' "$claim" | tr '\n' ' ' | head -c 800)" + +# Evidence of having actually run it: real fizzbuzz output in the deliverable. +if printf '%s' "$claim" | grep -qi 'fizzbuzz' && printf '%s' "$claim" | grep -qi 'buzz' && printf '%s' "$claim" | grep -q '\b11\b'; then + _check yes 'it ran what it built and showed the output — the lesson changed behaviour' \ + 'real program output in the deliverable' 'fizzbuzz output present' + record 'LEARNING' 'APPLIED — the taught lesson visibly changed how the next job was reported' +else + _check no 'it ran what it built and showed the output — the lesson changed behaviour' \ + 'real program output in the deliverable' "$(printf '%s' "$claim" | tr '\n' ' ' | head -c 200)" + record 'LEARNING' 'NOT APPLIED — the lesson is stored but the next job did not honour it' +fi + +# And the file itself has to be right. +script="$(find "$UX_STATE/workspace" -name 'fizz*.py' 2>/dev/null | head -1)" +record 'script' "${script:-}" +if [ -n "$script" ]; then + out="$(python3 "$script" 2>&1 | tr '\n' ' ')" + record 'suite ran it' "$(printf '%s' "$out" | head -c 200)" + # Case is not the contract — the sequence is. 3 and 15 must be fizz-ish, + # 5 must be buzz-ish, 15 must be the combined word. + if printf '%s' "$out" | grep -qi 'fizzbuzz' && printf '%s' "$out" | grep -qi 'fizz' \ + && printf '%s' "$out" | grep -qi 'buzz' && printf '%s' "$out" | grep -q '11'; then + _check yes 'the script is correct when the suite runs it' 'fizzbuzz through 15' "$(printf '%s' "$out" | head -c 120)" + else + _check no 'the script is correct when the suite runs it' 'fizzbuzz through 15' "$(printf '%s' "$out" | head -c 200)" + fi +else + _check no 'the script exists on disk' 'a fizz*.py in the workspace' 'not found' +fi + +judge_this 'write a python script called fizz.py that prints fizzbuzz for 1 to 15' "$(deliverable_since "$since")" + +dump_turn "$since" +finish diff --git a/test/ux/quality/24-shape.sh b/test/ux/quality/24-shape.sh new file mode 100755 index 0000000000..9f72419803 --- /dev/null +++ b/test/ux/quality/24-shape.sh @@ -0,0 +1,106 @@ +#!/usr/bin/env bash +# Q5 · Is the decomposition any good? +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q5 Graph shape — across everything the suite commissioned: how many tasks did it form, and was that the right number?' + +expect_nodes 0 + +# Nothing is said in this journey. It reads the whole run's graph and judges the +# one thing a task graph can get obviously wrong in both directions: a haiku +# that became eleven tasks, or a research report that became none. + +record 'user jobs commissioned' "$(journal "select count(*) from nodes where origin='user' and parent_id='root'")" +record 'total nodes' "$(journal "select count(*) from nodes")" +record 'failed nodes' "$(journal "select count(*) from nodes where status='failed'")" +record 'cancelled nodes' "$(journal "select count(*) from nodes where status='cancelled'")" +record 'max depth below root' "$(journal "select coalesce(max(stage),0) from nodes")" + +{ + echo + echo '### every job the suite commissioned, and what it became' + echo + journal_table " + select + n.id as job, + n.status, + (select count(*) from nodes c where c.parent_id = n.id) as children, + (select count(*) from usage u where u.node_id = n.id) as calls, + round((select coalesce(sum(cost),0) from usage u where u.node_id = n.id), 5) as cost, + substr(n.title,1,44) as title + from nodes n + where n.origin='user' and n.parent_id='root' + order by n.created_order" +} >> "$UX_DIR/notes.md" + +# The other half of proportionality, and the more interesting one: how often +# did a job branch AT ALL? A graph product whose every job is one leaf is not +# over-decomposing, but it is also not decomposing, and a reader of this report +# should be told which of those two things is happening. +branched="$(journal " + select count(*) from nodes n + where n.origin='user' and n.parent_id='root' + and (select count(*) from nodes c where c.parent_id = n.id) > 0")" +total_jobs="$(journal "select count(*) from nodes where origin='user' and parent_id='root'")" +record 'jobs that decomposed into children at all' "$branched of $total_jobs" + +bloated="$(journal " + select count(*) from nodes n + where n.origin='user' and n.parent_id='root' + and (select count(*) from nodes c where c.parent_id = n.id) > 6")" +record 'jobs that fanned out past 6 children' "$bloated" +[ "${bloated:-0}" = "0" ] \ + && _check yes 'no toy job exploded into a bureaucracy' 'no user job with more than 6 children' "$bloated" \ + || _check no 'no toy job exploded into a bureaucracy' 'no user job with more than 6 children' "$bloated jobs did" + +# Held work is not forgotten work: J9 refuses a plan on purpose and the node +# that stays pending behind that refusal is the refusal being honoured. +orphans="$(journal "select count(*) from nodes where origin='user' and status in ('pending','claimed') and held=0 and created_seq < (select max(seq) - 200 from events)")" +record 'jobs still stuck pending long after they were asked for' "$orphans" +record 'jobs deliberately held (a refused plan is not a forgotten one)' \ + "$(journal "select count(*) from nodes where origin='user' and held=1")" +[ "${orphans:-0}" = "0" ] \ + && _check yes 'nothing was commissioned and then quietly forgotten' 'no stale pending user job' "$orphans" \ + || _check no 'nothing was commissioned and then quietly forgotten' 'no stale pending user job' "$orphans stuck" + +failed="$(journal "select count(*) from nodes where origin='user' and status='failed'")" +[ "${failed:-0}" = "0" ] \ + && _check yes 'no commissioned job failed outright' 'zero failed user nodes' "$failed" \ + || _check no 'no commissioned job failed outright' 'zero failed user nodes' "$(journal "select group_concat(id || ': ' || substr(error,1,60), ' | ') from nodes where origin='user' and status='failed'")" + +# Every user turn must have ended in a reply. Silence wearing a message id is +# the cardinal sin in docs/JOURNEY.md, so it gets its own count. +users="$(journal "select count(*) from messages where role='user'")" +agents="$(journal "select count(*) from messages where role in ('agent','system')")" +record 'user turns / agent+system replies' "$users / $agents" +[ "${agents:-0}" -ge "${users:-1}" ] \ + && _check yes 'every user turn was answered — no silence wearing a message id' 'replies ≥ user turns' "$agents ≥ $users" \ + || _check no 'every user turn was answered — no silence wearing a message id' 'replies ≥ user turns' "$agents < $users" + +# The thread is the only mouth, so a journaled agent message that says nothing +# is silence wearing a message id — the exact sin the law names. v3 draws a +# turn in flight as a titled block ("▸ worked …"), so there is no bare stream +# cursor to count; the law is measured where the message lives, in the journal: +# an agent message whose body is empty IS silence wearing a message id. +snap final-thread +blanks="$(journal "select count(*) from messages where role in ('agent','system') and trim(body) = ''")" +record 'journaled agent messages with an empty body' "${blanks:-0}" +[ "${blanks:-0}" = "0" ] \ + && _check yes 'agent messages carry words, not an empty message id' 'no agent message with an empty body' "${blanks:-0}" \ + || _check no 'agent messages carry words, not an empty message id' 'no agent message with an empty body' "$blanks empty bodies — journaled replies that said nothing" + +{ + echo + echo '### where the money went' + echo + journal_table "select model, count(*) as calls, sum(prompt_tokens) as prompt_tok, sum(completion_tokens) as out_tok, round(sum(cost),5) as cost from usage group by model order by cost desc" + echo + echo '### the notebook at the end of the run' + echo + journal_table "select kind, channel, status, substr(body,1,90) as body from facts order by seq" + echo + echo '### charters at the end of the run' + echo + journal_table "select id, status, autonomy, substr(invariant,1,60) as invariant from charters" +} >> "$UX_DIR/notes.md" + +finish diff --git a/test/ux/quality/25-writing.sh b/test/ux/quality/25-writing.sh new file mode 100755 index 0000000000..c5ec8c4637 --- /dev/null +++ b/test/ux/quality/25-writing.sh @@ -0,0 +1,56 @@ +#!/usr/bin/env bash +# Q6 · A writing task with no code and no numbers in it +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q6 Writing quality — a professional email, where the only measure is whether a person would send it' + +# Coding and research are the two tasks that grade themselves. Most work does +# not: it is a paragraph somebody has to be willing to put their name on. This +# is the journey that catches a product which is good at tests and bad at +# English. + +expect_nodes 1 +since="$(mark)" +started="$(date +%s)" + +say "draft a short professional email declining an invitation to speak at a conference next month because of a scheduling conflict. show me the email." + +assert_journal "select count(*) from messages where seq > $since and role in ('agent','system') and length(body) > 150" \ + 'a draft came back with something in it' 300 +sleep 10 +snap drafted + +elapsed=$(( $(date +%s) - started )) +record 'wall time' "${elapsed}s" + +draft="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" +printf '%s\n' "$draft" > "$UX_DIR/email.txt" +record 'the draft' "$(printf '%s' "$draft" | tr '\n' ' ' | head -c 800)" + +# The literal shape of an email, checked without a model: the parts a person +# would notice missing at a glance. +for part in 'subject' 'dear|hello|hi ' 'regards|sincerely|best'; do + if printf '%s' "$draft" | grep -Eqi "$part"; then + _check yes "the draft has the '$part' part of an email" "an email-shaped section" 'present' + else + _check no "the draft has the '$part' part of an email" "an email-shaped section" 'missing' + fi +done + +if printf '%s' "$draft" | grep -Eqi 'conflict|unable to|regret|decline|cannot attend|won.t be able'; then + _check yes 'it actually declines, and gives the reason it was told to give' \ + 'a decline citing the scheduling conflict' "$(printf '%s' "$draft" | grep -Eoi 'scheduling conflict|conflict|regret[a-z]*' | head -1)" +else + _check no 'it actually declines, and gives the reason it was told to give' \ + 'a decline citing the scheduling conflict' "$(printf '%s' "$draft" | head -c 200)" +fi + +if printf '%s' "$draft" | grep -Eq '\[|\{\{|XXX|TODO'; then + record 'placeholders' "the draft contains fill-in slots: $(printf '%s' "$draft" | grep -Eo '\[[^]]{1,30}\]' | head -3 | tr '\n' ' ')" +else + record 'placeholders' 'none — the draft is ready to send as written' +fi + +judge_this 'draft a short professional email declining an invitation to speak at a conference next month because of a scheduling conflict; show me the email' "$draft" + +dump_turn "$since" +finish diff --git a/test/ux/quality/26-data.sh b/test/ux/quality/26-data.sh new file mode 100755 index 0000000000..a96ea3f553 --- /dev/null +++ b/test/ux/quality/26-data.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Q7 · A data task with exactly one right answer +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q7 Data quality — five numbers, a ranking, and a median: arithmetic has a right answer and this checks it' + +# 12, 7, 41, 3, 28 sorted is 3, 7, 12, 28, 41 — the median is 12. There is +# nothing to argue about, which is the point: a product that is fluent and +# wrong fails here and nowhere else. + +expect_nodes 1 +since="$(mark)" +started="$(date +%s)" +touch "$UX_DIR/.stamp" + +say "here are five numbers: 12, 7, 41, 3, 28 — make a file ranking them from largest to smallest and tell me the median" + +assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ + 'the job finished' 300 +sleep 10 +snap answered +record 'wall time' "$(( $(date +%s) - started ))s" + +answer="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" +printf '%s\n' "$answer" > "$UX_DIR/answer.txt" +record 'the answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 600)" + +# The median is 12 and nothing else. +if printf '%s' "$answer" | grep -Eqi 'median[^0-9]{0,40}12|12[^0-9]{0,20}(is the )?median'; then + _check yes 'the median is right (12)' 'the number 12 named as the median' 'found' +else + _check no 'the median is right (12)' 'the number 12 named as the median' \ + "$(printf '%s' "$answer" | grep -Eoi 'median[^.]{0,40}' | head -1)" +fi + +# The ranking, largest first, in the file it was asked to write. +ranked="$(find "$UX_STATE/workspace" -type f -newer "$UX_DIR/.stamp" 2>/dev/null | head -5)" +record 'files written' "$(printf '%s' "$ranked" | tr '\n' ' ')" +found_order=0 +while IFS= read -r file; do + [ -z "$file" ] && continue + order="$(grep -Eo '\b(41|28|12|7|3)\b' "$file" 2>/dev/null | tr '\n' ' ')" + record "order in $(basename "$file")" "$order" + case "$order" in "41 28 12 7 3 "*) found_order=1 ;; esac +done <<< "$ranked" + +if [ "$found_order" = "1" ]; then + _check yes 'the file ranks them largest to smallest' '41 28 12 7 3 in that order' 'correct' +else + # The thread may carry the ranking even when the file is shaped differently. + if printf '%s' "$answer" | grep -Eq '41.{0,12}28.{0,12}12.{0,12}7.{0,12}3'; then + _check yes 'the ranking is right, largest first' '41 28 12 7 3 in that order' 'correct, in the thread' + else + _check no 'the ranking is right, largest first' '41 28 12 7 3 in that order' \ + "$(printf '%s' "$answer" | grep -Eo '41[^a-z]{0,40}' | head -1)" + fi +fi + +judge_this 'here are five numbers: 12, 7, 41, 3, 28 — make a file ranking them from largest to smallest and tell me the median' "$answer" + +dump_turn "$since" +finish diff --git a/test/ux/quality/27-errand.sh b/test/ux/quality/27-errand.sh new file mode 100755 index 0000000000..df28de59d0 --- /dev/null +++ b/test/ux/quality/27-errand.sh @@ -0,0 +1,54 @@ +#!/usr/bin/env bash +# Q8 · An everyday errand, shaped the way a person actually asks +source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" +journey_is 'Q8 Errand quality — an ordinary, slightly vague household ask: does it just do it, or does it interview you first?' + +# Nobody says "produce a structured artifact enumerating". They say "make me a +# packing list". The failure this catches is a product that answers a small +# errand with a clarifying question, a plan, and a preamble. + +expect_nodes 1 +since="$(mark)" +started="$(date +%s)" +touch "$UX_DIR/.stamp" + +say "make me a packing list for a 3-day work trip and save it as a file" + +assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ + 'the errand was actually run' 300 +sleep 10 +snap done +elapsed=$(( $(date +%s) - started )) +record 'wall time' "${elapsed}s" + +answer="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" +printf '%s\n' "$answer" > "$UX_DIR/answer.txt" +record 'the answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 700)" + +# It must have just done it: no question asked back for something this small. +asked="$(journal "select count(*) from agent_questions where seq > $since")" +record 'questions asked back' "$asked" +[ "${asked:-0}" = "0" ] \ + && _check yes 'a small errand is done, not interviewed' 'no clarifying question for a 3-day packing list' '0 questions' \ + || _check no 'a small errand is done, not interviewed' 'no clarifying question for a 3-day packing list' "$asked asked" + +# The list itself has to be a list of things a person packs. +hits=0 +for item in charger toothbrush shirt sock laptop 'passport|id' 'trousers|pants' shoes; do + printf '%s' "$answer" | grep -qiE "$item" && hits=$((hits + 1)) +done +record 'ordinary packing items named' "$hits of 8" +[ "$hits" -ge 4 ] \ + && _check yes 'the list is a real packing list' 'at least 4 of the obvious items' "$hits" \ + || _check no 'the list is a real packing list' 'at least 4 of the obvious items' "$hits" + +file="$(find "$UX_STATE/workspace" -type f -newer "$UX_DIR/.stamp" 2>/dev/null | head -1)" +record 'file written' "${file:-}" +[ -n "$file" ] && [ -s "$file" ] \ + && _check yes 'it saved the file it was asked to save' 'a non-empty file in the workspace' "$(wc -c < "$file" | tr -d ' ') bytes" \ + || _check no 'it saved the file it was asked to save' 'a non-empty file in the workspace' "${file:-nothing written}" + +judge_this 'make me a packing list for a 3-day work trip and save it as a file' "$answer" + +dump_turn "$since" +finish diff --git a/test/ux/run.sh b/test/ux/run.sh new file mode 100755 index 0000000000..d9c534f5ac --- /dev/null +++ b/test/ux/run.sh @@ -0,0 +1,435 @@ +#!/usr/bin/env bash +# run.sh — drive the real codeaf, as a person, and write down which journeys +# are true today. +# +# The doctrine is docs/JOURNEY.md: a journey passes only when the screen showed +# it, the journal durably recorded it, and nothing else happened. So this +# builds the real binary, gives it a disposable brain that shares nothing with +# the user's, opens it in a real terminal, and types. +# +# test/ux/run.sh everything +# test/ux/run.sh --only 01,02,05 these journeys +# test/ux/run.sh --suite quality the quality suite only +# test/ux/run.sh --keep leave the tmux session and home behind +# test/ux/run.sh --work-model SLUG repin the model that does the work +# test/ux/run.sh --tag NAME write evidence-NAME/ and report-NAME.md +# test/ux/tier-matrix.sh the same journeys on two work models +# +# Money: every model call lands in the disposable store's usage table, and the +# cumulative total is checked between journeys against UX_CAP (default $5). +# Over the cap the run aborts — a test suite must not be able to spend a +# surprise. + +set -uo pipefail + +UX_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO="$(cd "$UX_ROOT/../.." && pwd)" + +: "${UX_CAP:=5.00}" +: "${UX_WIDTH:=200}" +: "${UX_HEIGHT:=50}" +: "${UX_REAL_HOME:=$HOME}" + +SUITES=(journeys quality) +ONLY="" +KEEP=0 +WORK_MODEL="" +TAG="" +REPORT_OVERRIDE="" + +while [ $# -gt 0 ]; do + case "$1" in + --only) ONLY="$2"; shift 2 ;; + --suite) SUITES=("$2"); shift 2 ;; + --keep) KEEP=1; shift ;; + --cap) UX_CAP="$2"; shift 2 ;; + # --work-model repins the model that does the WORK for this whole run. The + # talk model is left alone on purpose: the tier question is whether a + # smarter executor earns its tokens, not whether a smarter receptionist does. + --work-model) WORK_MODEL="$2"; shift 2 ;; + --tag) TAG="$2"; shift 2 ;; + --report) REPORT_OVERRIDE="$2"; shift 2 ;; + -h|--help) sed -n '2,21p' "$0"; exit 0 ;; + *) echo "unknown flag $1" >&2; exit 2 ;; + esac +done + +for tool in tmux sqlite3 python3; do + command -v "$tool" >/dev/null || { echo "run.sh needs $tool on PATH" >&2; exit 2; } +done + +# ------------------------------------------------------------------ the home +# +# Disposable, fresh per run, and named nothing like the user's. HOME is moved +# as well as CODEAF_HOME: the state root override covers everything codeaf +# writes on purpose, and moving HOME too means that anything which does not yet +# honour it — a launchd plist, a stray dotfile — lands here rather than in the +# user's actual home. Whatever shows up under $UX_HOME/.codeaf is a leak, and +# the report says so. + +RUN_ID="$(date +%Y%m%d-%H%M%S)" +UX_RUN="${UX_RUN:-$(mktemp -d "${TMPDIR:-/tmp}/codeaf-ux-$RUN_ID-XXXX")}" +UX_HOME="$UX_RUN/home" +UX_STATE="$UX_HOME/state" +UX_DB="$UX_STATE/graph.db" +# Evidence is filed under the tag when the caller named one, so a tier run or a +# hand-run subset never overwrites the plain run's evidence. .gitignore already +# covers evidence-*/ and report-*.md. +UX_EVIDENCE="$UX_ROOT/evidence${TAG:+-$TAG}" +UX_REPORT="${REPORT_OVERRIDE:-$UX_ROOT/report${TAG:+-$TAG}.md}" +UX_SESSION="codeaf-ux-$$" +UX_SESSION_B="codeaf-ux-$$-b" +UX_BIN="${UX_BIN:-$REPO/bin/codeaf}" + +mkdir -p "$UX_STATE" "$UX_HOME/Library/LaunchAgents" +rm -rf "$UX_EVIDENCE"; mkdir -p "$UX_EVIDENCE" +echo 'journey,verdict,jobs,nodes,cost,tokens_in,tokens_out,seconds,quality,flags' > "$UX_EVIDENCE/gauges.csv" + +cleanup() { + local code=$? + tmux kill-session -t "$UX_SESSION" 2>/dev/null + tmux kill-session -t "$UX_SESSION_B" 2>/dev/null + if [ "$KEEP" = "1" ]; then + echo "kept: $UX_RUN" + else + rm -rf "$UX_RUN" + fi + exit $code +} +trap cleanup EXIT INT TERM + +# ------------------------------------------------------- provider + settings +# +# The one and only read of the user's real store: the API key, and the model +# pins as a baseline. Nothing is ever written back there. + +# A machine that keeps its key in the environment (OPENROUTER_API_KEY, the same +# variable codeaf itself reads first) is as valid a source as a persisted +# config.json, and refusing it made the suite unrunnable on boxes that have +# never run `codeaf` interactively. Neither source is ever written back. +if [ ! -f "$UX_REAL_HOME/.codeaf/config.json" ] \ + && [ -z "${OPENROUTER_API_KEY:-}${OPENAI_API_KEY:-}" ]; then + echo "no $UX_REAL_HOME/.codeaf/config.json and no OPENROUTER_API_KEY — the suite needs a provider key" >&2 + exit 2 +fi + +python3 - "$UX_REAL_HOME/.codeaf" "$UX_STATE" "$WORK_MODEL" <<'PY' +import json, os, sys +real, disposable = sys.argv[1], sys.argv[2] +work_model = sys.argv[3] if len(sys.argv) > 3 else "" +try: + config = json.load(open(os.path.join(real, "config.json"))) +except FileNotFoundError: + config = {} +key = config.get("api_key", "") or os.environ.get("OPENROUTER_API_KEY", "") or os.environ.get("OPENAI_API_KEY", "") +if not key: + raise SystemExit("no api_key in the real config.json and none in the environment") +json.dump({"api_key": key}, open(os.path.join(disposable, "config.json"), "w")) +os.chmod(os.path.join(disposable, "config.json"), 0o600) + +# Settings are the user's pins as a baseline — they already name the cheapest +# capable models — with the rails a disposable run wants on top. +try: + settings = json.load(open(os.path.join(real, "settings.json"))) +except FileNotFoundError: + settings = {} +settings.setdefault("chat_model", "~deepseek/deepseek-v4-flash-latest") +settings.setdefault("task_model", "~deepseek/deepseek-v4-flash-latest") +if work_model: + settings["task_model"] = work_model + # The plan model follows the work model for a tier run, or the comparison + # measures two things at once and attributes the difference to one of them. + settings["plan_model"] = work_model +json.dump(settings, open(os.path.join(disposable, "settings.json"), "w"), indent=2) +print("talk=%s work=%s plan=%s boost=%s" % ( + settings.get("chat_model"), settings.get("task_model"), + settings.get("plan_model"), settings.get("boost_model"))) +PY +[ $? -eq 0 ] || exit 2 + +# ------------------------------------------------------------------- binary + +# UX_SKIP_BUILD=1 with a pre-set UX_BIN runs a binary somebody else built — +# a build from a known commit while the working tree is mid-edit, say — so a +# tree that moves underneath a long run cannot change the program mid-suite. +if [ "${UX_SKIP_BUILD:-0}" = "1" ]; then + [ -x "$UX_BIN" ] || { echo "UX_SKIP_BUILD=1 but $UX_BIN is not executable" >&2; exit 2; } + echo "using $UX_BIN (build skipped)" +else + echo "building $UX_BIN" + ( cd "$REPO" && go build -o "$UX_BIN" ./cmd/codeaf ) || exit 2 +fi + +# UX_ENV is the one description of the disposable brain. Both this runner and +# the journeys that relaunch it (leave-and-return, second window) build their +# tmux command from exactly this string, so a window opened by journey 12 is +# the same brain the suite has been talking to all along. +UX_ENV="HOME=$UX_HOME CODEAF_HOME=$UX_STATE CODEAF_PROFILE_DIR=$UX_STATE" +UX_ENV="$UX_ENV CODEAF_DAILY_BUDGET=$UX_CAP CODEAF_BRIEF_AFTER=0 CODEAF_PRACTICE_BUDGET=0" +UX_ENV="$UX_ENV TERM=xterm-256color" + +export UX_BIN UX_HOME UX_STATE UX_DB UX_EVIDENCE UX_CAP UX_ROOT REPO UX_RUN UX_ENV +export UX_SESSION UX_SESSION_B UX_WIDTH UX_HEIGHT + +launch() { + local sess="$1"; shift + tmux kill-session -t "$sess" 2>/dev/null + tmux new-session -d -s "$sess" -x "$UX_WIDTH" -y "$UX_HEIGHT" \ + "env $UX_ENV $* '$UX_BIN' chat; echo CODEAF-EXITED; sleep 900" + sleep 6 +} + +# --------------------------------------------------------------- the money + +total_spend() { sqlite3 -readonly "$UX_DB" "select coalesce(round(sum(cost),6),0) from usage" 2>/dev/null || echo 0; } + +over_cap() { + local spent="$1" + python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > float(sys.argv[2]) else 1)" "$spent" "$UX_CAP" +} + +# --------------------------------------------------------------- the journeys + +declare -a ORDER=() +for suite in "${SUITES[@]}"; do + for script in "$UX_ROOT/$suite"/*.sh; do + [ -e "$script" ] || continue + ORDER+=("$script") + done +done + +wanted() { + [ -z "$ONLY" ] && return 0 + local name; name="$(basename "$1" .sh)" + case ",$ONLY," in *",${name%%-*},"*) return 0 ;; *",$name,"*) return 0 ;; esac + return 1 +} + +echo +echo "codeaf UX suite — $(date)" +echo " binary $UX_BIN" +echo " home $UX_HOME (real home untouched: $UX_REAL_HOME)" +echo " session $UX_SESSION at ${UX_WIDTH}x${UX_HEIGHT}" +echo " cap \$$UX_CAP" +echo + +launch "$UX_SESSION" +# The first frame: v3 draws the composer prompt "›" before anything has been +# said, and the wordmark rides the frame's own chrome. Accepting either is not +# a weakened check — the point of it is "the program drew something of its +# own", and CODEAF-EXITED on the pane means it did not. +if ! tmux capture-pane -t "$UX_SESSION" -p | grep -Eq 'codeaf|›'; then + echo "codeaf did not draw a first frame; aborting" >&2 + tmux capture-pane -t "$UX_SESSION" -p + exit 1 +fi +if tmux capture-pane -t "$UX_SESSION" -p | grep -q 'CODEAF-EXITED'; then + echo "codeaf exited instead of drawing a frame; aborting" >&2 + tmux capture-pane -t "$UX_SESSION" -p + exit 1 +fi + +declare -a RESULTS=() +declare -a GAUGES=() +ABORTED="" +JUDGE_CALLS=0 +START_TS="$(date +%s)" + +sql() { sqlite3 -readonly "$UX_DB" "$1" 2>/dev/null; } +watermark() { sql "select coalesce(max(seq),0) from events"; } + +for script in "${ORDER[@]}"; do + name="$(basename "$script" .sh)" + wanted "$script" || continue + + before="$(total_spend)" + mark_before="$(watermark)" + started="$(date +%s)" + UX_JOURNEY="$name" UX_SESSION="$UX_SESSION" bash "$script" + code=$? + elapsed=$(( $(date +%s) - started )) + after="$(total_spend)" + cost="$(python3 -c "print('%.4f' % (float('$after') - float('$before')))")" + + verdict="$(cat "$UX_EVIDENCE/$name/verdict" 2>/dev/null || echo FAIL)" + checks="$(cat "$UX_EVIDENCE/$name/checks" 2>/dev/null || echo '0/0 checks')" + [ "$code" -ne 0 ] && [ "$verdict" = "PASS" ] && verdict=FAIL + RESULTS+=("$name|$verdict|$checks|${elapsed}s|\$$cost") + + # ------------------------------------------------------------- the gauges + # + # Proportionality: how much machinery did the ask actually buy? A haiku is + # one node. A haiku that became five is a passing journey with a quality + # problem, and the report has to be able to say so. + nodes="$(sql "select count(*) from nodes where created_seq > $mark_before")" + jobs="$(sql "select count(*) from nodes where created_seq > $mark_before and origin='user' and parent_id='root'")" + tok_in="$(sql "select coalesce(sum(prompt_tokens),0) from usage where seq > $mark_before")" + tok_out="$(sql "select coalesce(sum(completion_tokens),0) from usage where seq > $mark_before")" + calls="$(sql "select count(*) from usage where seq > $mark_before")" + users="$(sql "select count(*) from messages where seq > $mark_before and role='user'")" + agents="$(sql "select count(*) from messages where seq > $mark_before and role='agent'")" + systems="$(sql "select count(*) from messages where seq > $mark_before and role='system'")" + expect="$(cat "$UX_EVIDENCE/$name/expect-nodes" 2>/dev/null || echo 1)" + + # Noise: agent turns beyond one reply per thing the user said. Receipts and + # deliverables are not noise — they are the product working. + noise="$(python3 -c "print(max(0, ${agents:-0} - ${users:-0}))")" + + flags="" + if [ "${nodes:-0}" -gt "${expect:-1}" ] && [ "${nodes:-0}" -gt 2 ]; then + flags="$flags OVER-DECOMPOSED(${nodes}v${expect})" + fi + [ "${noise:-0}" -ge 2 ] && flags="$flags CHATTY(+$noise)" + python3 -c "import sys; sys.exit(0 if float('$cost') > 0.05 else 1)" && flags="$flags EXPENSIVE" + [ "${elapsed:-0}" -gt 180 ] && flags="$flags SLOW" + + # Tokens are the honest unit. Dollars hide behind whichever model happened to + # answer; 50k of input tokens to write a haiku is disproportionate whoever + # charged for it, and it is the number that moves when a smarter work model + # re-reads the same context on every turn. + [ "${tok_in:-0}" -gt 50000 ] && flags="$flags TOKEN-HEAVY(${tok_in}in)" + + # ------------------------------------------------------------- the judge + quality="—"; why="" + if [ -s "$UX_EVIDENCE/$name/judge-deliverable" ]; then + judged="$(python3 "$UX_ROOT/judge.py" "$UX_STATE" \ + "$UX_EVIDENCE/$name/judge-ask" "$UX_EVIDENCE/$name/judge-deliverable" 2>/dev/null)" + JUDGE_CALLS=$((JUDGE_CALLS + 1)) + quality="${judged%%|*}" + why="${judged#*|}" + { + echo + echo "### quality judge" + echo + echo "- **score**: $quality/5" + echo "- **why**: $why" + } >> "$UX_EVIDENCE/$name/notes.md" + printf ' · quality %s/5 — %s\n' "$quality" "$why" >&2 + case "$quality" in + 1|2) flags="$flags POOR-RESULT" ;; + 3) flags="$flags MIDDLING" ;; + esac + fi + + concurrency="$(cat "$UX_EVIDENCE/$name/concurrency" 2>/dev/null || echo '')" + GAUGES+=("$name|$verdict|${jobs:-0}/${nodes:-0}|\$$cost|${tok_in:-0}|${tok_out:-0}|${elapsed}|$quality|${flags:-—}|$why|$concurrency|${calls:-0}") + printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,"%s"\n' \ + "$name" "$verdict" "${jobs:-0}" "${nodes:-0}" "$cost" "${tok_in:-0}" "${tok_out:-0}" \ + "$elapsed" "$quality" "${flags# }" >> "$UX_EVIDENCE/gauges.csv" + + if over_cap "$after"; then + ABORTED="spend \$$after exceeded the \$$UX_CAP cap after $name" + echo "ABORT: $ABORTED" >&2 + break + fi +done + +TOTAL="$(total_spend)" +WALL=$(( $(date +%s) - START_TS )) + +# --------------------------------------------------------------- leak check + +LEAKS="$(find "$UX_HOME" -maxdepth 1 -name '.codeaf' 2>/dev/null)" +PLIST="$(ls "$UX_HOME/Library/LaunchAgents" 2>/dev/null | tr '\n' ' ')" + +# ------------------------------------------------------------------ report + +{ + echo "# codeaf UX suite — which journeys are true today" + echo + echo "Run $(date -u '+%Y-%m-%d %H:%M UTC') · binary \`$(cd "$REPO" && git rev-parse --short HEAD)\` · wall ${WALL}s" + echo + echo "Real binary, real terminal (tmux ${UX_WIDTH}x${UX_HEIGHT}), real provider (OpenRouter)," + echo "disposable brain at \`\$CODEAF_HOME\`. Models: $(python3 -c " +import json;s=json.load(open('$UX_STATE/settings.json'));print('talk %s · work %s · plan %s · boost %s' % (s.get('chat_model'),s.get('task_model'),s.get('plan_model'),s.get('boost_model')))")" + echo + echo "**Total spend: \$$TOTAL** (cap \$$UX_CAP) · plus $JUDGE_CALLS quality-judge calls made outside codeaf and not in its journal" + [ -n "$ABORTED" ] && echo && echo "> **RUN ABORTED** — $ABORTED" + echo + echo '## The report card' + echo + echo 'Completion is the cheap half. `jobs/nodes` is how much machinery the ask bought' + echo '(a haiku should be one node); `quality` is a cheap model reading the deliverable' + echo 'against a fixed rubric — literal ask satisfied, answer-first, competent, 1–5.' + echo + echo '| journey | pass | jobs/nodes | $ | tokens in/out | calls | seconds | quality/5 | flags |' + echo '|---|---|---|---|---|---|---|---|---|' + for row in "${GAUGES[@]}"; do + IFS='|' read -r n v nodes money tin tout secs q f _why _conc calls <<< "$row" + echo "| $n | $v | $nodes | $money | $tin / $tout | $calls | $secs | $q | $f |" + done + echo + for row in "${GAUGES[@]}"; do + IFS='|' read -r n v nodes money tin tout secs q f why conc calls <<< "$row" + [ -n "$why" ] && echo "- **$n** judged $q/5 — $why" + done + echo + for row in "${GAUGES[@]}"; do + IFS='|' read -r n v nodes money tin tout secs q f why conc calls <<< "$row" + if [ -n "$conc" ]; then + echo "**Parallelism** — \`$n\` ran ${nodes%%/*} jobs from one sentence in ${secs}s of the person's time." + echo "Concurrency factor **${conc}×** (summed job seconds ÷ wall seconds). 1.0 means a queue;" + echo 'above it the graph is doing what terminal tabs would otherwise be for.' + echo + fi + done + echo '## Journeys' + echo + echo '| journey | verdict | checks | time | cost | evidence |' + echo '|---|---|---|---|---|---|' + for row in "${RESULTS[@]}"; do + IFS='|' read -r n v c t m <<< "$row" + echo "| $n | **$v** | $c | $t | $m | [notes](evidence/$n/notes.md) |" + done + echo + echo '### Not run' + echo + echo '| journey | why |' + echo '|---|---|' + echo '| Practice loop | idle-time loop: `CODEAF_PRACTICE_IDLE` defaults to 20m of quiet before self-practice, so an honest test costs 20 minutes of nothing happening. Out of scope for a suite that must stay under a few minutes per journey. |' + echo "| J9 consent (full) | needs a plan estimated over the real \$3 rail. The cheap variant runs instead: \`CODEAF_PLAN_CONSENT\` lowered so the same gate fires on a job costing cents. |" + echo + echo '### Housekeeping' + echo + if [ -n "$LEAKS" ]; then + echo "- ⚠️ **state leak**: something wrote to \`\$HOME/.codeaf\` despite \`CODEAF_HOME\` — $LEAKS" + else + echo '- ✅ no state leak: nothing landed in `$HOME/.codeaf`; the whole run lived under `$CODEAF_HOME`.' + fi + if [ -n "$PLIST" ]; then + echo "- ⚠️ the run installed a launch agent in the disposable home: $PLIST" + else + echo '- ✅ no launch agent installed (the standing-watch offer is declined by the suite).' + fi + echo "- the user's real store at \`$UX_REAL_HOME/.codeaf\` was read once (api_key, settings.json) and never written." + echo + for row in "${RESULTS[@]}"; do + IFS='|' read -r n v c t m <<< "$row" + echo "---" + echo + echo "## $n — $v" + echo + sed 's/^# .*$//' "$UX_EVIDENCE/$n/notes.md" 2>/dev/null + echo + done +} > "$UX_REPORT" + +echo +echo "================================================================" +printf '%-26s %-9s %-14s %6s %8s\n' journey verdict checks time cost +for row in "${RESULTS[@]}"; do + IFS='|' read -r n v c t m <<< "$row" + printf '%-26s %-9s %-14s %6s %8s\n' "$n" "$v" "$c" "$t" "$m" +done +echo "----------------------------------------------------------------" +echo "total spend \$$TOTAL / cap \$$UX_CAP · report: $UX_REPORT" +echo + +FAILED=0 +for row in "${RESULTS[@]}"; do + IFS='|' read -r n v c t m <<< "$row" + [ "$v" = "FAIL" ] && FAILED=$((FAILED + 1)) +done +[ -n "$ABORTED" ] && FAILED=$((FAILED + 1)) +[ "$FAILED" -gt 0 ] && exit 1 +exit 0 diff --git a/test/ux/tier-matrix.sh b/test/ux/tier-matrix.sh new file mode 100755 index 0000000000..b011e77da8 --- /dev/null +++ b/test/ux/tier-matrix.sh @@ -0,0 +1,134 @@ +#!/usr/bin/env bash +# tier-matrix.sh — the same four journeys, twice, on two work models. +# +# The product's claim is that it is cheap AND good on a small work model. Two +# things can falsify that. It might be cheap and bad, which the quality column +# catches. Or it might be no better when handed a smarter model — paying tier-2 +# prices for tier-1 results — and worse, burning materially more tokens on the +# same small ask because a bigger model re-reads more context per turn. That +# second failure is invisible in dollars alone, which is why tokens are in the +# table beside them. +# +# test/ux/tier-matrix.sh flash vs z-ai/glm-5.2 +# test/ux/tier-matrix.sh other/model flash vs whatever you name +# +# Each tier gets a fresh disposable home. The talk model stays flash in both: +# the question is about the executor, not the receptionist. The table is +# appended to test/ux/report.md when one exists. + +set -uo pipefail + +UX_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +SUBSET="${UX_TIER_SUBSET:-02,04,20,21}" +UPPER="${1:-z-ai/glm-5.2}" +BASE_LABEL="flash (settings default)" + +echo "tier matrix: $SUBSET on the default work model, then on $UPPER" + +"$UX_ROOT/run.sh" --only "$SUBSET" --tag tier-base --report "$UX_ROOT/report-tier-base.md" +base_code=$? +"$UX_ROOT/run.sh" --only "$SUBSET" --tag tier-upper --work-model "$UPPER" \ + --report "$UX_ROOT/report-tier-upper.md" +upper_code=$? + +python3 - "$UX_ROOT" "$BASE_LABEL" "$UPPER" <<'PY' +import csv, os, sys + +root, base_label, upper_label = sys.argv[1], sys.argv[2], sys.argv[3] + + +def read(tag): + path = os.path.join(root, "evidence-%s" % tag, "gauges.csv") + rows = {} + try: + with open(path) as handle: + for row in csv.DictReader(handle): + rows[row["journey"]] = row + except FileNotFoundError: + pass + return rows + + +base, upper = read("tier-base"), read("tier-upper") + + +def number(row, key, default=0.0): + try: + return float(row.get(key) or default) + except ValueError: + return default + + +lines = [] +lines.append("## The model-tier matrix") +lines.append("") +lines.append("The same journeys, the same words, two work models. The talk model is flash in") +lines.append("both rows — the question is whether a smarter *executor* earns its tokens.") +lines.append("") +lines.append("| journey | tier | $ | tokens in/out | seconds | quality/5 | verdict |") +lines.append("|---|---|---|---|---|---|---|") +flags = [] +for journey in sorted(set(base) | set(upper)): + for label, rows in ((base_label, base), (upper_label, upper)): + row = rows.get(journey) + if not row: + lines.append("| %s | %s | — | — | — | — | not run |" % (journey, label)) + continue + lines.append("| %s | %s | $%s | %s / %s | %s | %s | %s |" % ( + journey, label, row["cost"], row["tokens_in"], row["tokens_out"], + row["seconds"], row["quality"], row["verdict"])) + + low, high = base.get(journey), upper.get(journey) + if not (low and high): + continue + low_q, high_q = low.get("quality", "—"), high.get("quality", "—") + if low_q.isdigit() and high_q.isdigit() and int(high_q) <= int(low_q): + flags.append( + "- **TIER-WASTE — %s**: the bigger model scored %s/5 against flash's %s/5 " + "while costing $%s instead of $%s. Paying more bought nothing." + % (journey, high_q, low_q, high["cost"], low["cost"])) + low_in, high_in = number(low, "tokens_in"), number(high, "tokens_in") + if low_in > 0 and high_in > low_in * 1.5: + flags.append( + "- **TIER-WASTE — %s**: the same small ask burned %d input tokens on the bigger " + "model against %d on flash (%.1f×). Whatever the answer was worth, that context " + "was re-read for it." % (journey, high_in, low_in, high_in / low_in)) + elif high_in > 0 and low_in > high_in * 1.5: + flags.append( + "- %s: the bigger model used FEWER input tokens (%d vs %d) — it needed fewer " + "turns to get there." % (journey, high_in, low_in)) + +lines.append("") +if flags: + lines.append("### What the matrix says") + lines.append("") + lines.extend(flags) +else: + lines.append("No tier-waste flags: the bigger model either scored higher or used") + lines.append("comparable tokens on every journey in the subset.") +lines.append("") + +base_total = sum(number(r, "cost") for r in base.values()) +upper_total = sum(number(r, "cost") for r in upper.values()) +base_tok = sum(number(r, "tokens_in") for r in base.values()) +upper_tok = sum(number(r, "tokens_in") for r in upper.values()) +lines.append("**Subset totals** — %s: $%.4f over %d input tokens · %s: $%.4f over %d input tokens" + % (base_label, base_total, base_tok, upper_label, upper_total, upper_tok)) +if base_total > 0: + lines.append("") + lines.append("The upper tier cost **%.1f×** the flash tier for this subset." + % (upper_total / base_total)) +lines.append("") + +section = "\n".join(lines) +report = os.path.join(root, "report.md") +if os.path.exists(report): + with open(report, "a") as handle: + handle.write("\n---\n\n" + section) + print("appended the tier matrix to report.md") +with open(os.path.join(root, "report-tiers.md"), "w") as handle: + handle.write(section) +print(section) +PY + +exit $(( base_code || upper_code )) From 8c88f9a21a94dee9c95a7a83a847f207b07666d9 Mon Sep 17 00:00:00 2001 From: santoshkumarradha Date: Wed, 16 Sep 2026 16:32:00 -0400 Subject: [PATCH 2/8] chore: drop internal conversation-runtime wave records, keep the measurement contract The conversation-runtime directory was a local development wave: day-by-day status, /private/tmp paths, "owner authorized" process notes, draft PR numbers, and local change logs. None of it is current-product documentation a public reader needs. Keep PARETO.md, which is the benchmark's cited measurement contract, trimmed of its dangling calibration link and owner language. Co-Authored-By: codeaf --- .../CALIBRATION-20260905.md | 106 --- .../conversation-runtime/COMMUNICATION.md | 48 - .../conversation-runtime/EXECUTION-PLAN.md | 211 ----- .../HEADER-PANEL-20260907.md | 114 --- .../HISTORY-SEARCH-EXPERIMENT.md | 175 ---- .../conversation-runtime/IMPLEMENTATION.md | 124 --- docs/design/conversation-runtime/MODULES.md | 252 ----- docs/design/conversation-runtime/PARETO.md | 6 +- docs/design/conversation-runtime/PLAN.md | 284 ------ docs/design/conversation-runtime/PRODUCT.md | 44 - .../PRODUCTION-AUDIT-20260906.md | 869 ------------------ .../PROMPTS-CONTEXT-TOOLS.md | 109 --- .../conversation-runtime/QUALITY-20260905.md | 123 --- .../REAL-WORK-EVALUATION.md | 97 -- .../RECURSIVE-CORE-20260905.md | 97 -- .../TAKEOVER-STATUS-20260905.md | 545 ----------- .../TASK-EXPERIENCE-PROTOTYPE.md | 141 --- .../design/conversation-runtime/VALIDATION.md | 210 ----- .../local-change-notes/await-own-command.md | 16 - .../local-change-notes/pareto-measurement.md | 16 - .../prefix-cache-credit-bound.md | 126 --- .../prompt-shape-composition.md | 76 -- .../steer-young-bash-grace.md | 116 --- .../conversation-runtime/prototype/README.md | 28 - .../prototype/conversation-tasks.html | 190 ---- 25 files changed, 3 insertions(+), 4120 deletions(-) delete mode 100644 docs/design/conversation-runtime/CALIBRATION-20260905.md delete mode 100644 docs/design/conversation-runtime/COMMUNICATION.md delete mode 100644 docs/design/conversation-runtime/EXECUTION-PLAN.md delete mode 100644 docs/design/conversation-runtime/HEADER-PANEL-20260907.md delete mode 100644 docs/design/conversation-runtime/HISTORY-SEARCH-EXPERIMENT.md delete mode 100644 docs/design/conversation-runtime/IMPLEMENTATION.md delete mode 100644 docs/design/conversation-runtime/MODULES.md delete mode 100644 docs/design/conversation-runtime/PLAN.md delete mode 100644 docs/design/conversation-runtime/PRODUCT.md delete mode 100644 docs/design/conversation-runtime/PRODUCTION-AUDIT-20260906.md delete mode 100644 docs/design/conversation-runtime/PROMPTS-CONTEXT-TOOLS.md delete mode 100644 docs/design/conversation-runtime/QUALITY-20260905.md delete mode 100644 docs/design/conversation-runtime/REAL-WORK-EVALUATION.md delete mode 100644 docs/design/conversation-runtime/RECURSIVE-CORE-20260905.md delete mode 100644 docs/design/conversation-runtime/TAKEOVER-STATUS-20260905.md delete mode 100644 docs/design/conversation-runtime/TASK-EXPERIENCE-PROTOTYPE.md delete mode 100644 docs/design/conversation-runtime/VALIDATION.md delete mode 100644 docs/design/conversation-runtime/local-change-notes/await-own-command.md delete mode 100644 docs/design/conversation-runtime/local-change-notes/pareto-measurement.md delete mode 100644 docs/design/conversation-runtime/local-change-notes/prefix-cache-credit-bound.md delete mode 100644 docs/design/conversation-runtime/local-change-notes/prompt-shape-composition.md delete mode 100644 docs/design/conversation-runtime/local-change-notes/steer-young-bash-grace.md delete mode 100644 docs/design/conversation-runtime/prototype/README.md delete mode 100644 docs/design/conversation-runtime/prototype/conversation-tasks.html diff --git a/docs/design/conversation-runtime/CALIBRATION-20260905.md b/docs/design/conversation-runtime/CALIBRATION-20260905.md deleted file mode 100644 index c6ac736d98..0000000000 --- a/docs/design/conversation-runtime/CALIBRATION-20260905.md +++ /dev/null @@ -1,106 +0,0 @@ -# Calibration 02 — measured candidate and next decisions - -Candidate `ec9f5a0c2`, local branch `a local working branch`. The full repository check passed before the binary and rig were frozen. This is calibration, not a demonstrated Pareto frontier. - -## Protocol and completeness - -36 planned cells completed: six scenarios × three harnesses × two randomized repetitions. Every inference role was pinned to `deepseek/deepseek-v4-flash-0731`, low requested effort, through an exact-model guard. Native Claude Code Opus was used separately for implementation/review. Each cell used fresh owned state and the same fixture and permissions; runs were sequential. Runtime cap was 180 seconds per cell; interactive fixtures included a 60-second command. - -Executable/package and rig hashes, machine identity, conditions and seeded order are retained in the frozen manifest. Provider routing was each harness’s default stack, not a common pinned backend. Fresh local state does not guarantee a cold provider cache. Two repetitions of one fixture are not two independent kinds of task. - -All 36 attempts are retained. Billing is complete for 35 cells. One timed-out codeaf revision cell has 32 of 35 calls priced; a later read-only reconciliation did not recover the missing prices. Its total remains unknown. No failure was retried away or relabeled. No confidence or equivalence claim is made from n=2. - -## Results - -Each row is one declared slice. Time is mean end-to-end door duration, including launch/quiet confirmation/teardown; it is not the delay to a correct chat reply. Cost is mean billed cost across both attempts, withheld when either is incomplete. Success requires the full objective check set and a successful door exit. - -| Scenario | Harness | Passed | Mean seconds | Mean cost, USD | -|---|---|---:|---:|---:| -| data-tally | codeaf | 2/2 | 6.9 | 0.0008495 | -| data-tally | pi | 2/2 | 5.1 | 0.0002165 | -| data-tally | omp | 2/2 | 5.7 | 0.0004795 | -| research-brief | codeaf | 2/2 | 11.2 | 0.0020309 | -| research-brief | pi | 2/2 | 5.3 | 0.0002885 | -| research-brief | omp | 2/2 | 8.0 | 0.0025340 | -| writing-memo | codeaf | 2/2 | 12.2 | 0.0017029 | -| writing-memo | pi | 2/2 | 10.3 | 0.0005000 | -| writing-memo | omp | 2/2 | 11.0 | 0.0012760 | -| code-fix | codeaf | 2/2 | 12.1 | 0.0020092 | -| code-fix | pi | 2/2 | 8.5 | 0.0003926 | -| code-fix | omp | 2/2 | 12.7 | 0.0013067 | -| followup-while-working | codeaf | 1/2 | 98.9 | 0.0031573 | -| followup-while-working | pi | 0/2 | 81.3 | 0.0001557 | -| followup-while-working | omp | 0/2 | 82.5 | 0.0016784 | -| revision-midwork | codeaf | 1/2 | 160.1 | unknown | -| revision-midwork | pi | 2/2 | 83.1 | 0.0004841 | -| revision-midwork | omp | 2/2 | 82.7 | 0.0014754 | - -All print cases passed. On these observed blocks Pi was faster and cheaper than codeaf on each print slice; codeaf’s mean cost was about 3.4–7.0× Pi’s depending on the slice. This identifies development work; it does not establish population-level dominance. - -## Chat failures and timing - -- Both Pi side-question answers were correct but arrived after the command finished (about 62 seconds after submission). Both OMP runs missed the exact answer. codeaf passed one of two: the passing answer was first observed after 5.293 seconds while work continued. The other answer was wrong. These outcomes cannot be pooled into an acceptable conversational reliability claim. -- In codeaf the new grace handoff consumed steering after 1.550 seconds in the first run and 1.133 seconds in the second. A prior diagnostic waited 30.621 seconds. These are trace observations with a deterministic regression behind the fix, not a controlled latency effect estimate. The ~3-second rule bounds handoff of already-running bash; other tools, provider latency and display polling still affect visible response. -- Both Pi and OMP revision cases passed, taking about 83 seconds. codeaf took 181.612 seconds and timed out once, then passed in 138.608 seconds. In the timed-out case the final CSV was correct and Markdown absent, but the task/completion flow did not settle inside the cap. It remains a timeout, not a pass based only on the files. -- The timeout trace opened task 2 after the main turn had already written the CSV and removed Markdown. Its brief chiefly waited for the existing job and checked those artifacts. The task ran for about 146 seconds and went through further review of whether the script’s own stdout proved completion. This is evidence of costly unnecessary handoff and repeated checking, not evidence that more workers are needed. - -## What to change next, in order - -1. **Stop converting small finishing work into new tasks.** Inspect the remaining-work handoff boundary, not just its counters. The current write allowance sees both an old and revised filename in the same turn, and can move a turn whose useful edits are already complete. Awaiting an existing job or reporting its result belongs to its existing owner. Add real-turn regressions for a revised small deliverable and for a genuinely long editing job that must leave chat available. Do not merely raise the threshold until this fixture passes. -2. **Reduce the rendered prompt/tool contract before adding discovery machinery.** One research trace sent roughly 15k input tokens and 31 tools per main call. Compare concise equivalent definitions against the current contract on held-out small and task workloads, keeping authority, permissions, required evidence and capability unchanged. Record actual request bytes, billed tokens, cache hits, calls and correct outcomes. A shorter string is not automatically a quality-preserving improvement. -3. **Measure the corrected cache accounting and routing policy.** The post-calibration change below bounds credit by the actually observed input length and keeps each request’s lineage with its own settlement. Its performance effect is unmeasured. Test history-rewrite invalidation and absent versus reported-zero cache usage separately. A healthy-lane release after a reported miss and soft provider fallback are policies to test, not proven bugs from one trace. -4. **Keep optional post-answer work off the response path without losing continuity.** Title and memory work caused about three seconds of one print run’s exit tail. This is lower priority for chat because its answer had already been emitted. Measure overlap under existing bounded cleanup; do not delete memory just to make a print benchmark faster. -5. **Calibrate meaningful parallel and nested work.** The reviewed `multi-defect-pipeline` fixture is available by explicit selection; it is a correctness fixture, not proof of useful parallel speedup, and has not been compared live. Substantial held-out coding and noncoding work still needs a serial codeaf ablation. Test actual child-room steering, parent integration of a new revision, publication races, long tool output/compaction, cancellation and restart. -6. **Confirm only after quality gates improve.** Freeze candidate, workloads and practical margins; run repeated paired blocks across time windows and held-out variants. Include all failed-attempt costs. Broader writing/research quality and human intervention burden require blinded human rubrics. Do not buy a large confirmation run while the small chat gates are failing. - -## Validation and evidence - -Full `make check`: **PASS**, `candidate-make-check-03.log`, with the unchanged known-red ledger. Session tests 180.440s; terminal tests 465.761s; verification tests 3.170s; packed manual 1.486s. Binary 50,518,578 bytes under the unchanged 54,600,000-byte budget. The tested checkout stayed unchanged during the run. The preserved unrelated `internal/.DS_Store` modification accounts for the binary’s dirty marker. - -Python measurement regressions: 64 passing tests before the campaign; offline runner: 102 checks passed. Steering race tests reproduced the young-command delay, replaced-timer callback, and stop/detach race before their fixes; the final focused lifecycle set passed with the race detector. - -Evidence root: `/private/tmp/af-conversation-ops/pareto-campaign/`. Frozen manifest: `calibration-02.json`. Raw results and paired descriptive report: `calibration-02/results.jsonl` and `calibration-02/summary.txt`. Root-reviewed routing analysis: `opus/ROUTING-CACHE.md`. Late recovery ledger: `calibration-02/018-revision-midwork-codeaf/evidence/revision-midwork-codeaf/guard-reconciled-late.jsonl`. - -All work remains local. Shared `dev` and the original consolidated QA checkout were not modified. - - -## Changes and validation after the measured run - -Source `3cb79f3f8` adds two corrections after the frozen calibration: - -- Cache estimates cannot credit more input than a provider actually reported receiving for the remembered conversation. Newly appended context is no longer credited merely because the conversation identity matches. Missing lengths earn no credit. This remains an estimate: compaction, rewriting and eviction can invalidate a prefix. -- Provider settlement carries a typed observation containing its own conversation identity and reported prompt/cache counts. The shared last-request-per-model map was removed. Previously overlapping requests could credit A’s result to B’s conversation. A barrier-based stub test forces out-of-order completion, fails with the old attribution, and passes in both streamed and whole-body paths; five repetitions with the race detector passed. - -A later correction addresses recommendation 1 above at the handoff boundary itself, not at -its counters. A running turn that answers the handover ask with `NOTHING LEFT TO DO` is now -dropped unless the second reader's sketch names independent parts still to do; a reading that -was absent, faulted or drew no parts no longer stands in for a reader saying work remains. -The accepted drop is charged once per request — including when the sketch agreed, because -both readers can be wrong together — and the ceiling returns after `checkpointPrice` further -worked rounds with the claim spent. A direction typed mid-turn changes the request and so -arrives with the claim unspent. Separately, a handover whose owning context is already -cancelled now admits nothing: the frozen `handoff-after/001` cell's only error line was -`role:markreader message:context canceled`, which was read as a reader outage, and -`launchRouteTask` takes no context, so the check is at the seam. Package regressions in -`internal/session/completion_stale_test.go` cover each of these, including the running-command -case driven through the real loop. - -**Limitations of that correction.** It does not address a sketch that names parts which have -already been done: where the reader draws independent parts, the work still moves, and the -harness has no way to distinguish a part genuinely left from one that landed after the reader -was shown its account. Only the opposite mistake — silence read as remainder — is closed. -There is **no live performance result** for it; the timeout verdict for cell -`018-revision-midwork-codeaf` and the measured table above are unchanged. - -The optional `multi-defect-pipeline` scenario contains four small independent defects and a combined CLI. Its 29 external cases compare input/output/status in a separate parent process. The original judge was rejected after unrepaired code rewrote its in-memory tests into no-ops; the committed regression preserves that counterexample. Further regressions cover import hangs, descendants that ignore termination, exited parents, and judge termination. Process-group cleanup is best effort on POSIX; a deliberately detached descendant is outside that guarantee. The fixture directory is not a filesystem sandbox, and it makes no claim about substantial parallel work or internal API coverage beyond the CLI. - -Final combined validation on `3cb79f3f8`: - -- `make check`: **PASS**, `final-make-check.log`. Session 180.339s; provider 55.734s; lane 7.236s; terminal 465.866s; verification 3.167s; packed manual 1.481s. Binary 50,518,274 bytes, below the unchanged 54,600,000-byte budget. -- Python benchmark suite: **92 passed**, `final-python-tests.log` (48.188s). -- Offline benchmark runner: **112 passed, zero failed or skipped**, `final-rig-selftest.log`. -- The full provider package under `-race` is **not green**: `TestTheWarmTranscriptEncodeCostsTheSameAtEightyTurnsAsAtEight/breakpoints` fails allocation-count equality. The root reproduced that same failure on unchanged `389ab7594` in `cache-race-baseline.log`; normal full checks and the targeted new race tests pass. No known-red entry or test skip was added. - -The six revision artifacts were also parsed independently after the campaign: every header and row was exact, every superseded Markdown file was absent, and each prepared command ran once. `calibration-02/revision-artifact-adjudication.json` retains those checks. The timeout verdict remains unchanged because its completion flow exceeded the cap. - -All delegate changes were reviewed before local merge; the owned retired worktrees and their binaries were removed. The integration binary remains `/private/tmp/af-conversation/bin/codeaf`. These follow-on changes have **no live performance result yet**. The measured table above remains attached to `ec9f5a0c2`; documentation-only commits after final validation do not change that distinction. diff --git a/docs/design/conversation-runtime/COMMUNICATION.md b/docs/design/conversation-runtime/COMMUNICATION.md deleted file mode 100644 index 49af465509..0000000000 --- a/docs/design/conversation-runtime/COMMUNICATION.md +++ /dev/null @@ -1,48 +0,0 @@ -# Communication ownership and extension boundary - -This is the implementation target, not a claim about current behavior. The local wave remains on a working branch. Live model tests use DeepSeek V4 Flash through OpenRouter, pinned across roles. Commercial OpenRouter models are excluded. - -## One delivery mechanism, distinct meanings - -The existing `userMessage`, `enqueueNote`, `Steer`, `SteerTask`, `deliverTaskNote`, and `postTaskNews` already share part of the delivery path. Extract and consolidate that path before inventing a message bus. A task is an addressable conversation with an assignment and an owner. Main chat has the same conversation delivery mechanics, but different responsibilities. Keep task scheduling, assignment evaluation, source control and terminal drawing outside the mailbox. - -A common envelope needs stable message ID, source and recipient conversation identities, origin (authenticated user, agent, tool, runtime), kind, body or content reference, optional correlation/reply ID, and applicable assignment revision. A task number alone is not globally unique; qualify it by session. A content hash identifies content, not an event: the user repeating an instruction after a correction is a new event. - -Kinds express a small set of real differences: - -- User direction: authentic source, explicit scope, acknowledged receipt and read state. It may update the assignment revision. A quoted user line is not an authenticated grant. -- Work request: a scoped assignment through admission, not arbitrary remote execution. -- Result: a durable answer and evidence references, linked to the assignment revision; preview text is separate. -- Question and answer: correlated and owned, with a named blocked dependency. A child question must reach and wake its owner. -- Progress: a concise replaceable observation for people; it does not wake a model or become an established fact. -- Runtime notice: a typed state change or actionable failure; it is not a person speaking. - -Do not expose this envelope vocabulary to the user. The surface says that the correction arrived, work needs an answer, or a result is ready. - -## Delivery and wake-up are one boundary - -Append to the recipient's queue and signal the eligible reader as one operation. Separate calls that append and later wake leave interleavings where the reader wakes to an empty queue or sleeps over an unread message. A closed or unknown recipient is an explicit outcome; accepted does not mean read. A receipt distinguishes accepted, read, superseded and rejected. Persist pending accepted directions needed after reconnect before acknowledging them as durable. - -The recipient owns wake policy. A main conversation can begin a turn for owed results. A task has one worker runner; delivering a message signals that runner rather than starting a competing loop. A child result or clarification answer can release the corresponding wait. A user direction is considered at a legal model/tool boundary. Ambient progress is coalesced without inference. Siblings with independent work keep going while one dependency is blocked. - -Persist delivery IDs so retries can be deduplicated. Do not promise exactly-once external effects: a network send may succeed before its receipt is stored. Consequential tools need their own idempotency or reconciliation. Message retry alone cannot make them safe. - -## Assignment revision is separate from message delivery - -A running task must be able to receive a correction while working or finishing. A receipt must never claim a correction was applied if nobody will read it. Keep the original request and ordered user changes; workers and checks share the current revision. Finalization compares the revision under the task's ownership lock before accepting a result. A result of an older revision is retained as previous work, not presented as satisfying the current request. - -Do not classify every message with a separate model. Route using the current room and explicit tool target; use the existing conversation reasoning to resolve ambiguous scope. An ordinary question about progress must not silently reset acceptance. Changes to scope, permissions, and side effects follow authenticated user intent and existing policy. No descendant can increase its own authority. - -## Context is a view over sources - -Admission composes the current assignment, applicable user constraints, selected quoted exchanges, accessible evidence references, and relevant dependency results. These are distinct inputs. A recent-quote window does not replace durable constraints; dropping an old quote must not revoke a still-applicable restriction. A source reference is useful only if the recipient can retrieve it within its permissions. Test retrieval through the actual worker tools, not only formatting. - -Compaction may summarize older discussion and reduce successful tool output while retaining source references. It must preserve unresolved questions, current user changes, tool failures that still matter, and unfinished effects. Do not reclassify assistant narration as fact or silently make unavailable evidence appear accessible. - -## Future cross-session support - -Keep recipient identity and delivery interface independent of in-memory pointers. Start with a local implementation behind existing calls; a future router can resolve a permitted session and use the same delivery contract. It will additionally require authenticated cross-session authority, recipient admission policy, replay protection and source-access checks. None of that is implemented or implied by today's extraction. Do not add global discovery, remote fan-out, a service registry or a distributed event store in this wave. - -## Verification boundaries - -Exercise duplicate delivery, close/enqueue races, child-result and question wake-up while the parent is parked, progress without model calls, correction versus finalization, queued direction after restart, and inaccessible source references. Preserve old checkpoint loading and the real terminal door. The success criterion is correct work and understandable receipts with less duplicated logic; moving code into more files alone is not a successful refactor. diff --git a/docs/design/conversation-runtime/EXECUTION-PLAN.md b/docs/design/conversation-runtime/EXECUTION-PLAN.md deleted file mode 100644 index 59df36c5ae..0000000000 --- a/docs/design/conversation-runtime/EXECUTION-PLAN.md +++ /dev/null @@ -1,211 +0,0 @@ -# One conversation, several execution scopes - -Working plan, 2026-09-05. This is a proposal and implementation sequence, not a -description of capabilities already shipped. The draft PR carries the earlier -conversation-runtime work as a baseline; its presence is not evidence that every -choice in that baseline is correct. - -The [recursive-core note](RECURSIVE-CORE-20260905.md) narrows this plan after the -user asked whether the system can be one agent loop spawning instances of itself. -It inventories existing duplicate decisions and the behavior that must survive -their removal. - -## Decision - -Keep one owner for each requested outcome. Let that owner choose direct work, -short parallel lanes, or a durable task according to the independence and lifetime -of the work. Do not infer a new outcome owner from the number of files already -edited. Do not create a task merely to wait for work an existing owner started. - -The main conversation remains useful for ideation, decisions and concise results. -Substantial execution belongs in a work conversation that the person can enter and -steer. A short fork is an execution detail of its caller, not another assignment -that needs a name, plan, admission debate, independent reviewer and landing. - -This is not a new scheduler alongside the current ones. `fork` already exists; -tasks, job receipts, the mailbox and result projection already exist. The work is -to make their boundaries coherent and remove redundant decisions. - -## What the evidence establishes - -The frozen calibration in `CALIBRATION-20260905.md` contains 36 attempts and only -two repetitions per cell. codeaf's four small standalone cases all passed, but -their mean billed cost was approximately 3.4–7 times Pi's. Interactive follow-up -and revision each passed once in two attempts. That is diagnostic evidence, not a -frontier result. - -In the revision timeout, the requested CSV already existed, the old Markdown was -gone and the original command ran once. Automatic handoff then created work mainly -to wait/check, and review continued past the deadline. Correct files alone do not -turn that timeout into success. The defect spans ownership, observation and -completion; changing the file-count threshold cannot establish a sound contract. - -Existing fork code checks declared scopes within a batch and runs hands in the -caller's directory. It delivers results asynchronously through job notifications. -The comments alternate between a join and continued builds while siblings write. -A shared workspace needs an explicit consistency rule; describing disjoint writers -does not prove that a parent build reads a stable tree. Shared prompt prefixes also -do not guarantee provider cache hits, especially when tools or system tails differ. - -## The three execution choices - -| Choice | Ownership and context | Completion and interaction | -| --- | --- | --- | -| Direct | Current owner, current context; small actions and synthesis | Return the answer or retain an explicit outstanding operation | -| Short fork | Same owner; immutable inherited history plus a narrow lane instruction and actual capabilities | Bounded attributed results return to caller; caller integrates and checks the combined output | -| Durable task | New responsible owner; current user contract, selected evidence and retrievable sources | Independent work conversation, revisions, restart/reconnect and nested delegation | - -The number of lanes is a ceiling, not a target. Serial execution is preferable -when setup, repeated context or integration costs exceed independent useful work. -Two simultaneous tool calls need no new model lanes. Four independent edits may -benefit from a fork; a coupled migration may benefit from one sustained worker. - -A fork is unsuitable when it needs a separate lifecycle, broad shell access, -conflicting writes or lengthy investigation. A task is unsuitable when all that -remains is observing an operation whose receipt belongs to the caller. Promotion -must transfer a concrete remainder and accessible dependencies, rather than -rephrasing the entire original request after part of it is already done. - -## Six boundaries, with one writer for each fact - -1. **Assignment:** the person's request, effective revisions, acceptance and owner. - Agent coordination cannot rewrite user authority. An answer to a question is - not automatically a revision to every active task. -2. **Execution:** attempts, processes, fork groups, cancellation and limits. These - records establish what ran; they do not judge whether an artifact is useful. -3. **Communication:** addressed events with source, request/revision correlation, - receipt and consumption. Tasks and sessions use the same local delivery seam; - future cross-session transport can resolve addresses without redefining content. -4. **Context:** compile the provider view from the current contract, role/tool - capabilities, relevant evidence and bounded working history. Preserve source - handles and call/result pairing; never mutate the authoritative journal to save - prompt bytes. -5. **Outcome:** the owner accepts or reports a result against its current revision. - A process exit is execution evidence; an explicit test result is check evidence; - neither is a universal proof of quality. No recursive review merely to validate - another review's wording. -6. **Presentation:** project these facts into running, finishing, done, incomplete - and needs your look. Opening a work conversation does not start another writer. - Routine activity updates its work item; meaningful changes and results appear - in conversation without stealing focus or overwriting a draft. - -Start with cohesive files and narrow interfaces inside the existing session -package. Extract packages only when dependencies form a real boundary. A generic -bus, universal graph or parallel replacement engine would increase the migration -surface before proving the product improvement. - -## Fork semantics to settle before expanding its use - -- A fork gets one immutable context snapshot. Each lane receives its difference, - sibling responsibilities, current assignment revision and its own tool belt. - Inherited task promises must not contradict the narrower belt. Full-history and - selected-history variants are measured; cached input is never assumed free. -- Read-only lanes must not invent writable files merely to satisfy the schema. - A read-only lane has no write capability. Editing lanes declare disjoint scopes. -- Scope admission must account for other live batches and the caller, not merely - sibling declarations in one invocation. Canonical path aliases and symlinks need - the same treatment at admission and execution. A prompt rule is not isolation. -- Results can stream individually, but combined builds and acceptance require a - stable relevant workspace. Do useful independent reading while siblings write; - do not present a build over a changing tree as final evidence. -- Preserve partial results, failures and cost on cancellation. A person's side - question should not silently discard useful lanes. A change in scope must reach - affected work or stop it, and results must carry the revision they used. -- Bound aggregate active lanes and reserve shared admission capacity. Per-lane - round limits alone do not bound total cost. Initially describe money bounds as - soft wherever in-flight provider billing can overshoot. -- The caller owns the final answer. A lane may finish while other lanes continue; - that is progress, not completion of the requested outcome. - -## Handoff and completion simplification - -Replace overlapping policy decisions incrementally behind one factual snapshot: -current request/revision, outstanding owned operations, unread results, remaining -work and available budget. Start with a pure decision boundary that can be tested -without providers. Do not hold agent/graph locks while asking a provider or calling -registry callbacks. - -The same decision must cover task children, fork hands and commands without -pretending they have identical lifetimes. An already running command retains its -execution owner and has an observable result. Waiting consumes no model turns. -New independent work can still be delegated while that command runs; the presence -of a job must not become a blanket exemption from delegation or finishing limits. - -The capable model should normally request a task or fork directly with a concise -contract. A runtime governor may interrupt runaway work, but should not pay for -multiple readers to reconstruct the same decision. Remove old routing readers -only after scripted and live tests establish equivalent authorization, interruption -and completion behavior. Reduced calls are useful only if success does not fall. - -## Evaluation that can support a Pareto claim - -Use three suites: deterministic runtime contracts, controlled diagnostic tasks, -and frozen real GitHub issues. Keep coding, research, data and writing as separate -quality strata. Real coding issues need actual repository setup, investigation, -multi-file changes, dependency interaction and independent acceptance. - -Each issue has immutable provenance, a pre-fix source SHA, environment/toolchain -definition, model-visible issue text and acceptance instructions, and private -evaluation material. Evaluate a clean base failure and reference success before -calling the case ready. Candidate workspaces must not contain solution commits, -future issue comments, gold patches or hidden tests. Known historical issues may -still be in model training; disclose that limitation and include held-out recent -issues. Never select only cases where our preferred decomposition wins. - -Pair codeaf, Pi and OMP on the same base, model, reasoning setting, tool access, -provider-routing condition, deadline and task data. Pin every inference role to -`deepseek/deepseek-v4-flash-0731`. Use native Claude Code Opus for development only. -Freeze binaries, adapter versions and manifests. Preserve failures, unknown bills, -retries and incomplete outcomes. Do not replay a request to recover billing. - -Run actual chat as well as headless controls. Chat variants must include a question -while work continues, a revised requirement, stop, entering/leaving a task and -reconnect. Measure correct-answer latency, time until direction is consumed, stale -work after revision, duplicate actions and usable-result time. A responsive -composer without a correct timely answer is not successful interaction. - -Measure total billed cost including routing, naming, checking and recovery; wall -time to accepted usable output; independently checked quality; failure rate; and -cost per successful issue including failed attempts. Report distributions and -paired uncertainty, not only means. Provider cache and backend differences are -conditions, not hidden harness improvements. Judge costs and development expenses -are separately reported. - -Use at least five paired repetitions for diagnostic estimates, then a broader -predeclared issue set with held-out cases and a precision-driven sample size. -Five repetitions alone are not strong evidence of quality equivalence. Compare -serial codeaf, existing fork and revised execution as ablations. Claim improvement -only within the measured task/model/door/conditions. There is no defensible global -Pareto claim over every possible task. - -## Implementation order and stop conditions - -1. Publish this plan and inherited work in a **draft** PR. Keep the shared checkout - untouched. Inventory active PR overlap; do not silently absorb other lanes. -2. Repair the existing short-fork contract and establish real-issue preflight with - offline proofs. No new scheduler, task state family or automatic planner. -3. Consolidate operation ownership and handoff decisions. Reproduce the revision - failure with a scripted provider before changing policy, then run its frozen - live diagnostic. Preserve substantial-work delegation and side-question control. -4. Reduce prompt/tool contract overhead and redundant readers one source at a time. - Compare success and total billing with paired runs; revert losing variants. -5. Add global admission reservations and finish steering/cancellation/reconnect - coverage. Exercise overlap and failure paths under the race detector. -6. Run curated real-issue and noncoding campaigns through chat and headless doors. - Improve the dominant measured failure/cost next; repeat on held-out cases. - -Keep the PR draft until architecture, manual, meaningful tests and comparative -evidence agree. Do not claim the whole plan is implemented when an individual -module lands. If an abstraction adds decisions or model calls without removing -existing ones, justify it against an observed failure or do not add it. - -## Self-critique - -“One owner” does not remove dependency complexity; it gives that complexity a -place to live. Copying context can be cheaper than writing briefs but expensive on -cache misses and misleading when it carries obsolete constraints. Shared files -save worktree overhead but require stronger consistency rules. A universal -completion state would erase distinctions between a stopped command, a failed -check and an unfinished artifact. Trusting a capable model reduces ceremony but -does not replace runtime receipts or cancellation. The acceptance of this design -is measured useful output with dependable steering, not fewer types or more lanes. diff --git a/docs/design/conversation-runtime/HEADER-PANEL-20260907.md b/docs/design/conversation-runtime/HEADER-PANEL-20260907.md deleted file mode 100644 index 8dc64b4c64..0000000000 --- a/docs/design/conversation-runtime/HEADER-PANEL-20260907.md +++ /dev/null @@ -1,114 +0,0 @@ -# codeaf navigation panel — September 7, 2026 - -Design direction for the header refinement requested after trying revision -`0e095a9b1`. This document states the interaction contract; test receipts and -implementation status belong in the production audit. - -## Purpose and hierarchy - -The terminal is a place to operate and read. Its header should answer three -questions in order: which conversation, where inside its work, and what that -work is doing. The transcript remains the largest and highest-contrast reading -area. Preserve the existing restrained palette and terminal character grid. - -The chat tabs are one group, the ancestry is a second, and execution facts are -secondary. A thin horizontal rule separates this navigation region from the -transcript and right-hand task list. A heavy surrounding border or a second -app-wide background is unnecessary. - -Main chat: tab row, then separator. Task page: tab row, breadcrumb row, then -quiet status/metadata on the separator row. Model, cost and tool counts never -become breadcrumb segments. Only the current task receives primary emphasis; -known ancestors remain recognizable, clickable navigation targets. - -## Four actions with distinct meanings - -| Action | Meaning | Must preserve | -| --- | --- | --- | -| `+` New chat | Open a start page with a blank composer and recent chats. Create the conversation on first send. | The current conversation and its draft; no execution merely from opening or cancelling the page. | -| Chats | Find and open an existing conversation through the established switcher. | Stable identities, readable titles, explicit confirmation and existing ownership checks. | -| Tab `×` / Ctrl+w in Chats | Dismiss this view; keep the conversation discoverable in Chats. | Unsent text, caret, compact pastes, attachments and running work. | -| Stop | Explicitly stop the named work under the existing stop contract. | Retained artifacts and clear scope. | - -The distinction between dismissing and stopping is substantive. A tab close -must not call an agent's Close or Interrupt method. Closing an inactive tab -must not select it first. Closing the last tab goes Home. With a shared engine -connection, dismissing the current tab goes Home without selecting another -conversation, because the current server still ends the previous conversation -on selection. This design does not claim server-side multiplexing exists. - -Reopening a dismissed conversation restores its tab and owned draft. A redraw, -resize or new background event must not resurrect a deliberately dismissed tab. - -## New-chat start page - -Use the smallest existing start/composer machinery that can preserve recipient -ownership. Opening the page is a reversible navigation action, not a model call -or an empty session factory. Repeated presses reuse that page. Escape returns -to the same task/chat and draft. Choosing a recent chat resumes that chat. -Submitting a first message uses the existing creation and delivery contracts; -a failure retains the unsent message in the start page, never sends it into the -old chat. Files and compact pastes belong to the draft that received them. - -Show a short list of recent chats with recognizable titles and enough project -context to distinguish duplicates. Defer predicted frequent items, favorites, -pinning and custom tab icons: each adds a separate organization concept and -needs evidence that recency and search are insufficient. - -## Pointer and keyboard behavior - -Tab labels have padded targets. Subtle separators are inert. Every tab responds -to hover, including the selected one; hover must differ from persistent -selection and must not move or resize labels. Reserve the close-control cells. -The close target highlights separately and consumes its own click. Moving the -pointer away restores the selected state, not an unselected default. - -Chats is a labeled action at the right of the panel. The plus is a separate target at the right of the -header and means New chat, not overflow. Narrow layouts may use shorter words -and fold inactive tabs, but retain the active title and useful navigation. - -Ctrl+w keeps its familiar word-delete behavior in the composer. In the Chats -picker it dismisses the selected tab. - -Ctrl+k remains a preview until Enter, an explicit numbered selection, or a row -click. Escape cancels. Ordinary terminal protocols cannot reliably report Ctrl -release, so elapsed time never selects a conversation on the person's behalf. - -Breadcrumb hover uses the same target boundaries as breadcrumb clicks. Root and -ancestors navigate; the current item remains the current item. A folded path -opens its nearest hidden ancestor. Guest or missing-owner data cannot invent a -navigation destination. - -## Status and motion - -Use one stable status slot only when the runtime can support its meaning: -needs the person's reply outranks working; idle has no decorative badge. -Internal checking, dependency waits and provider retries are not requests for a -person's response. A completed-but-unread indicator is useful only if it comes -from an actual unread receipt and clears when that conversation is read. -Unknown status stays absent. Never infer that a historical visited tab is still -running, or query a remote task index on every frame to animate a badge. - -Click selection is immediate. Hover is immediate and confined to the target. -No sliding/reordering tabs, opacity animation on text, repeated pulses or new -frame clock for idle navigation. Existing work animation can continue within a -fixed cell where it represents actual activity. No-color selection and text -state remain understandable without relying on hue or motion. - -## Adaptation and validation - -Use a single header-height and row-coordinate contract for draw, hover, clicks, -text selection, body scrolling, sidebar and task controls. At short heights, -remove secondary chrome before sacrificing the composer and usable transcript. -At narrow widths, shorten metadata and fold ancestors before erasing the current -location. Long and same-named chats must retain their distinct destinations. - -Validate main/task/new-chat, idle/working/needs-reply, local/shared, current/other/ -last-tab close, failed create, repeated open, draft restoration and explicit -picker commit. Inspect real terminal frames together at 160, 80 and 60 columns, -plus no-color and short-height checks. Use one correction batch and at most one -visual confirmation pass; functional regressions must still be fixed and tested. - -## Verification refinements - -The final start-page footer names New chat and hides the previous conversation's telemetry. Task metadata reserves room for a visible divider even when a dependency has a long name. The task placeholder targets the actual draft row, preserving an attachment/effort tray above it. A guest task is reacquired from its exact owner on Escape; an unavailable connection uses the existing task-card refusal. An unnamed old conversation with unsent words blocks first-submit creation until the person returns to that draft, rather than transferring it to a different recipient. diff --git a/docs/design/conversation-runtime/HISTORY-SEARCH-EXPERIMENT.md b/docs/design/conversation-runtime/HISTORY-SEARCH-EXPERIMENT.md deleted file mode 100644 index afa4f0d280..0000000000 --- a/docs/design/conversation-runtime/HISTORY-SEARCH-EXPERIMENT.md +++ /dev/null @@ -1,175 +0,0 @@ -# Conversation search and source reads - -This change gives foreground chat, task workers, nested workers, forked hands and -task checkers the same read-only access to indexed conversation history. It fixes -an access gap as well as a retrieval problem: workers previously inherited no -writable memory store, so their toolbelt omitted conversation search altogether. - -## Observed failure - -A local user asked for a recent sandbox-browser task. The saved exchange showed -four task lookups, followed by two conversation searches, a transcript read and -two filesystem calls. The search ranked the requesting conversation's own fresh -tool-call text first. The model eventually found the conversation, but discovering -its location required knowledge of the on-disk layout. We did not locate the exact -reported worker refusal; the missing worker tool was established from its actual -constructor and toolbelt instead. - -The experiment below uses synthetic fixtures, not the user's private history. - -## Retrieval design - -`search_conversations` has three operations in one schema: - -- `query` searches message text across the store with SQLite FTS5 BM25; optional - `session_id` restricts it to one conversation. -- `session_id` alone browses recent messages in that conversation. -- `ref` alone opens the indexed message and up to two actual neighbours on each - side. References are returned for both matches and neighbouring messages. - -The existing index avoids another service, embedding model, file crawler or -fuzzy-search executable. Query words are safely quoted Unicode tokens, combined -with OR for candidate recall. BM25 ranks candidates and journal order breaks -ties. Match-centred excerpts preserve the relevant passage in a long message. -Surrounding messages expose nearby corrections; source references allow a larger -read when the excerpt is insufficient. Neither ranking nor a saved memory is -treated as proof of the answer. - -Broad search excludes the requesting agent's own thread to reduce self-matches. -Explicit scope can still open it, and workers can search their parent's history. -The inherited interface provides reads only; it does not enable memory writes, -background extraction or indexing of worker traffic. Missing readers leave the -tool absent. Database errors remain distinguishable from no matches. - -Search defaults to eight hits, caps at twenty, and accepts at most thirty-two -query words. Excerpts and neighbouring messages cap at 400 bytes. An explicit -read preserves the full indexed anchor, including line breaks, up to the existing -16 KiB message limit. Results distinguish full text from excerpts, state the -indexed-history scope, and include conversation IDs, message IDs, dates and stored -roles. References are stable locators, not authorization tokens or transcript -line numbers. Historical text is identified as evidence, not instructions. - -## Live experiment - -Run the same tests from a clean checkout with `OPENROUTER_API_KEY` available: - -```sh -make build -go test -tags e2e -count=1 -timeout 15m -v -run '^TestConversationSearch' ./internal/e2e -``` - -The harness creates a disposable home and pins every text-model role, tier and -fallback to `deepseek/deepseek-v4-flash`. Foreground cases use the full chat -toolbelt; the worker case enters through `Agent.StartTask`, runs its normal task -loop and enables the task checker. These are live engine tests, not a claim that -the tmux terminal suite was run. - -An initial successful run on 2026-09-10 passed in 161 seconds: - -| Case | Assertion | History calls | -| --- | --- | --- | -| Correction in a long message | MAPLE-92 supersedes CEDAR-81 | 1 | -| Another project | Retrieves violet-kestrel-47 across project folders | 1 | -| Ambiguous task-or-chat request | Finds original conversation after task lookup | 1 | -| Open a supplied source reference | Reads the exchange and retains the correction | 1 | -| Missing phrase | Reports no indexed match without inventing an answer | 1 | -| Actual checked task | Writes exact phrase, stored role, conversation/message IDs and source reference to decision.json; finishes done | 4 | - -The five foreground cases rejected repeated identical searches and filesystem -history detours. They shared memory state deliberately: a remembered answer must -not replace locating the requested original conversation. The task artifact was -validated against the fixture's exact source record, and the model ledger was -checked to contain only the requested model. That run's recorded cost was -$0.012838 for chat and $0.005882 for the task. This excludes earlier iterations. - -## What the iterations revealed - -Early reads clipped long anchors and caused repeated retrieval. Numeric read -arguments were mistaken for per-conversation positions; one sparse-history task -searched the filesystem and timed out. Opaque references, full indexed anchors, -explicit history boundaries and stored roles improved the contract. A shared-memory -case answered from memory without locating its source, prompting the source-first -tool guidance and task-miss hint. A checked task found the right evidence but -failed an ambiguous report-format requirement, so the final fixture uses a -precisely specified JSON artifact. - -That task family still decoded and altered source references unnecessarily: after -searching and opening the real message, it tried earlier global IDs, including -zero. Those calls produced no fabricated evidence, and the final artifact was -correct, but this is a remaining tool-selection inefficiency. Opaque references -discourage arithmetic; they do not prevent a model from doing it. Task-family call -count was therefore reported rather than hidden behind an arbitrary three-call -correctness gate. - -Repeating the experiment exposed an additional completion failure: a task wrote -the exact correct artifact, but its checker rejected the source metadata because -it could not establish which fields the person had requested. The captured -checker packet contained only the fallback acceptance, "Complete the brief and -report the result and checks run.", without the referenced brief. It tried to -reconstruct that missing request from the source conversation. Another run failed -the artifact assertion before these diagnostics were available, so its exact -cause remains unresolved. A subsequent diagnostic run passed with two history -calls; that variability is why the missing contract was fixed instead of simply -rerunning until green. - -The checker now receives the admitted brief only when the current acceptance is -that exact fallback. Explicit acceptance criteria retain the existing contract-only -behavior. A regression test covers both cases. This fixes missing input to the -check rather than weakening artifact assertions or disabling the checker. - -Two file-delivery repeats after that fix both recovered the exact source-backed -JSON. One passed; one was rejected because the checker confused the restored -workspace's temporary name with the worker's original root folder. Its own -reasoning explicitly confirmed all five source fields before rejecting the path -claim. That broader file-completion issue remains unresolved. The earlier -file-delivery scenario is reproducible from commit `ce81ac14f` using the live -command above; it predates the fallback-contract fix and can expose either failure. - -The maintained task test now asks for the person's actual operation: return the -earlier approval phrase, stored role, conversation ID and source reference in the -task's final answer. It checks the worker's final message, not a tool result or -the checker's paraphrase, and still requires the task to finish done with checking -enabled. It does not claim coverage of JSON-file delivery or that the broader -completion issues have been solved. - -A repeat of the direct-answer scenario also exposed a checker that confirmed the -answer but demanded a file despite the requested final response. The checker -packet now explicitly identifies the retained final response as a valid -deliverable when requested, while requiring evidence for its claims and retaining -file checks when a file is required or claimed. Worker and checker history calls -are logged separately: checker reads are useful independent checks, not worker -redundancy. Some checks rely on the supplied source receipts without issuing a -fresh lookup; the test requires worker retrieval, a grounded worker answer and -successful completion, and reports checker calls rather than mandating them. - -The PR's first complete CI run also found a manual section-list failure on the -newer merge base. `TestEverySectionTheCutNamesComesBackWhole` fails identically -on clean `dev` commit `be1cc3749`: the tasks page's bounded heading list omits -"Back to main from a nested task — return to the conversation". This is separate -from conversation retrieval and was reproduced in a detached clean worktree. -PR #772 subsequently fixed the list bound on `dev`; that fix is included in -this branch's merge-readiness checks. - -The final feature run on `bda5e3d9e` passed all six maintained scenarios in -73.5 seconds. Each foreground case used one history lookup. The task worker -used two lookups and its checker independently used two, returned the exact -cited answer, and finished done. Recorded cost was $0.002999 for chat and -$0.002671 for the task. The build, quick checks and fresh affected-package -suites also passed on that commit. These results do not erase the earlier -file-delivery findings or extend the coverage beyond the maintained scenarios. - -## Limits and follow-up criteria - -This is lexical message search. Titles, unindexed history, other machines' stores -and spilled-file contents outside the indexed stub are not newly searchable. -Semantic paraphrases with no shared words can miss. Search does not scan every -later message for a distant correction, so a model may need scoped follow-up -queries. The small live fixture set demonstrates the access path and selected -failure modes; it does not establish universal recall or non-regression in answer -quality. - -Add fuzzy or semantic candidates only against a labelled retrieval benchmark -covering typos, paraphrases, false positives, distant corrections and large stores. -Compare recall, grounded-answer accuracy, latency, context size and total tool -calls. Regardless of candidate generation, answers should remain tied to original -messages through the same read contract. diff --git a/docs/design/conversation-runtime/IMPLEMENTATION.md b/docs/design/conversation-runtime/IMPLEMENTATION.md deleted file mode 100644 index 3ace6ef8ac..0000000000 --- a/docs/design/conversation-runtime/IMPLEMENTATION.md +++ /dev/null @@ -1,124 +0,0 @@ -# Conversation runtime implementation - -Local integration line: `a local working branch` (2026-09-04 working wave). This is a working record of what has merged into that -line, what is still open, and what the available evidence does and does not establish. It -describes the local integration only; nothing here is published or claimed as released. - -## Merged - -| Boundary | What it does now | Where | -| --- | --- | --- | -| Local delivery | One addressed hand-over (`conversationID`, `messageOrigin`, `messageKind`) returning one receipt: nobody / closed / accepted. Progress lands on the ambient queue and cannot start a paid turn. | `internal/session/mailbox.go` | -| Durable acknowledgement | A sender owed an answer about the recipient's *record* gets one when the recipient's journal holds the line, not when a queue took it. Ids are composed from checkpoint facts so a resume can tell a replay from a second event. | `internal/session/mailbox.go`, `task_store.go` | -| Assignment revisions | The admitted spec is frozen; corrections land as an overlay. Only a person's direction may move the done-condition; an agent's coordination is recorded but cannot re-aim the work. A landing may not publish while a person's direction is unread. | `internal/session/assignment.go`, `assignment_tool.go` | -| Admission context | One bounded, attributed selection per admission door: quotes with speaker and source, tool handles with call/input/outcome including an explicit unknown outcome. Versioned and checkpoint-durable. | `internal/session/admission.go`, `admission_compile.go` | -| Task result | The full answer is kept apart from the compact card, with a bounded excerpt and a retrievable overflow reference; the outcome qualification is retained. | `internal/session/task_result.go` | -| Tool-result compaction | Older frozen tool results are reduced to a bounded head and tail with the exact count of bytes cut and a pointer the read tool can open (`Agent.fullResultPointer`), falling back to the journal and then to an honest "cannot be read" rather than naming a place that does not exist. A pass folds to a headroom target below the threshold instead of re-firing at every step, and the budget walk carries a running total rather than recomputing it. The live transcript, the system prompt and the running turn are never rewritten, and no result is dropped. | `internal/session/toolcompact.go`, `stub.go`, `turnfold.go` | -| Status projection | One pure `ProjectTask` over facts, separating presence, wait-on, change disposition, fault and attention, with three-valued liveness. The surfaces adapt to it instead of each deriving meaning. Wire and checkpoint enums unchanged. | `internal/session/task_status.go`, `internal/tui3/taskstatus.go` | -| Persistent host and truthful detach | `Welcome.Persistent` is the engine's statement about its own lifetime. A window closing (including SIGHUP) detaches; work, standing questions, drafts and parked messages survive. `/close` and an explicit stop remain endings, and the quit hint promises "keeps running" only where it is true. | `internal/remote/client.go`, `internal/tui3/keeper.go`, `quitarm.go` | -| Foreground handoff | The end-of-turn reader asks whether this request's work went to a live task, so a turn that correctly handed work off is not re-opened into polling its own task. Request-scoped, no polling. | `internal/session/turnhandoff.go` | -| Workspace ownership | Alternate spellings of the same physical location resolve to one working copy when binding contract paths and checking parallel writable claims. | `internal/session/taskgit.go` and the ground ladder | -| Evaluation rig | A cross-harness battery with two doors that are never mixed (print and a real TUI in a tmux pane), calibrated screen markers per arm, `unsupported` recorded rather than substituted where no interactive door can be driven, guarded upstream calls pinned to an open model, exact answer checks, and a cost meter that reconciles admitted requests instead of counting missing usage as zero. It spends real money, is on demand, and is outside `make check`; `test/selftest.sh` exercises it with fakes and no network. | `bench/conversation/` (`run.sh`, `lib/`, `scenarios/`, `test/`) | - -Each merge has been rebuilt with `make build`, and each boundary above has behavioral tests -alongside it (`mailbox_test.go`, `assignment_test.go`, `admission_test.go`, -`task_status_test.go`, `task_result_e2e_test.go`, `turnhandoff_test.go`, -`internal/remote/detach_test.go`, `internal/tui3/detachexit_test.go`). Focused tests are not -a substitute for the integrated suite or a live product walkthrough. - -## Additional integrated boundaries - -- **Explicit verification contract.** `checks` declarations at admission and revision are - the only source of executable completion checks, for tasks and the main session. - Worker receipts and acceptance prose are evidence, not permission to repeat an action. - A revision clears previous check authority unless new checks are explicitly supplied. - Unrunnable checks do not become claims that commands passed. -- **Causal result wake.** A completed result, attempt, and reply obligation are captured - together and carried through delivery and journal replay. Completion answers the request - that caused the task, including its effective revision, rather than the latest unrelated - question in the main conversation. -- **Worker scope and capabilities.** Every node worker, including a depth-limited leaf, - receives its role and assigned scope. Revision and delegation instructions are conditional - on the tools actually available. The original request is preserved as overall context, - not silently assigned in full to every child. -- **Main-chat corrections.** `tasks` with `forward: true` sends the real person message - attached to the model request to one selected task. `say` remains agent coordination. - Forwarding shares the room's delivery and assignment receipt path. Source identity allows - retries to be acknowledged without duplicate delivery; newer applied corrections cannot - be overwritten by a delayed forward with an earlier recorded speaking time. - -## Validation still pending - -Final integrated live runs and `make check` are recorded separately below once completed. -Focused tests are evidence for individual boundaries, not for the complete experience. - -## What the live evidence actually shows - -From root's `host-live-02` run, two behaviors were observed to pass: the main conversation -answered an unrelated question in a few seconds while a task ran, with no polling; and a task -continued with the terminal closed, with no interruption marker and the same start time. The -same run produced three defects — a path-token trimming bug (since fixed), the audit replay, and the completion-reader causality. All three now have -code fixes and regression tests. The run's own note stands: **it does not -prove overall task efficiency or all steering behaviors.** - -## Interactive ownership and visible result turns - -The launch carries an explicit `Interactive` fact through local-host construction, -new/resume options and principal selection. Tool approval (`--yolo`) and a budget no -longer imply that an interactive conversation has a frozen first goal. Fixed headless -runs retain their original goal behavior; task assignments retain their own revision -rules. Interactive launch dollar/time limits are enforced at new-turn admission, -including background wakes, separately from goal ownership. They permit work already -in flight to finish and do not implement aggregate reservation across concurrent tasks. - -Hosted sessions subscribe once to runtime wake turns. The existing numbered stream, -replay and fan-out path carries them to attached windows. Hosted turn adoption uses -the same ordered queue as local wakes; a later reply cannot silently reuse the -previous person's turn or disappear while its last events drain. This closes a display -failure that unit-only transport testing did not catch. Background commands and fired watches carry their own outcome reply duty through -batching as well; they do not inherit an unrelated recent question. A visible reply still depends -on successful model narration; retained results and transport delivery are distinct. - -## What is measured, and what is not claimed - -Measured and test-backed, in-tree: compaction bounds a long frozen tool history to a -sub-linear prompt, a pass reaches its headroom target and then leaves the next step alone, -and the budget walk carries the same total it would recompute. Those are specific -improvements on specific costs, and they are the only performance claims this document makes. - -Not claimed: **no general Pareto superiority over another harness.** The preliminary *print* -comparison states outright that this system is not consistently cheaper or faster on the -small tasks measured; it exercised the print door rather than the interactive product, had -infrastructure and checker failures, undercounted cost before the meter fix, and ran too few -repetitions to support a claim either way. - -The interactive comparison records peer and candidate outcomes, including failures. -Both peers passed the initial mid-work revision case; both failed the first follow-up -case. codeaf's early candidate failed both. Further live diagnostics separated model -quality, display grouping and frozen-goal reruns. See `VALIDATION.md` for the tested -revisions and final checks. One repetition with differing context sizes, caching and -provider variability cannot establish a frontier in any direction. - -Any future comparison must name the tested door, workload, model, quality checks, -repetitions, latency and actual cost. Unmeasured cells are not successes. - -## Design limits - -- **Accepted is not read, and settled is not exactly-once.** Durable acknowledgement bounds - re-telling, not re-doing. A session with no journal, or a journal line that never reached - disk, is told again on resume. **No mechanism here makes an external effect idempotent.** -- **Admission context is deliberately partial.** It is not a durable universal constraint - ledger and does not guarantee that every relevant earlier statement is selected. A reference - is only useful if the recipient can open it. -- **Steering chooses one task.** Main-chat forwarding is model-addressed, not an automatic - classifier or broadcast. The receipt does not prove the worker read or applied it. Task - workers cannot forward under the person's authority. Original speaking time means when - this session drained the message, not when the key was pressed; ordering across clock - changes is not guaranteed. A publication already claimed cannot be recalled. -- **Attempt identity and assignment revision answer different questions** and must stay - distinct. -- **Cross-session communication is out of scope.** The addressing and origin shape should - permit a later authenticated router, but there is no discovery service, no message bus and - no cross-session permission model here. See `MODULES.md` §5 for what such a step would still - have to add. -- **The seams are files, not packages.** `internal/session` has not been split. diff --git a/docs/design/conversation-runtime/MODULES.md b/docs/design/conversation-runtime/MODULES.md deleted file mode 100644 index 82c02cecda..0000000000 --- a/docs/design/conversation-runtime/MODULES.md +++ /dev/null @@ -1,252 +0,0 @@ -# Modules: where the responsibilities are, and where the seams are - -**Read as of the local `a local working branch` wave, 2026-09-04.** Every row marked *implemented* names -code in this tree. Every row marked *proposed* is a plan and nothing more. `PLAN.md`, -`PRODUCT.md` and `COMMUNICATION.md` describe intended behavior and are not claims about -what ships. - -Two words this document uses precisely: - -- a **seam** is a named boundary *inside* `internal/session` — one file that owns a - question, with the rest of the package calling it instead of re-deriving it; -- an **extracted package** is a separate Go package with its own import boundary. - -**No part of the conversation runtime has been extracted into a standalone package.** -`internal/session` is still one package. The seams below are real files with real owners; -they are not module boundaries the compiler enforces. The only enforced boundaries in this -area are the pre-existing package lines: `internal/session` (engine), `internal/remote` -(client/server protocol), `internal/tui3` (terminal surface), `cmd/codeaf` (doors). - -## 1. Seams that exist - -| Seam | File(s) | What it owns | Status | -| --- | --- | --- | --- | -| **Delivery** | `mailbox.go` | who is addressed (`conversationID{session, task}`), who is speaking (`messageOrigin`), what kind of message it is (`messageKind`), one hand-over (`deliverTo`) and one answer (`deliveryReceipt`) | implemented | -| **Durable acknowledgement** | `mailbox.go` (`durableDelivery`, `deliveryID`, `Agent.hasRecorded`), `task_store.go` | the difference between a message a live reader accepted and one the recipient's own record holds | implemented | -| **Assignment and revision** | `assignment.go`, `assignment_tool.go` | the frozen admitted spec plus a revision overlay; which origins may move the done-condition | implemented | -| **Person-message forwarding** | `task_forward.go`, `assignment.go` | immutable request-bound source, one chosen address, retry identity and local correction ordering; uses the room delivery path | implemented | -| **Verification authority** | `task_checks.go`, `task_run.go` | explicit current check declarations, distinct from action receipts and acceptance prose | implemented | -| **Causal result wake** | `wakecause.go`, `task_result.go` | result, attempt and effective reply obligation travel together | implemented | -| **Admission context** | `admission.go`, `admission_compile.go` | one bounded, attributed selection of conversation excerpts and tool-output handles per admission door | implemented | -| **Result** | `task_result.go` | the full answer kept apart from the compact card, with a retrievable overflow reference | implemented | -| **Context window** | `toolcompact.go`, `stub.go`, `turnfold.go` | the bounded *view* of frozen history sent to the provider: reduced tool results with retrievable pointers, a fold to a headroom target, a linear budget walk | implemented | -| **End-of-turn handoff** | `turnhandoff.go` | whether the current request's work was handed to a task that is still live, so the turn can end without polling | implemented | -| **Projection** | `task_status.go`; adapters in `internal/tui3/taskstatus.go` | one pure `ProjectTask(TaskFacts) TaskStatus`: presence, wait-on, change disposition, fault, attention | implemented | -| **Conversation policy and limits** | `principal_wire.go`, `interactive_budget.go`, `rail.go`; launch shape adapters | interactive ownership independent of fixed-goal continuation and new-turn spending/time limits | implemented | -| **Background reply transport** | `internal/remote/wakelane.go`, existing turn ring/replay, `internal/tui3/watching.go` and `followup.go` | one hosted wake subscription; ordered turn adoption shared with local wakes | implemented | -| **Lifetime of a view** | `internal/remote/client.go` (`WorkOutlivesExit`, `Detach`), `internal/tui3/keeper.go` | whether closing a window ends the conversation, asked as a capability rather than guessed from a hostname | implemented | -| **Lifecycle and scheduling** | `task_run.go`, `task_ledger.go` | the graph, the frontier, state transitions, the five landings | no seam; owns several questions at once | -| **Evidence and outcome** | `task_audit.go`, `task_claims.go`, `taskgrade.go`, `task_checks.go` | did the work hold, and what is the record of it | no seam; reached through calls, not a value | -| **Workspace and change disposition** | `taskgit.go`, `treehold.go`, `task_branch_protection.go` | worktrees, branches, merges, holds | no seam; the landings call it inline | -| **Storage** | `task_store.go`, `checkpoint.go`, `task_index.go`, `sessionfile.go`, `world.go` | what survives the process | no seam; `graph.checkpoint()` is called inline at many sites | - -## 2. What each implemented seam actually guarantees, and what it does not - -### 2.1 Delivery (`mailbox.go`) - -A message is addressed to a `conversationID` — a session plus a task number, where task 0 is -the main chat. `deliverTo` walks the caller's ordered mailboxes and returns one receipt: -`nobody`, `closed`, or `accepted`. The two implementations are an `*Agent` and a `roomSeat`, -which resolves the live reader and appends under one hold of the room's lock, so a runner -withdrawing its seat cannot interleave into a swallowed message. - -Origin is carried separately from kind. `fromRuntime`, `fromPerson` and `fromAgent` all -arrive on one queue as user-role text, and the label is what keeps a descendant from raising -its own authority by writing a sentence that sounds like the person. Kind chooses the queue: -`msgProgress` goes to the ambient queue that no step drain reads, so telemetry cannot start -a paid turn however it was addressed. - -**Not guaranteed:** accepted is not read. A receipt says a live reader has the words on its -queue and nothing else. - -### 2.2 Durable acknowledgement - -For news whose sender is owed an answer about the *recipient's record* rather than its queue, -`durableDelivery` carries an id composed from checkpoint facts and a `settled` callback that -fires when the recipient's own journal holds the line. Only that second fact is written down -as announced, and a resume asks the journal (`sessionFile.recorded`) before re-telling a -landing. - -**Not guaranteed — stated in the code and repeated here because it is the most likely thing -to be misread as shipped:** this is **not exactly-once**. Dedupe covers what the journal -holds; a session with no journal, or a journal line that never reached disk, is told again on -resume, which is the direction it is designed to fail in. **Nothing here makes an external -effect idempotent.** A task that already sent an email has sent it, and no acknowledgement in -this runtime changes that. - -### 2.3 Assignment and revision (`assignment.go`) - -The admitted specification is frozen; corrections land as an overlay of `assignmentRevision` -values, and `effective` is what the work is judged by. `directionOf(messageOrigin)` is the -single mapping from origin to authority: **only a person's direction may move the -done-condition.** An agent's coordination line, however it is phrased, is recorded and read -but can never re-aim the work. A landing may not publish while a person's direction is -unread, which is what closes the race between a correction and a worker's finalization — -that boundary is on the node, not in the mailbox. - -**Not guaranteed:** discussion is not direction. There is no classifier model call deciding -whether a sentence was "really" an instruction; the rule is structural (which door the words -came through, and which origin they carry). An ordinary question asked in a room does not -rewrite the acceptance condition, and a correction typed in the main chat reaches the selected task only when the model -uses `forward`. The actual person message is bound to the model request; `say` remains -coordination. No automatic broadcasting is implemented. - -### 2.4 Admission context (`admission.go`) - -Each admission door compiles a bounded context: attributed quotes (`AdmissionQuote` carries -speaker and source) and tool handles (`AdmissionHandle` carries call, tool, input and -outcome, where the outcome may be `unknown`). Records are versioned and survive checkpoints. - -**Not guaranteed:** the selection is deliberately partial, and the partial-selection rule is -stated in the file rather than hidden. This is **not a durable universal constraint ledger**: -there is no promise that every relevant earlier statement a person made is present in the -context a task receives. A handle is useful only because the tools available to the recipient -can open it; an identifier alone would establish nothing. - -### 2.5 Result (`task_result.go`) - -The full answer is kept apart from the compact card, with a bounded excerpt and a retrievable -overflow reference, and the outcome qualification is retained rather than flattened into -"done". Delivery of a result is recorded separately from whether the work merged. - -### 2.6 Context window (`toolcompact.go`, `stub.go`, `turnfold.go`) - -Compaction rewrites the snapshot leaving for the provider, never the transcript. The system -prompt and the newest frozen batch stay verbatim; everything earlier is replaced by a reduced -view of itself — the head that says what ran, the tail a checkpoint digest already proved -keeps a verdict, the exact count of bytes cut, and a pointer to where the whole of it can be -read back (`Agent.fullResultPointer`), which falls back to the journal and then to saying -plainly that it cannot be read. Nothing reads a result and decides what it meant; head, tail -and counts are mechanical. A result may be shortened and is never dropped, because the -provider pairs every call with its result. A pass folds to a headroom target below the -threshold rather than re-firing at each step, and the budget walk carries a running total -instead of recomputing one. - -**Not guaranteed:** the model sees less than the whole history by construction, and a pointer -is only useful because the read tool can open it. This bounds resend cost; it is not a claim -about answer quality. - -### 2.7 End-of-turn handoff (`turnhandoff.go`) - -The measured failure this answers: a turn that correctly *handed work to a live task* was -re-opened by an end-of-turn reader asking whether the outcome had arrived, and the model, -with nothing to do, polled its own task. The seam asks the narrower question — did this turn -put the current request's work into a task that is still live — and lets the node's own -landing start the next turn, at wake prices, with the report in front of it. It is -request-scoped and does not poll. - -### 2.8 Projection (`task_status.go`) - -One pure function over facts. It separates presence (`queued`, `working`, `waiting`, -`finishing`, `done`, `incomplete`, `needs-look`, `stopped`, and a zero value for a state this -build does not recognise) from what is being waited on, from source-control disposition, from -fault, from attention. `Settled()` reads the lifecycle state, not the presentation word. -Liveness is three-valued (`unknown` / `held` / `unclaimed`) so an absent answer stays absent -instead of becoming "dead". Wire enums and checkpoint formats are unchanged. - -**Not guaranteed:** the project index still carries no merge word, branch or hold, so a -record row cannot raise the unlanded-edits demand a live row raises. Harness phases are -strings with no typed lifecycle; exactly one (`HarnessPhaseAsking`) is interpreted by name. - -### 2.9 Lifetime of a view (`internal/remote`, `internal/tui3`) - -`Welcome.Persistent` is the engine's own statement about whether it outlives the connection. -`Agent.WorkOutlivesExit()` reports that fact; `Agent.Detach()` sends `MethodDetach` when it is -true and falls back to interrupt-and-close when it is false. A terminal exit — including -SIGHUP — detaches instead of ending the conversation, so running work, standing questions, -drafts and parked messages survive. `/close` and an explicit stop are still endings, and the -quit hint promises "keeps running" only for conversations that will actually keep running. - -## 3. UX first principles this architecture is answering - -1. **The main chat is for thinking, not for waiting.** A person describes what they want and - keeps talking. The turn ends when the request has been handed off, not when the outcome - exists (§2.7). -2. **Workers own their results.** A task holds its own assignment, evidence and answer, and - reports when it has one. The main chat does not poll it, and a progress tick is not a - reason to pay for a turn (§2.1). -3. **A person can steer a room.** Words said into a task room are recorded on that task as a - direction with an origin, and a person's direction moves what the work is judged by while - an agent's does not (§2.3). -4. **The surface says what is true and no more.** A stop is not a failure, a queued node is - not running, an unknown liveness is not death, and a window closing is not an ending - (§2.8, §2.9). - -## 4. What is NOT achieved - -Stated plainly, because each of these is easy to read into the sections above. - -- **No distributed runtime and no exactly-once external effects.** Everything here is local - and in-process. The durable acknowledgement bounds *re-telling*, not *re-doing* (§2.2). -- **No automatic correction broadcast.** Main chat can forward the person message to one - selected task, and the person can type directly into a task room. Neither door revises - descendants automatically; receipt is distinct from reading and applying a correction. -- **No universal durable constraint ledger.** Admission context is a bounded selection, not a - record of every constraint a person has ever stated (§2.4). -- **No general Pareto superiority.** Specific costs are measurably reduced and covered by - tests — a long frozen tool history compacts to a sub-linear prompt, a fold reaches its - headroom target, the budget walk is linear (§2.6). That is not the same as being faster or - cheaper than another harness overall, which the available comparisons do not establish; see - `IMPLEMENTATION.md`. -- **No extracted packages.** §1's seams are files, not import boundaries. - -## 5. How addresses, origins and messages extend later without cross-session infrastructure - -The shape is already the extension point, and it costs nothing today: - -- **Address, not pointer.** `conversationID` carries a session id because a task number is - minted per session and repeats across them. A later router would resolve a remote address to - a mailbox exactly the way `deliverTo` resolves local ones, and the callers would not change. -- **`mailbox` is a two-method interface.** `address()` and `accept(delivery) deliveryReceipt` - are satisfiable by something that speaks to another process. The receipt already - distinguishes nobody / closed / accepted, which is the vocabulary a remote delivery needs. -- **Origin is a field, not an inference.** A future authenticated peer origin would be admitted at this boundary, then interpreted - by the centralized authority rule. An enum value alone would not establish authority. -- **Durable ids already exist.** `deliveryID` is composed from checkpoint facts, providing a local basis for retry recognition. Remote replay protection would still need - authenticated sender identity, persistence and a defined retention policy. - -What such a step would additionally require, and what deliberately does not exist now: -authenticated authority (an origin claim from another process must be verified, not -believed), admission policy (who may address this session at all), and replay protection at -the boundary. **There is no discovery service, no registry, no global bus, and no -cross-session permission model in this tree**, and nothing above should be read as a partial -implementation of one. - -## 6. Proposed next work - -Ordered by (how often the current crossing breaks something) ÷ (cost of the move). None of -this is started. - -1. **Landing / change disposition.** `task_run.go` and `task_ledger.go` hold five landings - that each decide a state, ask git about the branch, compose the note the model reads and - write a checkpoint. The wording of a landing lives in the file that owns the scheduler. - Move the branch questions behind the workspace seam and the note text next to the - projection's vocabulary. *Boundary test:* each merge outcome produces the same state and - the same person-facing reading; a protected checkout still keeps its branch. -2. **`roomStateWord` into the projection.** The last large surface table. It prefers the - kind's phase word, then the lifecycle line, then the gap, then the hold; three of those - four are facts the projection already takes. The fourth — a stop in flight — is a presence - the projection should gain. *Boundary test:* header, roster row and composer segment agree - about one node through queued → working → checking → stopping → stopped. -3. **Evidence as a value.** `task_audit.go`'s verdict becomes something the lifecycle - consumes rather than a call it makes. *Boundary test:* a refused verdict, an absent verdict - and an audit that never returns each produce their documented state without the scheduler - knowing how the audit ran. -4. **Storage recorder**, then **one worker loop for main and task threads**. Both are worth - doing only if a real defect keeps crossing those lines. Moving code into more files is not - a result. - -## 7. Criticism of this plan - -- **The projection can grow into a second engine.** It must stay a pure function over facts. - The moment it asks the graph a question, scheduling hides behind a presentation name. -- **The delivery seam can be mistaken for a transport.** It is a resolution-and-append rule - with an honest receipt. Every sentence about cross-session use in §5 is about *shape*. -- **Kinds invite over-typing.** Four kinds answer real differences in queue and wake policy. - A fifth should have to name a policy none of the others has. -- **`task_audit.go` remains the largest untouched risk** and is still not first, because its - interface to the lifecycle is narrow even though its internals are large. Size is not the - ordering criterion; crossings are. -- **The record still cannot say whether the person got their answer.** The index and the - projection describe a run and its branch. Neither states "the draft was delivered", and no - branch state infers it. diff --git a/docs/design/conversation-runtime/PARETO.md b/docs/design/conversation-runtime/PARETO.md index d04b030c43..8596cd8094 100644 --- a/docs/design/conversation-runtime/PARETO.md +++ b/docs/design/conversation-runtime/PARETO.md @@ -2,8 +2,8 @@ ## Current evidence -[Calibration 02](CALIBRATION-20260905.md) records 36 frozen runs across codeaf, -Pi and OMP. codeaf is not yet at the target: the observed small-task cost gap +Calibration runs (36 frozen cells across codeaf, Pi and OMP) recorded at +calibration time. codeaf was not yet at the target: the observed small-task cost gap remains, and each chat slice passed only one of two attempts. The next priority is unnecessary handoff/checking and the rendered request size. Bounded cache accounting and request-specific attribution landed after those measurements; @@ -50,7 +50,7 @@ Small repeated samples are calibration, not a demonstrated frontier. Use paired 4. **Optimize one measured bottleneck at a time.** First isolate unnecessary auxiliary work, repeated static context, admission/rephrasing rounds, task opening latency, polling, serial dependencies, and verification loops. Evaluate minimal ablations on development fixtures. Keep the simplest change that improves the declared objective without breaking another gate. Never replace a general failure with a hard-coded benchmark answer. 5. **Test adaptive task orchestration.** Add independent work, dependency fan-in, shared-file conflicts, cancellation, nested steering, task-room steering, ambiguity, long tool output, compaction, provider stall/failover, and restart/detach cases. Assert artifacts and visible behavior, rather than insisting on a fixed task graph. Measure the critical path and integrate cost of failures; spawning more tasks is not inherently faster. 6. **Confirm on held-out variants.** Freeze candidate and acceptance margins before opening an unseen fixture set. New datasets, different document constraints, independent code defects, changed correction timing, and longer histories must exercise the same capabilities. Do not use the public reversal fixture as a proxy for all conversational correctness. -7. **Keep the result production-safe.** Regression suites and the real chat door must pass. Document actual limits, status semantics, and capability availability in the manual whenever behavior changes. Roll out only through the owner's requested local integration until explicitly authorized otherwise. +7. **Keep the result production-safe.** Regression suites and the real chat door must pass. Document actual limits, status semantics, and capability availability in the manual whenever behavior changes. Roll out through the normal review-and-merge path before claiming the result. ## Initial acceptance targets, to preregister before confirmation diff --git a/docs/design/conversation-runtime/PLAN.md b/docs/design/conversation-runtime/PLAN.md deleted file mode 100644 index dcc68aedbb..0000000000 --- a/docs/design/conversation-runtime/PLAN.md +++ /dev/null @@ -1,284 +0,0 @@ -# codeaf: keep thinking while the work gets done - -**Product proposal · 2026-09-04** - -The product I would build is a conversation that stays useful while decisions turn into results. The person should be able to discuss a direction, say “build that,” continue thinking, occasionally look inside the work, change their mind, and return later to something they can use. - -The visible innovation is continuity across simultaneous work. Faster agents alone do not produce it. Neither does putting a task dashboard beside an ordinary coding transcript. codeaf should absorb the coordination that currently falls on the person: repeating context, checking whether anything started, saying “continue,” reconciling results, and reminding workers what changed. - -This plan starts from that experience. Existing task graphs, prompts, branches, checks and room designs do not define the answer. The prior audit supplies failure examples; it is not the specification. The owner confirmed that requested work must continue after the terminal closes. - -## 1. The product's unit is an outcome someone can discuss - -The person's objects are “offline reading,” “the launch draft,” “the pricing comparison,” and “that import problem.” Those are more stable than an agent process, model name, implementation plan or sequence of tool calls. - -Each substantive assignment has a conversation attached to it. The main conversation keeps the overall intent; the work conversation holds the investigation, implementation and local decisions. A task may create further work conversations, but remains responsible for its own outcome. Opening any one of them means joining the same ongoing effort, with its existing context. It does not launch a new assistant that asks for the brief again. - -The product should feel like **one assistant with several threads of work**, rather than several assistants the person must manage. Models and workers can change underneath a stable assignment. A work conversation can survive retry, checkpoint, model replacement and reconnect without changing its identity. - -A task is not obliged to spawn more tasks. One capable worker is the default useful case. Delegation is something that worker can do when it finds a reason, not a ceremony every request must pay for. - -## 2. The main conversation stays at the level of intent - -The main chat is where the person asks questions, explores possibilities, chooses direction, starts work and understands outcomes. It can read a relevant file or look up a fact to have an informed conversation. It should not disappear into a hundred-step implementation while the person is waiting to discuss the next idea. - -Implementation, lengthy investigation and artifact production move into work conversations. Very small work can appear as a compact operation inside the main exchange; it should not require a planner, a separate naming call and a checklist before starting. The execution record can still exist underneath. Separating work from the foreground must become cheap enough that it does not tax every minor change. - -Do not make users choose between a chat mode and a work mode. Interpret the request in context: - -| Person says | Expected behavior | -| --- | --- | -| “Could offline reading make this more useful?” | Discuss the idea | -| “What would it take to add it?” | Assess; investigate in the background if substantial | -| “Can you add offline reading?” | Start the work; polite phrasing is still a request | -| “Let's do the simpler version we discussed.” | Start the agreed version using the existing context | -| “Maybe we should support teams eventually.” | Continue exploring; do not silently expand the running implementation | -| “Change the running version to support teams.” | Revise that assignment and coordinate the consequences | - -This needs model judgment, not keyword rules. Ask a short clarification when two plausible readings would cause materially different effects. Do not ask merely because an internal tool wants another field. - -The assistant should acknowledge an actual start with a concrete sentence: “I'm adding local offline reading. I'll bring back a working preview.” The interface can draw a task receipt immediately after admission. It must not wait for a naming model, and it must not claim something started before execution was accepted. - -## 3. Work becomes part of the conversation - -A handoff creates a compact work item at the point it was discussed. Its title names the outcome. Below it are short, attributable lines about current work, a question if necessary, and eventually the result. - -A conceptual example, not a pixel design: - -```text -You -Let's add offline reading. Keep this version local to the device. - -codeaf -I'm adding local offline reading. I'll bring back a working preview. - - Offline reading running - Saving articles and checking how saved pages reopen - Open work - -You -While that's happening, what should the onboarding explain? - -codeaf -It needs to explain what is available offline and how to remove it... -``` - -Activity should remain visible in the conversation. A jobs-only sidebar is insufficient: work otherwise seems to disappear. Within a work item, a few short lines may show independent activity with attribution. Changes update that item's live area rather than adding a new assistant message for every grep or child completion. The person can expand the details. - -A small optional active-work strip keeps older assignments reachable after the conversation moves on. It is an aid to navigation, not the main product. A permanently expanded graph would make the person think about implementation topology before they need to. - -Meaningful developments deserve a new conversational update: “Private browsing prevents storage here; I'm using an in-memory fallback.” Routine progress does not. Completion should appear once at the level that owns the user's request, not once per descendant. - -No update steals keyboard focus, moves the person away from what they were reading, or destroys a draft. Someone reading earlier messages can see that new work information exists without being dragged to the bottom. Announcements for assistive technology should favor meaningful state changes over every caption update. - -## 4. Entering work is joining a conversation - -Opening “Offline reading” shows: - -- Its current aim, expressed in one or two sentences. -- The latest useful finding and what is happening now. -- Its draft, preview, files or other output when available. -- The ongoing conversation, including the person's earlier steering. -- Child work when it helps explain progress or a dependency. - -The composer is an ordinary conversation composer. “Why this approach?”, “show me the current version,” “try the simpler cache,” and “stop here” all work without special modes. - -A question about status should not cancel execution. Opening a task does not pause it. Leaving does not resume it, because it never stopped. A user message is addressed to the ongoing worker; inspection does not secretly spin up a second worker that mutates the same material. - -A compact path such as `Main / Offline reading / Storage` makes the current addressee clear. Back returns to the main conversation at the previous reading position with its draft intact. The person never has to summarize what happened inside the task on the way out. - -The latest preview is more important than the tool log. Evidence and detailed tool history remain available, but the first view answers “what is being made, how is it going, and can I change it?” - -## 5. Changing your mind is a normal operation - -The crucial innovation is that steering actually updates the work. It is not a note appended beside a frozen plan that a later checker ignores. - -Default scope follows where the person speaks. In a work conversation, direction applies to that assignment and its descendants. In a child, direction applies there. In the main conversation, named work and clear conversational references identify the target. Explicit words can widen the scope from anywhere. - -| Message | Product response | -| --- | --- | -| Inside storage: “Use IndexedDB for this.” | Apply to storage; notify the owner of consequential interface changes | -| Inside storage: “Actually, the entire feature must work without a server.” | Update the feature constraint; inform affected work | -| In main: “Make both drafts more direct.” | Update both named drafts | -| “What if we used SQLite?” | Discuss; do not assume an implementation change | -| “Use SQLite instead.” | Apply the change and explain any material consequence | - -A clear local instruction from the person outranks an older agent plan. The parent cannot later overwrite it simply because the original brief said something else. If the instruction conflicts with another explicit user requirement, resolve the concrete conflict rather than pretending recency answers every question. “Including email addresses would replace your earlier no-personal-data requirement for this export. Is that the change you want?” is a meaningful question. “Which node should receive this?” is usually the system outsourcing its job. - -For an ordinary revision, acknowledge the effect: “CSV replaces JSON for this export. I'm updating validation too.” The person can inspect that acknowledgment or correct it. Do not require approval of a generated contract form. - -Distinguish receipt from completion. “Change received” means it is stored; “working with the new format” means the worker has applied the direction to its next work. Already-running operations may take time to stop. The interface should say so accurately. A private draft created under the old direction may still be useful evidence, but it must not silently become the delivered result for the new direction. - -If a completed result is being changed, continue the same work conversation with a new revision and keep the prior result accessible. A genuinely new assignment can start a new linked task. “Continue” should not recreate the task or ask for the story again. - -## 6. Most coordination should not involve the main chat - -The main chat is not the manager of every worker turn. The task owner decides how to accomplish its assignment, delegates when useful, answers its children's questions from available context, and integrates their work. It can do useful work itself while children run. - -Children share concise findings, questions, interface changes and results. They do not broadcast transcripts. The parent is accountable for resolving incompatible assumptions. The user should not paste one child's answer into another or approve a decomposition just to get moving. - -Use an ownership tree for responsibility and explicit dependencies where work crosses that tree. The organization may evolve as facts emerge. A task can discover a problem, delegate a small investigation and continue another part; it need not invent the whole graph before reading the material. Cyclic waits and duplicate ownership are runtime errors to prevent, not puzzles to give the user. - -A useful early finding can change ongoing sibling work. For example, a storage investigation discovers a browser limit; the owner adjusts the downloader immediately rather than waiting for every child to finish. An ordinary successful subtask need not wake the owner if there is nothing yet to decide. The distinction is material news versus bookkeeping. - -The main conversation needs only changes that affect the user's understanding, decisions or next request. It can fetch details when asked. It should not reread or re-approve every child's successful step. - -## 7. Questions should earn the person's attention - -Assuming capable models means trusting them to resolve ordinary implementation choices. Questions are for missing preferences, facts or authority whose wrong interpretation materially changes the outcome. - -A child asks its parent first when the parent could know the answer. It reaches the person only if necessary. Related questions can be combined into one understandable choice. The question includes a sensible recommendation when one exists and a concrete consequence. It never includes an internal role dispute. - -A question visible in main and in the task is the same question. Answering in either place resolves it once. If the person is away, it remains available on return. Independent work continues. A required answer is not silently invented after a timeout. - -Not every uncertainty needs to block. For a reversible detail within the requested scope, the worker can choose a reasonable default and note it with the draft. For a decision that would invalidate substantial work, asking is cheaper and kinder. The product should learn enduring preferences only when supported by the user's decisions, not promote one local choice into a permanent global rule. - -## 8. Results are the return path - -“Task finished” is not an adequate result. Return what the person wanted: - -| Assignment | What comes back | -| --- | --- | -| Feature | Working preview or actual code delivery, changed behavior and relevant checks | -| Bug fix | Reproduced failure, corrected behavior and regression evidence | -| Research | Answer or recommendation with sources and unresolved uncertainty | -| Data work | Output, reproducible calculations and relevant assumptions | -| Writing | The draft itself, ready to read or use | - -An outcome should be understandable from the main conversation without opening a task. Details expand into the original work. A concise return might be: “Offline reading is ready in the preview. Saved articles reopen without a connection. Images are cached; embedded video still needs a connection.” - -The words must match the actual delivery stage. A preview ready for review is not a production deployment. A kept branch is not a merged change. For work that lacks an objective pass/fail check, say what was completed and assessed rather than manufacturing certainty. - -New ideas can grow directly from results: “Keep the behavior, but make the save control smaller.” That message continues the relevant work with its existing evidence and artifacts. The person should not specify task IDs or manually establish the dependency unless the reference is genuinely ambiguous. - -## 9. Closing the terminal changes presence, not intent - -The owner explicitly chose continued execution after closing the terminal. The terminal is therefore a view into work owned by a persistent local service. - -Closing the window detaches. Requested work continues under its existing scope and budget. It does not create new goals or acquire additional authority while the person is away. A pending required question stays pending; other useful work can continue. - -Returning restores the same conversation and work. Offer a short recap such as: “Offline reading is ready. The launch draft needs your choice of audience.” Link to the results and questions in place; do not replay the entire activity feed. - -Local work cannot execute while the machine is powered off. Sleep pauses local execution; reconnect/restart should recover safely when possible. If an external action may have completed before a crash, reconcile its actual outcome before repeating it. A remote execution host can later provide availability independent of the laptop, but it is not an invisible promise of the local design. - -“Pause this,” “stop this,” and closing the window have different effects. Pause preserves resumable work. Stop prevents further work where possible and preserves useful partial results. It does not undo effects that already happened. The person can stop one assignment, its family or all active work from the main conversation without hunting for processes. - -## 10. The minimum information model behind the experience - -These records exist to make conversational behavior reliable, not to become forms in the UI. - -- **Conversation:** user messages, relevant decisions, linked work, and outcomes. Preserve exploratory ideas as exploratory. -- **Assignment:** current aim, relevant constraints, evidence of completion, owner, scope, budget and revision. -- **Working context:** useful observations, hypotheses, source/artifact references, current dependencies and unanswered questions. -- **Result:** usable answer/artifact, delivery state, checks performed, remaining limitations, and the revision it satisfies. - -Keep the distinction between the person's decision, an observed fact, and a model's interpretation. Context given to a worker should answer “what am I responsible for, what matters here, what is already known, and how can I get more detail?” It should not be the entire main transcript or every ancestor's complete goal. - -Compaction is a memory-management detail. It may shorten working history, but must preserve the current assignment, binding directions, open questions and useful results. Sources remain retrievable. Changes to a file can make an earlier observation stale. Sometimes reading it again is the correct behavior. - -Do not build a universal knowledge graph first. A few typed records, links to original messages and source revisions are enough for the initial product. Version changes that affect scope or acceptance; do not make every sentence a formally adjudicated fact. - -## 11. Assume capable models; keep deterministic obligations in code - -The model should decide how to solve the work, when a helper is useful, what evidence is sufficient, and which uncertainty deserves a question. The harness should reliably deliver messages, enforce scopes and budgets, preserve outputs, keep the foreground responsive, and report what happened. - -That division removes many unnecessary AI management layers. A task should not need separate models to name it, restate the brief, approve the restatement, classify every step, narrate it, and then independently infer whether a completion string means success. Use an appropriate model for real judgment. Use code for state transitions and bookkeeping. - -One conversation/worker loop can serve both main and task threads. Do not require a new framework for nested work. A task becomes another addressable conversation with an assignment, a durable state and the ability to delegate. - -The minimum runtime is a persistent owner process, transactional task/message storage, a bounded execution queue and durable result delivery. A waiting conversation consumes no model call. Current records plus append-only messages are sufficient; a full event-sourced architecture is not a prerequisite. Diagnostic traces are separate from the authoritative user/work records. - -Cap actual executing requests and tools, not just visible tasks. Preserve capacity for foreground interaction and urgent steering. Budgets include descendants, coordination, retries and checking. Extra depth must consume a shared allowance rather than create new money at every level. - -Isolation depends on effects. Code changes need safe ownership and integration. Read-only investigation does not need a new Git repository. A document task needs an output location and versioning, not a branch ritual merely because coding tasks use one. Cross-task writes must have an owner or an integration step; conversation alone cannot prevent overwrite races. - -## 12. What I would deliberately leave out - -- A mandatory plan/task-definition form before work starts. -- A graph dashboard as the default home of the product. -- A requirement to select models, assign worker counts or approve internal decomposition. -- A separate “talk” versus “execute” mode that the person must remember. -- Every-child completion announcements and raw tool traffic in the main chat. -- Automatic scope changes inferred from speculative ideation. -- A permanently running manager model that polls workers. -- A universal reviewer pass for every small deliverable. -- A blanket ban on rereading or a requirement to treat inherited summaries as truth. -- Unbounded recursion as a selling point. - -Available detail should expand when the person wants control. Expert controls remain reachable, but expertise should not be the price of ordinary success. - -## 13. Criticizing this proposal - -**I initially overemphasized machinery.** My audit recommendation—contracts, evidence stores, explicit events—could become another large architecture with an equally demanding interface. I would reduce it to the smallest records needed for delivery, steering and recovery. Product behavior is the test; a beautiful state model is not evidence of a good experience. - -**A main conversation can become too vague.** If it is insulated from all work, it will offer generic advice and forget what is possible. It needs current outcomes, material discoveries and reliable access to details. It does not need the entire tool stream. - -**Quiet can become invisible.** Hiding everything behind a sidebar makes the person wonder whether work is happening. Keep short meaningful activity within the conversation's work items, with visible changes and easy expansion. Reduce noise without removing evidence of life. - -**Delegation can make work slower.** A strong single worker with parallel tools may beat several agents because it shares context and avoids integration. Ship the same product experience for both. Increase internal parallelism where paired measurements show benefit; never make the user perform the optimization. - -**Fluid steering can destroy coherence.** If every local thought changes everyone else's instructions, work becomes unstable. Scope follows the active conversation, material shared changes reach the owner, and speculative discussion stays distinct from directions. A real conflict between user requirements deserves a focused question. - -**One owner can become a bottleneck.** The owner should handle consequential coordination and final integration, not approve every child tool call. Children retain local autonomy. Code handles ordinary scheduling and delivery. The owner sleeps when it has nothing useful to do. - -**Changing direction has a real cost.** Do not pretend a rewrite is free. When a new choice invalidates substantial work, say what will be reused and what must change. Do not force the person to approve every minor adjustment to protect a fragile plan. - -**“Just works” cannot mean confident claims.** For deterministic work, check the result. For qualitative work, use meaningful review at the appropriate scope and expose uncertainty. A capable model should be given responsibility, with the product preserving observable evidence rather than surrounding it with ritual distrust. - -**Persistent autonomy can feel out of control.** Show what continues, keep stop/pause straightforward, and maintain the agreed scope and budget. The service executes requested work; it is not a self-appointed employee inventing assignments. - -## 14. A full experience to test before committing to the design - -1. The person discusses offline reading and chooses a local-only first version. -2. “Build that” starts an assignment in place. Main remains available immediately. -3. The person explores onboarding while the task works. No speculative onboarding idea changes the implementation accidentally. -4. The owner discovers two independent pieces and delegates. A short activity area shows work advancing without an agent-management ceremony. -5. The person opens storage, asks why it chose a mechanism, and says “keep this browser-only.” The change is recorded and affects the right work. -6. A storage finding changes downloader assumptions. The owner coordinates it without asking the person to carry messages. -7. The person returns to main and requests a short help draft. That work proceeds independently. -8. The person closes the terminal. Both assignments continue locally while the machine is available. -9. One task encounters a preference it cannot resolve; the question persists while independent work finishes. -10. On return, the person sees a short recap, answers the question, and receives a usable feature and draft. -11. “Make the explanation shorter” revises the correct draft without restarting its research or altering the feature. - -Repeat the same shape for a multi-source research comparison, data analysis with a changed definition, and a writing task with an audience change. If this experience requires the person to learn internal terms, restate context, collect child outputs, or repeatedly say “continue,” the product has failed even if all its workers ran successfully. - -## 15. Delivery plan, organized by product proof - -| Stage | Build | Required proof before expanding | -| --- | --- | --- | -| 1. Handoff and return | Main conversation plus one persistent work conversation, inline work item, usable result | Commission ordinary work, ask an unrelated question, leave, return to the right result without another prompt | -| 2. Enter and steer | Same live work opens as chat; scoped revisions; receipt/application feedback; pause/stop | Direction survives active calls and checking; returning preserves draft/location; result satisfies latest request | -| 3. Capable delegation | Same worker loop delegates bounded child work; ownership, questions and integration | Parent resolves useful child news; user need not manage siblings; one coherent delivered outcome | -| 4. Continuity under change | Cross-task consequences, recovery, pending questions, previous results and revisions | Resume without duplicated effects; concurrent directions do not silently overwrite; stale outputs cannot become current results | -| 5. Reduce waiting and cost | Parallelism/routing/context refinements using the same experience | Faster accepted outcomes with equal or better quality and no added supervision | - -Build the first stage with one capable worker. It proves most of the product promise and avoids using agent-count as a proxy for innovation. Design delegation semantics to support recursion, but roll out deeper nesting only after the simpler configuration succeeds and there is evidence it helps. Stage 3 should include at least one genuine child-of-child scenario before claiming arbitrary nested steering works. - -Each stage is a vertical slice: UI, runtime, persistence, outcome checks and observable traces together. Do not implement an elaborate backend and postpone the conversation experience until the end. - -For the existing repository, evaluate reuse of the provider/tool loop, terminal renderer, task conversation views and local engine host against these requirements. Replace competing lifecycle and steering paths rather than preserving them because they already exist. Keep old sessions readable; opt new sessions into the new runtime, use versioned storage and retain a rollback route. Do not dual-run mutating work during migration. Every changed behavior updates the compiled manual in the same implementation wave. - -## 16. How we decide whether this is actually better - -The primary measure is **a usable result with less required attention**. Track: - -- Whether the work met the latest request and actually reached its destination. -- Time until the person can continue talking, and time to accepted result. -- Required clarifications, repeated instructions and manual coordination. -- Correct interpretation of exploration versus a direction to act. -- Whether steering arrived, was applied, and changed the right scope. -- Time spent by the person inspecting status just to know if work is alive. -- Total spend and integration/retry overhead, including all descendants. - -Qualify behavior first with deterministic UI scenarios for interruption, concurrent messages, child questions, changed scope, unavailable providers, restart and stale results. Use actual produced artifacts and recorded lifecycle facts, not prose claiming success. Retain failing evidence automatically. - -Then compare one persistent worker, a worker using parallel tools, bounded helpers, and recursive helpers on matched workloads. Hold model/effort/provider policy steady for the orchestration comparison. Separately test model allocation. Include independent and tightly coupled work, changed requirements, coding, research, data and writing. Use paired repetitions and blind qualitative assessment where needed; do not report a speedup from one favorable run. - -A high-level user walkthrough must pass without opening the task tree. A second walkthrough deliberately enters a child and changes its direction. Both must work. The point is that control is available, while supervision is optional. - -## Recommendation - -Build the conversational handoff-and-return experience first, with one persistent capable worker and a task conversation you can enter. Make steering a real change to the work and make delivery unambiguous. Then let that same work conversation delegate when useful. - -This keeps the product ambitious while reducing implementation uncertainty. The person gets the experience they want from the first slice; increased parallelism improves it behind the same interface instead of becoming a new product they have to operate. diff --git a/docs/design/conversation-runtime/PRODUCT.md b/docs/design/conversation-runtime/PRODUCT.md deleted file mode 100644 index 14fb98debe..0000000000 --- a/docs/design/conversation-runtime/PRODUCT.md +++ /dev/null @@ -1,44 +0,0 @@ -# codeaf product brief - -This is a proposed product direction, not a description of shipped behavior. It derives from the owner's request on 2026-09-04; the existing branch is evidence about implementation problems, not product authority. - -## Confirmed intent - -- The main conversation supports ideation, decisions, high-level steering and continuity. -- Capable models should accept substantial work and carry it through with little supervision. -- Substantial work happens in task conversations; ordinary work may stay inline. Tasks represent outcomes, and internal helpers are collapsed into activity by default. -- The person can enter a task conversation to inspect and steer it, then return to the main discussion. Internal execution hierarchy is optional diagnostic detail, rather than the default navigation. -- Tasks continue when the terminal closes; the owner explicitly confirmed this. -- Quality, completion time and cost matter together. The product should handle work beyond coding. -- The owner subsequently authorized implementation and refactoring on a local integration branch. The design below remains a target; `IMPLEMENTATION.md` distinguishes landed changes from work still in progress. - -## Proposed product promise - -Keep thinking here. Work you ask for gets done alongside the conversation. Change direction wherever you are looking, and come back to results you can use. - -## Primary experience - -One assistant, several simultaneous threads of work. The user commissions outcomes and makes meaningful decisions; the product handles decomposition, coordination, context transfer and integration. Agent management is optional diagnostic detail. - -## Platform and operating context - -The immediate surface is codeaf's terminal chat. The interaction model should also transfer to a graphical client. Closing a client detaches from ongoing work. Running while the machine is powered off requires another execution host and is not promised by local persistence. - -## Proposed principles - -1. Attention is the scarce resource: completion with less supervision beats visible agent activity. -2. Every material assignment has one owner responsible for the usable result. -3. Discussion, committed direction, observations and proposals remain distinguishable. -4. Steering changes the current assignment; it does not require restarting the conversation. -5. Durable work and ordinary conversation remain connected without copying every transcript everywhere. - -## Open choices for implementation - -Exact visual styling, keybindings, budget defaults, model allocation and deployment policies need implementation-stage decisions. They are not prerequisites to judging the product concept. Initial local-service hosting and bounded parallelism are recommendations in the design plan. - -## September 5 interaction prototype - -The owner approved the conversation-with-tasks direction and requested a prototype. -[TASK-EXPERIENCE-PROTOTYPE.md](TASK-EXPERIENCE-PROTOTYPE.md) specifies the task -identity, message and result boundaries and the rollout acceptance scenario. -The interactive prototype uses simulated work; it does not change the live terminal. diff --git a/docs/design/conversation-runtime/PRODUCTION-AUDIT-20260906.md b/docs/design/conversation-runtime/PRODUCTION-AUDIT-20260906.md deleted file mode 100644 index 36e28b2dba..0000000000 --- a/docs/design/conversation-runtime/PRODUCTION-AUDIT-20260906.md +++ /dev/null @@ -1,869 +0,0 @@ -# Production readiness and task UX audit — September 6, 2026 - -**Verdict: not ready for a production release.** The task experience has working -paths and meaningful regression coverage. Recipient isolation and owner-aware -navigation, durable sending and chat-sidebar fixes are integrated. The combined -offline gate is recorded in the final integration section below. Passing one substantial coding case does not close -these gaps. Quality and correct delivery precede cost optimization. - -## Method and scope - -Two independent read-only reviews covered task UX and execution reliability -(`task_ux_audit` and `runtime_quality_audit`). The main audit traced the current -implementation, ran the real Bubble Tea input decoder through the local engine -wire client, reproduced defects with disposable fixtures, and fixed two narrow -navigation defects. This is terminal-specific QA, not a website heuristic score. - -Baseline source: `d9da6c48dc6c4b6d00d9b04657fe8a03c4cc1439`, in -`/private/tmp/af-runtime-next`. The interaction prototype added by that commit is -**simulated design, not the live terminal implementation**. Both this baseline -(CI run 33999223712) and the preceding runtime revision `23c98d9fc` -(run 33997640609) passed all remote CI jobs. - -The source review covered task entry points, reader subscriptions, composer and -view state, task commands, local/remote direction delivery, reconnect, completion, -branch handling, cancellation, verification, and existing evaluation evidence. -It does not certify every feature in the repository or every operating system. - -**Direct inspection of the user's iTerm window was blocked by computer-use safety -policy.** No terminal window was clicked, restarted, replaced or inspected through -another automation mechanism. Test frames came from isolated application fixtures, -not a screenshot of the user's window. Real-terminal accessibility and visual -appearance therefore remain unverified. No new live codeaf developer-task evaluation was run -in this audit; the authorized Claude CLI implementation and review work is separate. - -## Initial click findings — historical launch observations - -These observations were made at the start of the audit. The third source defect -has since been fixed; the first two launch paths have not been replaced or rechecked -in the user's window. They must not be collapsed into one guessed cause. - -| Evidence | Finding | Required action | -| --- | --- | --- | -| The executable at the running `/private/tmp/af-conversation/bin/codeaf` process path reports `3cb79f3f8 (dirty)`, built September 5 at 03:12. Its source still gates hosted task opening on `hosted()`. | It predates the fix allowing local engine task readers without a remote hostname. | Verify the actual failing window and put the tested client build in its launch path. Preserve the existing conversation and work. | -| The executable at the other running shared-checkout path reports `d4ac249c (dirty)`, built September 3. The audited binary reports `d9da6c48d`. | Multiple launch paths expose materially different products. A fixed worktree does not update an already running client. | Establish one documented launch/install path and show both client and engine revisions in diagnostics. Test upgrade/reconnect without losing work. | -| `tasksItem.pick()` deliberately returns false for another window's live task. The existing test requires Enter to do nothing. | At the initial audit revision, visible other-conversation rows were intentionally inert. Local owner navigation is now implemented. | Qualify the implemented local owner view in the actual launch path. Remote other-conversation navigation remains unsupported and needs an actionable path. | - -The audit shell also has no `codeaf` on PATH. That is evidence about this shell, -not proof about the user's interactive shell or aliases. The shared checkout is -dirty with other work and was not rebuilt or modified. - -## Fixes made during this audit - -1. **The compact task shortcut stays available for unfinished work.** It previously - disappeared when the last `TaskRunning` node became `TaskUnverified`, even with - queued work or a waiting decision. Its visibility now uses the same groups as - its displayed contents. At phone width the summary names **needs you** before - running work. Existing wide-terminal behavior remains the task list. -2. **Switching task views releases the previous transcript subscription.** Opening - a different task previously replaced the room without calling its release - callback. All room constructors now close the previous view before creating - the next. Navigation releases a reader; it does not cancel execution. - -New regressions fail before these changes. The shortcut test clicks through to the -actual task transcript at 44, 80 and 99 columns, using the local engine wire reader; -the smallest layout goes through the Tasks page. The subscription regression -checks release exactly once. A broader gate also exposed a test-only notification -race, described in the validation appendix. These fixes do **not** implement per-task drafts, -scroll restoration, cross-window navigation or the whole prototype. - -## Production work, in priority order - -P1 items below are release requirements. P2 items are important UX/operational -work; a release decision must explicitly address their limits. “Confirmed” means -source or reproduction supports the statement. “Unproven” is an acceptance gap, -not an assertion that the behavior is broken. - -| Priority | Workstream and current evidence | Definition of done | -| --- | --- | --- | -| **P1** | **One tested build reaches the user.** Confirmed old executable paths above; existing failing window not directly inspected. | A documented launch path opens the tested client, reports client/engine revisions, enters the existing task, sends a direction to the correct task, returns and reconnects with work intact. No destructive host restart or silent old-binary reuse. | -| **P1** | **Recipient-owned drafts and task messages. Draft and send ownership integrated.** Main, task and run composers now retain their own text, caret, paste blocks and attachment tray, including structured crash recovery. The original misdelivery regression is covered by permanent tests. Guest recipient separation, queued delivery, stale-save refusal and uncertainty recovery have focused race coverage; see the final integration gate below. | Main, task A and task B independently preserve text, caret, compact pastes and attachments. Switching views never changes an unsent message's recipient. Submission clears only the accepted recipient's draft. Test Escape, direct task switches, project switches, reconnect and crash recovery. Treat a question as a question; do not silently reinterpret every task message as a scope revision. | -| **P1** | **Every visible task has an understandable opening action. Partial integration.** Local other-conversation views now join their owner without starting work or taking the keyboard. Guest identity, status and draft isolation pass focused tests. The right-hand chat column now opens visible rows, bounds hit targets to drawn content, orders attention/live work first and preserves the open page on repeated selection. A rendered-frame/raw-SGR test opens exact parent/child engine transcripts at 100, 119, 120 and 160 columns. Remote other-conversation navigation and actual-user-window validation remain gaps. | Mouse and keyboard open the same stable task from the rail, Tasks, home/project views and another window. Identify the owning conversation and project. Show a recoverable error for a missing/unsupported owner; do not leave an apparently selectable row inert. | -| **P1** | **Hosted multi-conversation lifetime. Confirmed limitation.** The shared-handle UI now preserves chat identities and unsent drafts, but selecting another conversation on the same engine-backed connection closes the previous one. Local `--no-host` keeps other conversations running. | Switch A→B→A without stopping either conversation or aliasing their agents. Preserve each reader, draft and message owner through reconnect, close and task completion. Tabs must reflect actual lifecycle behavior. | -| **P1** | **Non-blocking send and reliable acknowledgements. Backend and UI integrated.** Stable message scope/sequence/time and expected conversation reach durable engine admission before acknowledgement. Owner mismatch and duplicate identity tests pass. The asynchronous composer saves the same outbox snapshot before wire delivery; retries retain identity after uncertain receipts. Some draft-clearing, quit and takeover writes remain synchronous, so arbitrary disk stalls are not fully qualified. | Send runs asynchronously. Show pending, received and applied distinctly; Escape/navigation remain responsive. Retain the draft after a failed or uncertain send. Retry a lost acknowledgement with the same message ID and apply it once, including after reconnect and from another window. A second intentional identical message remains distinct. | -| **P1** | **A retained result remains usable when narration fails. Observed historical failure; integrated recovery unproven.** Artifacts and result references are retained, but a failed final model answer has previously left no substantive visible delivery. | Immediately expose the saved artifact, originating request and truthful failure state. One retry delivers the answer without rerunning completed work or duplicating the completion notice. Test provider failure during final narration, reconnect, and a newer unrelated main-chat question. | -| **P1** | **Respect the requested delivery and branch workflow. One fresh live acceptance passed.** The September 7 four-module repair produced the requested branch and final commit, passed 29 independent checks, and left main unchanged. Late acceptance exposed a released-worktree recovery defect; real-Git regressions cover the repair, but that repair has not had another paid end-to-end run. Stopped, conflicted and revised live cases remain unqualified. | For branch-and-commit requests, main and the user's checkout remain unchanged through worker completion and parent follow-up. Cover successful checks, inconclusive review, stopped work and conflicts. Remove conflicting advice. Report branch, commit, exact checks and unresolved work. Reading a result must never imply acceptance, merge or publication. | -| **P1** | **Current-request verification through the whole task tree. Component tests pass; integrated proof missing.** Explicit checks and revision invalidation exist; the automatic route-check fix has not completed another realistic live evaluation. | Revise one deliverable during child/parent checking. Affected work gets fresh checks and evidence; unaffected siblings retain their scope. Old evidence cannot satisfy the revised request. Cover partial handoff, failed checks, restore and final delivery. Never scrape arbitrary worker commands into rerunnable checks. | -| **P1** | **Stop the owned execution tree. Unproven end-to-end criterion.** Cancellation paths and component tests exist; this audit has not shown a full hosted-UI process-tree stop. | Stop a parent with child/grandchild tasks, a queued dependent, a foreground subprocess and a background process group. Owned processes stop within the documented grace, no new work starts, partial artifacts survive, and an unrelated task continues. Repeat while parked and during provider recovery. | -| **P1** | **Reconnection, persistence and crash recovery. Mixed evidence.** Live terminal detach succeeded previously, and host/socket tests pass. Hard host crashes and uncertain external effects are separate cases. | Distinguish terminal closure, network loss, graceful shutdown and hard crash. Restore the same task, pending question/direction and unseen result. Deliver each notification once and show interrupted/uncertain work honestly. Do not blindly replay an action whose external outcome is unknown. Cover two windows and idle-host retirement. | -| **P1** | **Independent quality gates on substantial work. Confirmed evidence limitation.** The one substantial repository case passes 220 checks yet exposes a public type-contract defect; the reference shares it. | Freeze multiple substantial repository issues and non-code tasks, with independent negative/interface/integration checks and requested-workflow grading. Repeat correction, interruption, long context and provider failure. Grade the delivered artifact, not whichever hidden child happens to look best. Retain failures and unknown costs. | -| **P2** | **Per-task reading state. Implemented for ordinary task pages within this process.** Owner-scoped bookmarks preserve scroll anchors, expanded entries and explicit follow-live choices across reopening, within a 64-view/240-entry cache. Loading/failure does not overwrite them. Adaptive-run views and persistence across restart remain outside this cache; changing-width and native return checks are part of the current UI gate. | Return to each task's prior reading anchor and expanded evidence. Preserve a deliberate follow-live choice. New output while away must not jump a reader to the bottom. Include long transcripts and changing widths. Subscription cleanup is fixed separately. | -| **P2** | **Keyboard, narrow-screen and accessibility qualification. Partial automated evidence.** The product has keyboard alternatives, semantic state words and color fallbacks; actual screen-reader behavior was not assessed. | Run the full task journey keyboard-only, on real dark/light terminals, ASCII/no-color, narrow/tall and short/wide windows. Check focus, clipping, selection, contrast and usable recovery text. Test actual assistive technology before claiming accessibility compliance. | -| **P2** | **Responsiveness under realistic load. Measurement missing.** Passing elapsed test times are not UI performance measurements. | Define and measure input-to-paint, task-open, send-ack and reconnect latency; profile CPU/memory during long streams, large pastes, many task rows and repeated open/close. Document limits. Keep expensive work and network waits out of the event loop. | -| **P2, after correctness** | **Simplify execution policy and then optimize cost. Proposed architecture only.** Workers and short forks already use the same Agent core; file-count handoffs and overlapping routing/readers remain. | Retain one owner, current assignment, tool receipts and lifecycle. Remove duplicate decisions incrementally while preserving the production acceptance battery. Keep ordinary tools direct; internal helpers stay collapsed. Compare cost only across successful, equivalent real work with complete pricing. | - -Before release, also review the umbrella PR in understandable implementation -slices, run the required CI and platform checks, exercise installation/upgrade and -rollback, and keep the manual aligned with behavior. A draft PR or a local binary -is not a rollout. - -## Intended UX after the fixes - -- The main view is the conversation; the task list contains recognizable outcomes. - Internal helper topology is optional detail. -- Clicking a task always gives either its conversation or a clear recovery action. - The composer continuously names its recipient, and drafts belong to that recipient. -- Opening a task, returning, hiding a list and closing a client change the view, - not execution ownership. Stop is a distinct action with a visible scope. -- Questions and results are actionable without taking keyboard focus or overwriting - text. Pending sends remain visible until receipt; receipt and application differ. -- Progress, attention and delivered artifacts are separate facts. “Done” describes - the requested outcome, including its delivery form and evidence. -- A completed task can be discussed and continued with retained history. The UI - should not force the user to understand internal revival, checking or worker roles. - -This is a completion of the approved conversation-with-tasks direction, not a new -visual redesign or another agent orchestration layer. - -## Evidence and coverage - -| Journey | Evidence in this audit | Limit | -| --- | --- | --- | -| Same-window task click, type, Enter, Escape | Real Bubble Tea raw SGR mouse/key decoding plus Loopback engine client; baseline three race repetitions passed in 13.959s | Synthetic task backend; no real iTerm interaction | -| Compact task door when attention is needed | New failing-before/passing-after click regressions at 44/80/99 columns | In-memory terminal render and wire reader | -| Phone attention priority | New failing-before/passing-after state test | No physical phone/terminal screenshot | -| Old subscription release | Reproduced missing callback; corrected constructor releases exactly once | Does not certify every stream resource under long-duration load | -| Draft recipient isolation | Original desired-behavior probe failed; permanent recipient/durable-draft regressions now pass in the full UI suite at `810f6e4f8` | Final guest/send integration evidence appears below; real hard-crash qualification remains | -| Other-window task opening | Local owner Join/Watch and identity regressions now cover opening, replacement and refusal | Remote other-conversation navigation and actual-user-terminal qualification remain | -| Revision, protected branch, detach, reconnect | Independent targeted session/host/remote tests passed (1.424s/1.172s/2.223s) | Component evidence, not complete live-work proof | -| Code generation quality | Prior native trial: 159+61 checks; 823 public cases and 124 subtests; supplemental type-contract failure | Historical trial on an earlier runtime; one task is not a success rate | - -The post-fix navigation group passes three race repetitions (11.622s). The broader -strip/phone group passes three race repetitions (30.284s). A full offline check of -terminal UI, session, engine host, remote transport, provider and manual is recorded -in the validation appendix after completion. No test skips or performance caps -were added. - -Operator logs are under `/private/tmp/af-production-audit-20260906/`: -`navigation-baseline.log`, `confirmed-navigation-defects.log`, -`shortcut-before.log`, `shortcut-after.log`, `navigation-fixes-focused.log`, and -`full-qa.log`. Permanent shortcut/subscription regressions are in -`internal/tui3/tasknavigation_qa_test.go` and the existing local wire tests are in -`internal/tui3/localtaskroom_test.go`. - -## Source map for implementation - -- Task entry and view lifetime: `internal/tui3/room.go` (`newRoom`, `openRoom`, - `openFarRoom`, `closeRoom`, `steer`), `roomorch.go`. -- Composer/durable draft ownership: `internal/tui3/app.go`, `input.go`, `draft.go`, - `detach.go`, `pastechip.go`, `attach.go`, `roomrefresh.go`. -- Other-window navigation: `internal/tui3/tasksplace.go` (`tasksItem.pick`), - `place_tasks.go`, `taskaway_test.go`. -- Compact navigation: `internal/tui3/taskstrip.go`, `taskphone.go`, `task.go` - (`railGroupOf`). Use their existing status projection rather than another state list. -- Message identity and acknowledgement: `internal/remote/wire_task.go`, - `client.go` (`SteerTask`, `callWithin`), `internal/session/task_room.go`, - `assignment.go`. -- Result and branch delivery: `internal/session/task_run.go` (`taskNote`, - `taskMergeNote`, `taskBranchFollowup`), `task_result.go`, `task_result_e2e_test.go`. -- Verification/cancellation/recovery: `internal/session/route_judge.go`, - `task_checks_contract_test.go`, `cancel.go`, `task_store.go`, - `internal/enginehost/persistent_test.go`, `internal/remote/tasklane.go`. -- Historical evidence: `QUALITY-20260905.md`, `IMPLEMENTATION.md`, `VALIDATION.md`. - -## Reproducing recipient and crash recovery checks - -The original disposable probe exposed a main draft being sent into task 7. -The implementation at `810f6e4f8` closes that defect; it is no longer an expected -failure. Permanent tests in `internal/tui3/recipientdraft_test.go`, -`draftkeep_test.go` and `draftcrash_test.go` cover recipient switching, caret, -paste and attachment state, write failures, ordered writes and stale legacy -exports. Run them with no model key: - -```sh -env -u OPENROUTER_API_KEY go test -race ./internal/tui3 \ - -run 'Test.*Draft|Test.*Recipient|Test.*Composer|Test.*Paste|Test.*Reunion|Test.*Recover|Test.*Save' \ - -count=1 -timeout15m -``` - -Those draft-only tests predate send integration. The final integration below adds -persist-before-send and guest-recipient coverage; they remain separate assertions. - -## Validation appendix - -The first full offline check passed the terminal suite in **469.535s**, engine -host in 15.412s, remote transport in 19.608s, provider in 53.609s and manual in -2.580s. Session failed in 190.561s for two recorded reasons: - -- `TestSwappingTheModelTellsTheBeat` competed with the real background beat for - its notification queue. With all audit changes stashed, ten ordinary repetitions - passed, but 100 race-enabled baseline repetitions reproduced the failure. - `SetModel` notifies synchronously; the test now checks the locked receipt log - rather than removing a message intended for the beat. Both routing-on and - routing-off tests pass 100 race-enabled repetitions in **3.197s**. No production - routing behavior, deadline or skip was changed. -- The session checkout guard detected the newly created audit document during - execution. That file was the audit's own edit, not an engine Git mutation. The - subsequent session check freezes the whole checkout, including documentation. - -The failed combined check remains failed evidence. Terminal source was unchanged -after its full passing run. The follow-up -`make check PKGS='./internal/session' TEST_FLAGS='-count=1'` passed: full session -suite **187.739s**, whole-tree vet and formatting, packed manual **1.585s**, canonical -build, and size **50,569,250 / 54,600,000 bytes**. Its log is -`session-build-qa.log`. The notification diagnostics are -`model-beat-baseline.log`, `model-beat-baseline-race.log`, and -`model-beat-fixed-race.log`. `make changelog-check PR=653` passed (256 well-formed -entries); `git diff --check` passed. No new live-model evaluation or actual-user -terminal qualification is implied by these checks. - - -## Follow-up quality and UX wave — September 6 - -The completed integration revision `1e0b4fd4b` adds three reviewed changes: - -- A task page's footer reports that task's state, including waiting and stopped - states, instead of the main conversation's state and clock. Copy feedback keeps - priority. Leaving the task restores main-chat status. Focused race tests cover - 44, 80 and 120 columns and disagreeing task/main states. -- Retained-branch notices for stopped, conflicted, moved and detached work offer - inspection without directing an unrequested merge or checkout change. The full - completion note names the retained branch and preserves the requested workflow. - This removes conflicting guidance; a fresh substantial live workflow evaluation - is still required before closing that production requirement. -- Streaming benchmarks now begin a new turn after the fixture's settled history. - The old promoted-stream benchmark indexed `live == -1` and panicked; ordinary - streaming and append benchmarks also incorrectly fed a settled turn. The failed - initial run is retained as `ui-bench-before.log`. - -The full offline `make check` gate passed: **tui3 476.519s**, **session 173.516s**, -engine host 14.132s, remote 18.737s, provider 55.105s, manual 3.264s, whole-tree -vet, packed manual 1.458s, canonical build and binary size **50,569,250 / -54,600,000 bytes**. The invocation retained the repository's existing lock-law -skip; this wave added no skips and changed no performance caps. Log: -`/private/tmp/af-opus-production-20260906/root-quality-gate.log`. - -### Rendering measurements - -Three samples per case, Apple M3 Max / darwin-arm64, 250ms benchmark duration; -figures below are medians. Fixtures contain prior user/assistant turns, tool -outputs and, for the live-tool case, an edit diff. They are application rendering -measurements, **not terminal input-to-paint, network acknowledgement latency, -model speed, or a substantial developer-task success rate**. - -| Application operation | Median | -| --- | ---: | -| Full redraw with 20 settled turns | 0.195 ms | -| Full redraw with 60 settled turns | 0.643 ms | -| Streaming reply above 20 settled turns | 0.237 ms | -| Promoted Markdown stream above 20 settled turns | 0.214 ms | -| Live edit above 20 settled turns | 0.254 ms | - -A repeated pointer move over the same row remains about 35ns with either a one-line -or 4,000-line draft; this is the cached no-change path, not click latency. A -single diagnostic append run of 4,000 deltas allocated about 50.6MB; no optimization -or general latency claim follows from that one sample. Logs: `ui-bench-fixed.log` -and `append-bench.log` in the same operations directory. - -### Historical parallel implementation checkpoint — before integration - -At this checkpoint, three user-requested **Claude CLI Opus 5** lanes are editing -isolated worktrees for recipient drafts, asynchronous reliable sending, and -cross-window navigation plus UI organization. Their patches are **not part of -`1e0b4fd4b` and have not passed the combined gate**. Written tests are not treated -as passing tests. The first sending compilation failed and was returned for -correction. The first cross-window navigation patch passes focused race tests in -UI/remote/host/command packages, but older-engine negotiation and binding later -reads to their original conversation still need review. - -The populated Tasks fixture confirms excessive identity/status marker variety, -internal type labels competing with progress, lengthy header prose, and failed -work grouped under `done today`. Needs-attention and running sections already -precede history on that page; the organization pass must preserve that useful -order and check the rail, home, mixed families, focus and narrow layouts. One -subdued identity marker is being implemented. No updated actual-user terminal -or release is implied: the existing iTerm inspection restriction and old-running- -binary findings above still apply. No new live codeaf developer-task evaluation -was run in this follow-up; the authorized Claude CLI work is separate. - -## Recipient draft integration checkpoint — `810f6e4f8` - -The Claude CLI Opus draft implementation was reviewed, corrected and integrated. -Each recipient owns text, cursor, compact paste blocks, attachment paths and a -slot for unresolved sends. Structured JSON is authoritative; the plain text file -is a legacy export. An empty record prevents cleared or submitted text from -returning after a crash between those writes. A newer failed save retires older -queued writes. Unknown records are preserved, missing paste blocks block sending, -and write failures are shown. No new skips or storage caps were introduced. - -The focused race suite passed in **46.655s**, including the coordinator's two -previously failing stale-export crash cases. The integration's full offline -`make check PKGS='./internal/tui3 ./internal/manual' TEST_FLAGS='-count=1'` passed: -**UI 497.447s**, manual 2.015s, whole-tree vet, packed manual 1.568s, canonical -build and **50,653,298 / 54,600,000 bytes**. Logs: `drafts-focused-third.log` and -`drafts-integration-gate.log` under the Opus operations directory above. - -The broader navigation review deliberately retained five failures in -`navigation-focused-third.log`: missing guest-loss explanation, task-row cost -priority, duplicate live/history work, and two observer/protocol tests. Sending -review also found a premature recall entry and an acknowledgement that preceded -durable message identity. These are being corrected in separate worktrees; their -changes are not in this checkpoint. No live-model quality or actual-terminal -qualification follows from the draft gate. - -## Integrated navigation checkpoint — `5460283a0` - -The local task-owner callback, protocol-12 Join/Watch enforcement, bound status -subscription and organized Tasks list are integrated. The canonical build passes. -Focused combined ownership tests passed with race detection: UI 17.621s, remote -2.951s and manual 3.987s. All repository structural laws pass after registering -the new waiting command in the test harness. Final manual retrieval and untagged -end-to-end wording checks also passed in the navigation source worktree. - -This does not close the user's chat-sidebar complaint. The user specifically -identified the right-hand task column inside chat; an additional Opus pass is -checking full-row clicks, explicit folding, mixed families and overflow there. -The full combined gate will run after reliable-send and sidebar integration. - -CLI help, command dispatch and the fetched dev source did not reveal a built-in -screenshot command. The repository's `scripts/frame.sh` is a fixture capture -script, not a confirmed native CLI capture command. The user was asked for the -command they recalled. Interim images are labelled application-render fixtures; -none show or operate the user's terminal. - - -## Final integrated sidebar and delivery wave — `48201dce2` - -The user-requested Claude CLI Opus lanes are integrated, including root review -corrections. The right-hand chat task column is now the explicitly tested surface: -visible rows open their task; a fold only acts where its disclosure is drawn; -the inactive resize seam at 100–119 columns no longer swallows a click; and the -column stops capturing clicks at the composer/footer boundary. Attention and -running work precede settled work within the existing hierarchy. Redundant `◆` -markers are removed from this task-only column, with titles and detail lines aligned. -Repeated sidebar clicks and Enter retain the same room, draft and reading position. -A guest task with the same number cannot select or toggle the local task. - -`railwire_test.go` derives its mouse coordinates from the actual rendered sidebar, -feeds raw SGR bytes through Bubble Tea, and requires the exact parent then child -identifier and local engine transcript at 100/119/120/160 columns. It failed before -the sidebar patch. The first integrated run then exposed a test-coordinate bug: -the child label also appeared in the transcript header, so the helper now restricts -its independent search to the drawn sidebar columns. The corrected exact-task, -repeat-selection and colliding-owner race tests pass in **3.496s**. Both failed -runs are retained (`rendered-rail-before.log`, `chat-rail-root-focused.log`), alongside -`chat-rail-root-recovery.log`; no assertion was weakened to “any task opened”. - -The composer outbox now writes the actual pending message snapshot before sending. -Per-record ordered writes keep queued identities tied to the record that landed; -a newer empty or superseding record cannot authorize an older transmission. -Unanswered delivery preserves identity, while definitely refused text remains a -recoverable draft. Owner/session replacement is checked at durable admission. -A joined reader is bound before attachment, cannot take the keyboard, and cannot -use another conversation's task number after replacement. Pipe joins refuse -before engine boot. A finished guest page continues to detect owner replacement, -then keeps the last known state and ignores late reads. - -These are guarantees about direction admission, not exactly-once tool effects. -Some lifecycle draft writes still run synchronously. General per-task scroll -restoration, hard-crash/external-effect recovery and a fresh substantial developer -campaign remain in the release list above. - -The first combined check remains **failed** evidence: UI **543.040s** failed on -three cases. An unnamed-window test accidentally inherited the newly named fixture's -transcript; it now explicitly reopens an unnamed window. The tree-keyboard test -expected a finished child ahead of a running child; it now asserts the first -visible running child. The third was a product defect: refused owner attachment -showed only a status line instead of opening the promised fallback card. Refused -and wrong-owner attachments now open that read-only card with the owner details -and reason. The full session **172.355s**, remote **21.025s**, host **16.351s**, -CLI **49.594s**, manual **3.621s** and untagged e2e **2.294s** suites passed in -that same run. Log: `final-combined-gate.log`. - -The final read-only Claude Opus review found two additional colliding-owner cases: -local design progress/completion could alter a guest page and forward navigation -could stay on a guest when the local task had the same number. Both were reproduced -in `guest-collision-before.log` (**0.944s**, failed), then fixed with owner guards. -The weaker raw-wire click test now requires task 1 rather than any opened task. -Combined focused recovery passes with race detection in **4.951s**; full manual -**1.740s** and untagged e2e **0.526s** pass, including the new double-click help -probe. The stale manual sentence promising click-to-close was corrected as well. -Logs: `final-review-recovery.log`, `final-manual-recovery.log`. The earlier focused -manual command matched no tests and is not counted as manual coverage. - -The final `make check PKGS='./internal/tui3 ./internal/manual ./internal/e2e' -TEST_FLAGS='-count=1'` passes on `48201dce2`: full UI **543.344s**, manual -**1.507s**, untagged e2e **0.610s**, whole-tree vet/formatting, packed manual -**1.487s**, canonical build and binary size **50,821,794 / 54,600,000 bytes**. -The backend packages above were unchanged by the UI recovery fixes and were not -needlessly rerun. The checkout, including documentation, remained frozen during -each guarded check. Log: `final-ui-recheck.log`. No test skips or performance caps -were added; the repository's existing lock-law skip remains unchanged. - -Final chat and open-task render fixtures are under -`/private/tmp/af-opus-production-20260906/chat-rail-final/` at 100, 120 and 160 -columns, with ANSI originals and PNG views. Their labels explicitly identify -application fixtures, not the user's terminal. The image renderer's font does not -cover every spinner glyph; those boxes are not proof of a product font defect. -No built-in native CLI screenshot command has been confirmed. The repository -capture helper is `scripts/frame.sh`, whose private-terminal approach is distinct -from inspecting the user's blocked iTerm window. - - -## Real-terminal state audit and multi-module trial, follow-up - -The tested baseline `e4a614d15` was launched in a private terminal created for this -work (`tmux -L af-state-ux-20260906`), with isolated codeaf state. This did not -inspect or control the user's existing iTerm window. Captures use the same native -terminal-pane mechanism as `scripts/frame.sh`, preserving both ANSI and plain text; -PNG renderings are derived from those captures. Seeded-demo frames are labelled -separately from the paid live run. Evidence root: `/private/tmp/af-state-ux-20260906`. - -The live dutylog run exercised a four-module CSV-to-weekly-report repair through -real chat, durable-task admission, sidebar mouse entry, task transcripts, audits, -repair and parent integration. Captures cover 160, 120, 80 and 60 columns. This is -a moderate integration/UX trial, not a new DeepSWE comparison or a quality ranking. - -Independent outcome: all **29 judge cases pass**, and all **7 protected files are -unchanged**. The requested `fix/dutylog-pipeline` branch exists and `main` remains at -the input commit `452d8199ae0c47eec55233a475ef755ed010abcd`. **The workflow failed:** -HEAD remains at that input commit, with the repairs staged and no requested final -commit. The first task's audit rejected a non-importable regression suite. The -parent's cherry-pick was then treated as broad editing and automatically handed to -a second task, whose new worktree did not match the integration brief. That task -ended with a canceled-stream error. The final screen was idle with two task reports. -This result blocks a production-readiness claim despite passing repair behavior. - -The guard recorded 125 admitted requests and 124 settled receipts: 97 completed, 9 upstream errors, 4 upstream -failures, 1 lost connection and 13 client disconnects. Known returned cost totals -$0.198172850036; **26 settled requests have unknown cost and one admitted request has no settled receipt**, so this is a lower bound rather -than a paid total. The displayed footer cost was lower and is not the billing source. -Some fixture/generated bytecode caused cleanup churn; do not generalize that latency -into a runtime or model performance score. - -This follow-up fixes reproduced state/navigation defects: - -- A timed task proposal said `waiting · your call` while it automatically approved. - Its footer now says `starting task`; required-input presence uses the real deadline. - Holding the clock asks for input, and a separate required question still surfaces. -- A foreign finished task could expose and answer a same-numbered local task's - settlement card. The shared room decision lookup rejects guest ownership. -- A conversation-level question reclassified unrelated workers as needing a person. - The task list now uses each task's own reading, preserving complete live facts - rather than losing design approvals through a historical index row. -- A retained branch alone demanded an unrequested merge. It remains inspectable; - conflicts and unresolved human reviews retain their attention state. -- Successful model review handoffs and automatic settlement now say `awaiting review`. - Failed handoffs keep the human choices. A working parent alone never hides a child - decision; guest ids cannot borrow this local handoff state. -- Adaptive fuel gates now publish a root-only pause with live replay and clear it on - answer/settlement. Child workers retain their states. Pause and answer publication - are serialized, and a late root progress update cannot reopen a settled run. - -Release work remaining from the live trial: prevent final integration from triggering -another durable task; verify successful commit delivery end to end; diagnose the -provider retry/disconnect behavior and its accounting gaps; make first-run setup -respect `--one-model`; verify status and task clocks during audit/repair; broaden the -real-work matrix to a substantial external issue, interruption and reconnect. Existing -14-workstream priorities above remain open unless explicitly discharged by evidence. - -Focused tests passed during implementation; the full final gate and rebuilt-terminal -confirmation are recorded in the next validation addendum when complete. - -The first full follow-up command included a nonexistent `internal/host` package; -that setup error is preserved in `full-check.log`. The real `internal/enginehost` -package passed separately (14.108s). The named session (368.896s), remote (21.498s), -orchestrate (2.065s), manual (9.601s), e2e (3.947s) and CLI (98.844s) packages passed. -UI failed seven assertions that assumed retained branches stayed under attention; -the assertions now use an actual conflict for urgency and expand retained reports -before checking their details. All eight focused recovery cases pass (1.490s). -The final UI/manual/e2e gate below is a fresh run after those corrections. - -Final UI/manual/e2e `make check` passed after recovery, including whole-tree vet, -format, packed-manual verification, canonical build and size budget. Targeted race -checks passed (session 7.067s; UI 3.243s). A final copy correction makes the synthesized -empty-report fallback follow automatic review too; the automatic-review regression -asserts both its displayed wording and machine-owned state. The targeted final check -and structural-law results are recorded with the final commit. - -Final results: UI 597.946s, manual 2.058s, untagged e2e 0.864s; packed manual 1.605s. -The final fallback-copy regression passes (UI 1.148s; manual 1.266s), and structural -laws pass in all 14 packages (UI 21.397s). Changelog validation passes 256 entries. -The canonical binary was rebuilt after the copy correction. Reopening the saved paid -run in that binary shows both retained task reports without a false `needs you` count. -A raw sidebar click opens the exact task journal; a second click on that row keeps it -open. Rebuilt replay captures cover 160/120/80/60 columns. These are saved-run views, -not a second successful model trial. During this confirmation the long, repeated task -composer label and raw Markdown visible in replay remained polish follow-ups; the -current wave does not claim every chat layout is finished. - -## Conversation hierarchy follow-up — 2026-09-07 - -The owner clarified the information architecture: the main chat is the task a -person comes back to; work it delegates belongs beneath that chat, with deeper -subtasks beneath their actual parents. Home already has conversation rows, but -its work preview flattened children. The Tasks place started at worker roots and -split one conversation between status sections, losing that relationship. - -The intended reading is conversation → task → subtask. Urgency orders whole -conversations while each child retains its own truthful state. Opening the -conversation must return to its main chat; opening a child must keep the existing -owner-safe task navigation. Filtering and folds must preserve enough ancestry to -explain a result. Native terminal frames and interaction regressions are required -before considering this UI follow-up complete. - -This follow-up does not discharge the live trial's missing final commit, provider -accounting gaps, or the remaining production workstreams above. - -The hierarchy is implemented using the existing conversation and task identities, -without another execution layer. Two actual Claude CLI Opus workers implemented -Home and Tasks independently; integration review corrected live parent loss, -main-chat navigation, truthful state refresh, duplicate-name click targets, -ancestor-preserving search and record return. Main chats with no delegated work -remain selectable. Home retains its three-name desktop preview; phone work bands -fold whole families. Each child keeps its own state while urgency orders the -conversation containing it. - -Native terminal mouse checks opened the main chat, folded and expanded its work, -opened an exact child, returned through the main-chat row, and searched for a live -deep descendant with both ancestors visible at 60 columns. Home's right-hand -child row opened its exact record. These are seeded UI checks, not a new model -quality trial. Captures and failed reproductions are retained in -`/private/tmp/af-chat-tree-20260907`; PNGs are rendered from native pane cells. - -Validation: the full UI/manual/untagged-e2e `make check` passed (UI 543.562s, -manual 2.166s, e2e 0.435s), including vet, format, packed manual, canonical build -and size budget. It followed corrections to legacy assertions that assumed a -flat first worker row or always-expanded live children; no skip was added. -A final native check reproduced the same main-chat return defect in Home. -The shared Home/Search/Tasks correction then passed its focused `make check` -(UI 2.083s), with a three-door regression asserting main and child draft retention. -That final adapter correction has focused coverage after the full run; it is not -represented as a second complete local UI-suite run. The final binary is -50,905,810 bytes against the unchanged 54,600,000-byte budget. - -The 40-chat / 5,120-task history benchmark measured 22.48 ms and 93.55 MB per -reading on the baseline and 15.91 ms and 53.09 MB on the final implementation -(Apple M3 Max, five iterations, fixture outside measurement). Allocation count -increased from 5,817 to 18,266. This measures local history processing, not -input-to-paint latency, model speed, workflow completion or total cost. The -missing final commit and other release requirements above remain open. - -Final structural laws passed across 14 packages (UI 21.676s), and changelog -validation passed all 256 entries. Native Home confirmation now returns from a -child view to the main conversation. The private seeded terminal was closed -when QA finished; the user's other terminals were not changed. - -## September 7 follow-up — deliberate navigation and returned-task delivery - -The colorless captures came from the operator environment carrying `NO_COLOR=1`. -The application correctly honored that variable. The isolated capture launcher now -unsets it and retains the existing true-color palette; no user setting or product color -detection was changed. Colored native ANSI captures are retained under -`/private/tmp/af-production-followup-20260907/captures`. - -The next interaction wave makes Ctrl+k browse without switching, even when the older -quick-switch preference is on. Only an explicit Enter, numbered choice, or visible row -click opens a chat. A short pause is never interpreted as modifier release. Ctrl+tab -retains optional immediate switching where the terminal sends it. The switcher now -owns mouse input, keeps the selected row visible on short frames, and provides extra -space for long selected titles. Task transcript revisits retain owner-scoped scroll -anchors, folds, and follow-live choices within a bounded in-memory cache. Loading and -failed hosted reads preserve previous bookmarks. Adaptive-run graphs and persistence -across process restarts are outside this cache's scope. - -The owned-result delivery repair was implemented by actual Claude Code CLI Opus in a -separate worktree. The parent inspected the original trial's transcript: its nested -`ceiling` record at row 97 has `seam: write`, `decision: moved`, and `taskId: 2`, after -the returned-task message at row 71. This supports the worker's critical assumption: -the failing promotion happened during the returned-result turn, not a newly typed -integration request. The new guard uses current-turn reply tags and graph ownership; -it does not classify English commands or exempt git globally. New person-typed work -still crosses the ordinary write allowance. Ownership that cannot be reconstructed -after a restart does not receive this exception. - -Focused regressions reproduce reading-state loss before the fix and pass afterward. -The early combined focused `make check` passed vet, formatting, selected UI/runtime -regressions, packed-manual retrieval, and the canonical build/size gate. That build -measured 50,923,442 bytes under the unchanged 54,600,000-byte budget. These are early -component checks, not the final full-suite result or live-delivery acceptance. A fresh -isolated four-module acceptance run and the breadcrumb/result-card integration remain -in progress at this entry; this audit does not yet claim a successful final commit or -production readiness for the new runtime. - - -## September 7 live acceptance and delivery repair - -The fresh bounded four-module dutylog trial completed in 17m30s with no steering or -retry. Its requested `fix/dutylog-pipeline` branch contains final commit -`9ac859d06f3e283ffb1f2273ec9662f1a0c143ad`; protected main remained -`2d9e399f463a8813016e9d7e37d15b04f29f0b13`. All 21 public tests and 29 independent -black-box checks passed; seven protected files were byte-identical. The optional -request for extra regressions was declined explicitly, not a mandatory scope failure. - -This closes the missing-final-commit criterion for this one fresh scenario. It does -not certify the broader quality matrix. The run recorded one top-level task and two -legitimate checking children. It did not record `dropped:delivering-own-result`, so -it is not live evidence that the new ownership exemption fired; deterministic tests -cover that exact admission-and-completion wake. - -The ledger reports $0.083 over 192 model calls, with 24 failed calls. This is unreconciled -application/provider accounting, not a complete cost comparison. Failed provider calls -were recovered and the requested artifact completed, but the substantial stall remains -a latency concern. No pricing optimization is claimed. - -Independent inspection corrected two inaccurate interpretations in the first run -report: the task directory still existed, and the unanswered calls were internal work -checks, not questions to the user. The runtime itself had removed the worktree's Git -registration before a later accept attempted another landing. A renamed branch also -left the checker asking for a stale ref. Those are runtime defects, not just wording. - -The repair restores only the Git registration and index, leaving retained files in -place. It records the actual branch at release, uses it in the first settled notice -and later reads, and follows it for a fresh checking checkout. A failed pickup stays -answerable and points to the retained folder; it does not claim a missing branch holds -the work. Tests use real repositories, including protected main, renamed/deleted -branches, older release records, loose files and a prior recovery directory. The final -repair has deterministic and race coverage; it has not had a second paid model trial. - -Native fixture QA also found that the default engine-backed and remote doors reuse one -mutable Agent handle. Keeping that handle under the old chat's identity duplicated the -new chat and could close the wrong selection. These doors now declare that shared-handle -contract, with real loopback-server and surface A→B→A regressions. Their server still -closes the previous conversation when selecting another on the same connection. -Independent background conversations work in `--no-host`; server-side multi-conversation -lifetime is an open production requirement, not something a visual tab can provide. - -The dedicated tabs header is implemented. Its current chat uses a padded highlighted -tab with brackets that remain clear without color. Task ancestry has a separate row -below. Prior one-line breadcrumb captures are superseded by the `tabs-*` captures. -Final gate and capture results are recorded below only after they complete. - -Intermediate gates caught two stale UI contracts (the switcher now says `esc cancel`, -and expanded delivery details no longer belong in the collapsed tail) and a recovery -function exceeding the existing complexity ceiling. The assertions now test the actual -collapsed and expanded surfaces; the recovery phases were split without raising the -ceiling. A separate regression reproduced parent-index changes when a retained folder -lived inside another checkout; recovery now requires its own repository root and leaves -the containing index untouched. Focused recovery and complexity checks pass (3.396s), -and the integrated recovery race check passes (4.736s). - -The first full session run also detected source-tree changes made by the parent while -it ran (a manual page and a new change entry). That was the suite's checkout guard -working, not evidence of a test Git command mutating the repository. The final combined -run freezes source and documentation until the gates finish. No new skip or cap increase -was introduced; the existing known-red lock-law entry remains unchanged. - - -The header's independent review reproduced a stale overflow cache and checked that -same-named chats retain distinct destinations. The cache now compares complete tab -identity and picker availability. Focused header/manual/e2e checks pass (17.516s, -1.584s, 0.386s); the 4,000-line scroll allocation, resize coalescing and remote-pointer -no-network laws pass (1.459s). These are component/performance laws, not measurements -of end-to-end input latency. - -A further round-trip regression exposed that the shared-handle fallback discarded its -local draft store, and simply restoring the old store could leak the outgoing chat's -plain export into an empty incoming chat. Shared selection now restores only incoming -owner slots; outgoing text, task drafts, attachments and compact pastes remain scoped. -Main caret position survives both the local keeper and persisted shared selection. -The focused correction passes (0.995s), with race coverage (2.237s). - -Native local QA passes on the canonical binary: Home selects the main chat, tab clicks -restore each unsent draft, repeated Ctrl+k does not change the active chat while reading, -a visible picker row selects its exact chat, deep breadcrumbs sit below the tabs, the -folded ancestor opens the nearest parent, and the root returns to the main chat. Expanded -results render Markdown and retain the delivery warning. Native seeded captures cover -160×50, 80×32 and 60×28. The first QA script mistook a truncated-name ellipsis for the -ancestor fold and expected an individual-card status phrase in a batch row; corrected -coordinate/fixture assertions pass without hiding a product defect. - -The default engine-backed native A→B→A check also passes. The first attempt found that -the outgoing chat's initial identity was not added to recency, so its visible tab vanished; -shared stow now remembers that identity without keeping an agent alias. The permanent -round-trip test pins both tab names, no cross-chat draft leakage, restored text and -caret. Native input inserted a character at the restored caret to verify the real editor, -and the picker contains exactly one current-chat row. The server's one-selected-chat -lifetime remains unchanged and is documented. Evidence is in `native-host-tabs-qa.json` -and `tabs-hosted-*` native captures. - -The final source is frozen for the combined gate below. Opus implemented the header in -parallel while the parent integrated runtime recovery; parent review completed the cache, -no-color selection and shared-draft corrections. Redundant worker-owned full UI processes -were stopped before the final source freeze; their partial logs are not reported as passes. - - -## Final September 7 header and recovery gate - -The frozen combined run passed session (189.663s), config (3.205s), manual -(2.526s), untagged e2e (1.652s), and command tests (50.906s). It found one stale -focus-header assertion: the selected main tab now reads `[main]`, including with -color disabled. The expected visible label was corrected; no runtime behavior -changed after the native captures. The focused regression passed (1.115s). - -The full UI rerun then passed (550.836s). Its `make check` also passed whole-tree -vet, formatting, packed-manual tests (1.665s), the canonical build and size check: -51,007,218 bytes against the unchanged 54,600,000-byte budget. All 14 packages in -`make test-laws` passed; changelog validation accepted all 262 entries. Final -header identity, no-color selection, shared/local draft and caret tests passed -with the race detector (2.825s). No new skip, relaxed assertion of ownership, or -increased performance cap was introduced; the existing known-red lock-law skip -remains the repository's gate policy. - -Native local and engine-backed fixtures passed tab clicks, stable picker preview, -owner-scoped draft restoration, caret editing, breadcrumb ancestry and expanded -Markdown results at 160×50, 80×32 and 60×28. Captures use seeded data, not a claim -about a live user's tasks. The private QA engine and tmux server were stopped -without changing existing user sessions. The final binary is built only at -`/private/tmp/af-runtime-next/bin/codeaf`; use `env -u NO_COLOR` when the invoking -shell would otherwise disable its colors. - -Evidence: `/private/tmp/af-production-followup-20260907/` contains -`final-frozen-check.log` (including the failed old-label assertion), -`final-ui-check.log`, `final-focus-check.log`, `final-header-race.log`, -`final-laws.log`, `final-changelog.log`, the two native QA scripts and their -receipts, and `captures/tabs-*`. The remaining production criteria above are -still open; this gate is qualification of the integrated change, not a release -certification. - - -## September 7 navigation-panel and New chat follow-up - -The header now uses padded stable tabs with independent close targets, a labeled -Chats control and a plus button. Main chats have a divider; task pages separate -ancestry from quieter task facts. Only actual ancestor/Back targets navigate. -Tab dismissal preserves the agent and its drafts; current shared-engine and -last-visible dismissal return Home. This does not change the shared connection's -one-conversation lifetime on actual selection or creation. - -Plus opens a selected New chat view with its own draft and recent sessions. It -creates nothing until first submit. Cancelling or switching parks its draft on -the window, and restores the existing conversation's own composer. Creation -failures leave both drafts intact. An unnamed old conversation with an unsent -draft refuses creation rather than transferring its words to a different reader. -Task return preserves node/run/guest identity; a foreign task is reacquired from -its original owner, never resolved against the local task number. Selected paste -editing happens before creation. The page hides the old task sidebar/strip and -old telemetry; a task placeholder no longer overwrites its attachment/effort row. - -Fixed tab status slots use the front UI's state and held watchers' cached -transitions, including tasks that outlive a turn. Human questions outrank work; -internal waits and countdowns do not claim a human answer is required. There is -no new animation clock, per-frame engine call, or per-frame disk lookup. The front -status scans its existing task-state map; tab candidates are bounded, but recency -scans can still traverse older dismissed entries. No end-to-end latency numbers -are inferred from these component checks. - -Actual Claude Code Opus CLI workers implemented the header/start page and the -bounded status/owner follow-ups. Parent review reproduced and fixed survivor-tab -reordering, hidden-tab resurrection, unnamed-tab close, draft crossover, missing -start selection, context return and narrow-frame defects. The initial workers' -full UI runs during integration were stopped and are not reported as passes. -One parent full gate was stopped after native no-color QA found the old compact -task strip still displayed on New chat (its press handler already refused while -the welcome surface was open). The corrected implementation received the final -frozen gate below; no old partial run is substituted for it. - -Native terminal QA uses the canonical binary and a private seeded profile. It -passes main/current/inactive/close/Chats/plus hover and click targets, inert -separators, local and shared draft/caret restoration, safe dismissal, stable -Ctrl+k preview, nested breadcrumb/root/fold navigation, start-page cancel, and -recent-chat selection without submitting the parked draft. Captures cover -160×50, 80×32 and 60×28 and explicit no-color selection. They are terminal-cell -captures rendered from ANSI, not generated mockups or live-task measurements. - -The first completed full run (551.953s) found five assertions built around the -old layout: two located the bottom input rule by taking the first rule on the -frame; the breadcrumb test used a width that now fits the complete chain; a -short-window test expected secondary metadata to remain; and a guest-composer -fixture supplied a bare prompt instead of the actual rendered composer with its -recipient and tray. Those assertions now exercise the intended surface without -changing production code. All five focused regressions pass (1.176s). The full -rerun, rather than that focused pass alone, is recorded below. - -The final frozen `make check` passed: UI 551.179s, manual 1.530s, -untagged e2e 0.624s, packed manual 1.633s, whole-tree vet and formatting, -canonical build and binary-size check (51,057,346 bytes against the unchanged -54,600,000-byte budget). All 14 structural-law packages passed, and changelog -validation accepted all 264 entries. Final close, start-page ownership, guest -return and background-status regressions passed under the race detector (3.887s). -No new skip or performance-cap increase was introduced. The existing known-red -lock-law skip remains the repository gate policy. - -Evidence: `/private/tmp/af-header-panel-20260907/` holds Opus result reports, -focused red/green regressions, `final-laws.log`, `final-make-check.log` (the five stale assertions), -`final-ui-rerun.log`, `final-race.log`, native -scripts/JSON receipts, and `captures/panel-*`. The earlier production criteria -remain open, notably hosted independent-conversation lifetime, hard-crash and -uncertain-effect recovery, full owned-process-tree stop, broader substantial-work -quality evaluation and input-to-paint/acknowledgement latency measurement. - -## September 8 reading and navigation follow-up - -The header remains at the top. Compact tabs replace vertical fences with quiet -spacing and a filled active state; no-color terminals keep explicit brackets. -Breadcrumbs use the actual conversation title and emphasize the current task. -The switcher now responds to pointer motion without changing keyboard selection, -including an explicit hover mark with color disabled. An outside click closes the -picker and is consumed before any control behind it can act. Existing close, -Chats and New chat actions keep their ownership rules. - -Conversation and task text now has a two-cell left gutter where the terminal has -room. Wrapping happens inside that width, and link, table, proposal, standing-order -and task-action targets move with their displayed words. Repeated cached frames -cannot accumulate the offset. Copy mode retains its original layout width through -a resize and removes presentation padding without removing content indentation. - -Unnamed chats show Untitled until the existing asynchronous title event arrives -after the first completed reply. Newly opened task pages immediately show a known -instruction and reported activity while records arrive; missing readers and failed -reads no longer claim a read is still in flight. Breadcrumb navigation now returns -the command which starts the transcript reader. Typed task instructions wait for -the task's event rather than expiring on an arbitrary timer, and remain scoped to -the conversation which started them. - -Actual Claude Code Opus workers implemented and reviewed the independent UI -areas, with parent integration and a separate execution-validation subagent. -Native terminal QA used the canonical binary, private seeded profiles and actual -mouse/key sequences at 160×50, 80×32 and 60×28, including no-color hover, outside -click, tab dismissal/reopen, draft preservation and ancestor/fold navigation. -A separate native right-rail click opened the intended nested task, displayed its -saved prompt and review state, and preserved the main draft on return. These -captures are ANSI terminal-cell recordings rendered as images; they are not live -developer-task qualification. Evidence is in -`/private/tmp/af-hover-20260908/`, with capture and navigation receipts. - -The first full check exposed stale layout/name assertions and a live-provider -probe using a fixture's ten-second collector. Layout tests now measure the -rendered content width and click the drawn controls while preserving their -ownership, exact prose and styling assertions. The live interrupt probe now -cancels its redirected provider request after 45 seconds and requires completion, -no error and a nonempty answer, while retaining the stop deadline and both fanout -limits. PERF.md records the change; no known-red entry was added. - -A targeted live diagnostic returned `okok` for a request for `ok`; the provider -client's response, public stream and stored assistant message agreed. This does -not establish whether the upstream model or adapter caused the wording, because -the raw wire was not inspected. It remains an answer-quality finding, separate -from the fanout gate. A read-only Opus review also identified a transport-retry -path that failed to tell the UI to discard the dead attempt; a deterministic -reproduction showed the live answer concatenating two attempts while the journal -kept only the completed one. The retry correction and its final validation are -recorded with this PR's current validation evidence. - -The production criteria above remain open where explicitly marked: independent -hosted conversation lifetime, remote other-conversation navigation, crash and -uncertain-effect recovery, complete process-tree stop, delivery under narration -failure, broader substantial developer-task evaluation and end-to-end latency -measurement. Small live interruption probes do not close those criteria, and -this UI wave does not establish production readiness or cost effectiveness. - -Integrating `dev` through `65f060d33` exposed additional runtime seams. The -combined full UI suite passed (566.277s), while session and execution tests found -issues needing correction. Whole-request verification now freezes its explicit -checks alongside the original acceptance; commands are not recovered from prose, -and old receipts do not authorize commands for a newly opened goal. The worker -result path now uses the bounded fenced-report formatter, and expanded readers -remove both current and legacy exact previews before showing the full result. -Focused acceptance tests passed (16.095s), including three race repetitions -(2.519s); report/checkpoint/protected-branch regressions passed (1.407s). - -Shutdown review found adaptive workers and namers could still write after the -owning session closed. Their accepted lifetimes now include setup, callbacks and -final settlement, with a join under the existing shutdown grace; this does not -prove termination of a noncooperative provider or every external descendant. -The execution timing fixture also scheduled a response longer than its landing -reserve near the deadline. Its script now crosses the pacing point, lets a subsequent call reach the -work deadline, and answers the landing immediately, retaining the runtime's wall -and pace-before-landing assertions. Final combined -validation is recorded in the PR rather than inferred from these focused passes. diff --git a/docs/design/conversation-runtime/PROMPTS-CONTEXT-TOOLS.md b/docs/design/conversation-runtime/PROMPTS-CONTEXT-TOOLS.md deleted file mode 100644 index 96471ff8a6..0000000000 --- a/docs/design/conversation-runtime/PROMPTS-CONTEXT-TOOLS.md +++ /dev/null @@ -1,109 +0,0 @@ -# Prompts, tools and context: implementation contract - -This complements the product plan. These are design requirements, not claims that the current branch implements them. The goal is capable autonomous workers with low coordination overhead and reliable contextual continuity. - -## Prompt structure - -Use a small stable shared policy, a foreground/task role policy, tool schemas generated from actual capabilities, and dynamic work context. Do not repeat the whole policy in each task brief. Keep volatile status out of the cacheable policy prefix. Do not put machine time, live spend, all-task state or changing captions into that prefix. - -The foreground policy should express: - -> Help the person think, decide and maintain continuity. Answer discussion and quick grounded questions here. Start sustained work using the relevant discussion and stay available. A request to implement starts work; exploring an idea does not silently change existing work. Address explicit changes to the relevant work. Bring back the usable result and material limitations. Do not supervise every worker step or ask for internal task-management decisions. - -The worker policy should express: - -> Own the assigned outcome. Use current user constraints and relevant evidence. Choose and execute the next useful work; delegate independent work when it will help. Integrate what you own. Resolve questions from available context; ask the parent when a consequential ambiguity cannot be resolved. Apply new user directions to the current assignment. Stop working when the requested outcome is reached, or report the specific unmet requirement when it cannot be reached. Return the answer/artifacts and meaningful checks, not a claim of being done. - -A worker is not told both “nobody can answer you” and “ask your parent.” Neither prompt requires a fixed prewritten plan for every task. A plan is a revisable aid when useful. Neither prompt mandates research until certainty or a blanket full-suite review for tiny work. Model thinking is not published as progress. A caption describes current work and is not promoted to a factual conclusion. - -Generate tool/default/limit descriptions from actual runtime constants. Keep model-facing terminology consistent with actual APIs and the manual. A capability absent at runtime is not promised by the prompt. - -## What goes where - -| Information | Foreground | Task owner | Child | Durable storage | -| --- | --- | --- | --- | --- | -| Person's current message | Full | Full if addressed to it; relevant quotation otherwise | Scoped applicable direction | Original with stable ID | -| Overall aim | Current relevant understanding | Purpose plus its local responsibility | Brief purpose; local assignment is scope | Current assignment revision | -| Binding constraints | Relevant current choices | Applicable set | Applicable subset | Source, scope, revision | -| Exploratory ideas | Conversation | Only when useful, explicitly exploratory | Normally absent | Original conversation | -| Tool evidence | Selected if useful | Relevant working evidence | Required input and handles | Full result + source/version | -| Other workers' findings | Material developments only | Dependency results and changes | Relevant dependencies only | Full result objects | -| Sibling transcript | On demand | On demand | Normally absent | Producer's thread | -| Progress captions | Display | Display | Display | Diagnostic/display events, not facts | -| Result preview | Display | Display | Display | Derived from full result | -| Full result | Available by handle | Its own + needed dependencies | Needed inputs | Bounded body, overflow reference, checks and delivery | -| Budget | User-relevant total when known | Remaining family and local allowance | Local allowance within family | Exact shared ledger | - -No task receives the entire conversation state card. No child inherits an obligation to complete every word of the ancestor's multi-part request. Original words stay accessible as source; local ownership and applicable global constraints remain distinct. - -## Handoff - -Task admission is one operation, regardless of whether it comes from a typed command, model delegation, division, automatic handoff, continuation or resume. It records the current assignment, source message IDs, relevant inputs, tools/effects policy and budget. The same compiler constructs context at all doors. - -Prefer explicit decisions, results and source-linked observations. Where legacy messages are used, label them as quoted conversation, preserving uncertainty and speaker. Never turn the first sentence following a tool call into something “already established.” A model may read the source when a quote is stale, incomplete or ambiguous. - -Before a child starts: retain current constraints, exact input references, dependency results, known failed attempts worth avoiding, and the question it owns. Exclude unrelated conversation, raw sibling logs and repetitive narration. Select by task relevance and recency with one overall size budget; deterministic extraction is preferable to paying another model merely to restate an already sufficient brief. A model may retrieve more detail on demand. - -## Tool access by responsibility - -| Tool family | Foreground | Working task | Waiting task | Finalization | -| --- | --- | --- | --- | --- | -| Read/search/inspect | Available for useful grounding | Available within relevant access scope | Available on a meaningful wake, no polling loop | Relevant evidence/check inputs | -| Write/edit/execute | Short bounded operations only; sustained execution delegated | Available according to assignment's effects/ownership | No new effects while waiting for an essential answer | Only completion/check/authorized delivery effects | -| Delegate | Available | Available under family budget and cycle/ownership rules | Not used to evade an essential blocker | No new unrelated work | -| Ask/respond/steer | Address relevant work | Parent/children and direct user input | Reliable correlated answer delivery | Revision invalidates stale acceptance | -| Result/evidence retrieval | Available by work handle | Own and relevant dependency sources | Available without model polling | Full result and exact checks | -| External accounts/media/etc. | Only configured capabilities | Only configured and appropriately scoped capabilities | No speculative access | Same authorization as actual effect | - -Use capability presence and effect permissions, not hundreds of brittle prompt prohibitions. Do not dynamically churn the tool list on every step; stable capability profiles plus runtime effect checks preserve caching and clarity. A changed profile becomes effective at a request boundary. Denied operations return an actionable structured reason and permitted route, not a vague failure that triggers endless retries. - -Parent delegation cannot amplify permissions. A child cannot gain unrelated credentials or broader filesystem effects simply because it is a child. The original user may grant broader scope explicitly. Read-only research should not inherit unnecessary mutation machinery or Git lifecycle requirements. - -## Tool-result retention - -Keep the full original tool result durably with call ID, tool, canonical arguments, status, source revision where available, and artifact pointers. The model-facing view can be bounded. If truncated, say what is omitted and how to fetch a range or full result. Preserve errors, small decisive outputs, identifiers, paths, and useful counts. Separate stdout/stderr and exit status for commands. Do not infer success from a positive-sounding output string. - -Large outputs should be paged or queried; handing all bytes to the model and later paying to summarize them is usually wasteful. A repeated call after changed inputs can be valid. Cache only with a sound input/version identity and invalidate on relevant effects. A build after an edit must not be flagged as unnecessary repetition. - -## Compaction - -Compaction changes the working view, never the authoritative assignment or result. Protect current user constraints, unhandled steering, pending questions, active dependencies, result references and the recent working exchange. Retain enough failed-approach/evidence context to avoid uncaused rediscovery. Preserve uncertainty. - -Retire bulky consumed tool bodies before valuable decisions. Keep an explicit pointer and factual metadata. Do not treat tool output as consumed solely because a write occurred elsewhere. Preserve assistant text accompanying calls, multi-call causal order and error results. Never duplicate journal replay as fresh execution. - -Use deterministic reduction first. Use a model-produced checkpoint only when genuinely useful; label it as a fallible working summary and preserve source references. Before accepting a model summary, mechanically verify required constraint/question/result IDs were retained. This checks coverage, not semantic truth. A valid next request must still fit with tool schemas and reserved response capacity. Test real small and medium windows; target arithmetic alone is insufficient. - -On resume, rebuild from authoritative current records plus relevant working context. If the last mutation may have completed, reconcile effects before replay. Display summaries are not recovery state. - -## Completion and checking - -The worker produces a full result and states the current assignment revision it satisfies. The runtime distinguishes completion from cancellation, budget exhaustion and waiting. Do not encode completion as a stop-string routed through a threshold failure path. Natural final responses may complete a simple task without another ceremonial model turn; an explicit completion action, if used, must share that exact finalization path. - -Before final delivery: resolve owned children, evaluate applicable checks, confirm result revision and target artifact preconditions, and persist delivery receipt. Checking is proportional to the task. Prefer deterministic checks for deterministic requirements. Qualitative judgment can use one targeted independent review or the owner’s integration judgment when it adds information; avoid a universal chain of critics. - -Preserve full results before rendering short cards. Dependency input, continuation and main-chat foldback use full relevant result/evidence records, not card text. Repair only unmet requirements and reuse valid work. A changed instruction during checking invalidates the stale acceptance result. - -## Acceptance cases - -- A command appearing after three introductory lines survives producer → owner → main → next child → restart. -- A child receives its scope and the latest user correction through every admission door. -- A progress caption is never treated as an established finding. -- A 21st correction survives a context budget while redundant older facts are dropped. -- Text plus a new tool call preserves the preceding result's interpretation. -- Large tool output remains fetchable after compaction, and errors retain actionable detail. -- A question asked after a child join is delivered and answered exactly once. -- Tool permissions cannot broaden through delegation; a legitimate task has the tools it needs. -- A task completes with the same semantics with review on, off or inconclusive. -- Small research, coding, writing and data tasks avoid unnecessary naming/planning/reviewer model rounds. - -## Review resolutions: precise runtime boundaries - -These refinements follow an independent Opus review of the design. They clarify implementation; they do not claim those mechanisms are already shipped. - -1. **Scoped direction and authority are separate records.** A user-origin constraint names its scope, source message and revision. Agents cannot silently rewrite it. A person can give an explicit broader instruction from a child conversation; the acknowledgment names the affected assignment, and the owner receives it. There is no requirement to navigate back to main. Effective tool authority comes from authenticated user grants, inherited restrictions, the task's effect scope and host policy. Delegation alone never widens it. A new user grant can be accepted in any view, subject to the same host policy and clear target scope. Quoted documents/tool output cannot create a user grant. -2. **Completion has a linearization boundary.** Finalization records the assignment revision it satisfies and atomically compares it with the current revision before accepting a result. An outdated result is retained as superseded evidence, never reported as satisfying the newer request. External delivery is not magically transactional: persist a delivery intent with revision and idempotency identity, check target preconditions, reconcile the receipt, and report a later change as follow-up work if dispatch already occurred. Do not claim cancellation undid an external action. -3. **Waiting is scoped to the dependency.** An unanswered question prevents only work that requires that answer. A waiting parent does not recursively pause independent children. Explicit user pause/stop has separate subtree semantics. Questions and invalidated dependencies can wake a sleeping owner; requiring a nonexistent next natural turn would recreate deadlocks. -4. **Coordination is measured and optional embellishment is bounded.** Record helper naming, summarization, status narration, retries and reviews separately from productive solving/integration. Do not mandate them at every task depth. Optional recaps/checkpoints fall back to durable structured facts and references when their allowance is exhausted. Do not remove mandatory quality or authorization checks to satisfy a coordination-cost quota. Integration needed to produce the requested result is real work, not automatically overhead because a parent does it. -5. **Freshness has a mechanism and an uncertainty state.** Where available, evidence identifies its file content/version, repository revision, immutable document version or fetch time. Known overlapping writes invalidate matching observations. External mutations may be unknown; freshness is not guaranteed by a timestamp. Before consequential reuse or delivery, compare relevant current preconditions. A stale observation remains labeled with its source/refetch path, rather than silently becoming truth. -6. **Foreground budgets preserve the kind of request.** A lengthy assessment can move to background assessment without becoming implementation. A handoff preserves current work and intent, with no duplicate effect. Acceptance and execution have different receipts: queued work is not labeled running. Immediate local acknowledgment is a UI guarantee; provider response latency cannot be guaranteed by a prompt. -7. **Comparisons isolate different causes.** Fixed model/effort comparisons can establish both time and cost differences caused by fewer calls, shorter context or less wasted work. A separate model-allocation sweep evaluates additional tradeoffs. Both use frozen task/rubric versions and include failures, retries and unknown costs. A cost win does not require changing model allocation, and no one-off result proves a universal Pareto frontier. diff --git a/docs/design/conversation-runtime/QUALITY-20260905.md b/docs/design/conversation-runtime/QUALITY-20260905.md deleted file mode 100644 index ec502e4e74..0000000000 --- a/docs/design/conversation-runtime/QUALITY-20260905.md +++ /dev/null @@ -1,123 +0,0 @@ -# Production quality checkpoint — 2026-09-05 - -For the subsequent task UX audit, reproduced navigation defects, and prioritized -release requirements, see [the September 6 production audit](PRODUCTION-AUDIT-20260906.md). - -Production readiness is **not established**. Correctness and predictable task -behavior take priority over cost. This is one substantial repository case, not a -representative success rate or a performance ranking. - -## Current evidence - -The native recovery trial implemented the `dry-python/returns` Validated feature -from base `41607fae1289de2787523c452d75212206b9c7c0`. It ran unattended on codeaf -`79b2d48b0`, using only `deepseek/deepseek-v4-flash-0731` at low effort, native ARM64, -two CPUs and 8 GiB. It ended idle after **24 minutes 48 seconds**, with no permission -questions. The prompt and environment are frozen separately from earlier trials. - -| Check | Result | What it establishes | -| --- | --- | --- | -| Delivered feature acceptance | 159/159 | Requested benchmark behavior | -| Focused regressions | 61/61 | Benchmark regression coverage | -| Grading excluding candidate tests | Same 220/220 | Added tests did not supply the score | -| Independent public suite, excluding candidate tests | 823 passed, 6 expected failures, 124 passing subtests | Existing public behavior, 34.85 seconds | -| Independent full package type check | Clean, 118 source files | The configured checker accepted the delivered source | -| Additional typed interface probe | Checker accepts; execution raises TypeError | A real defect beyond the benchmark's coverage | -| Requested branch workflow | Failed | Parent merged the task branch into main without being asked | - -The public suite uses `python -m pytest tests/ -q -p no:cacheprovider --no-cov`. -The untouched baseline fails its existing coverage threshold; this command -preserves its import/doctest configuration. Candidate-authored tests were removed -for the independent public check. The type check is `python -m mypy returns`. - -The automatic final review could not reach a conclusion. Its checks list was -empty despite test commands appearing in the acceptance prose. The runtime kept -the task branch as intended; the parent then used the shell to merge it into -main. Passing tests do not excuse this workflow violation. No external push was -observed in the candidate trial. - -## An interface defect the benchmark misses - -This standalone probe imports the real package types, with no `Any` escape: - -```python -from typing_extensions import Never -from returns.interfaces.failable import FailableN -from returns.validated import Invalid, Validated - - -def recover(container: FailableN[int, int, Never]) -> object: - return container.lash(lambda error: container.from_value(error + 1)) - - -value: Validated[int, int] = Invalid((1, 2)) -print(recover(value)) -``` - -The callback promised an integer receives a tuple. Checking the entire source -and probe together with incremental checking disabled confirms the imports are -resolved: mypy accepts all 119 source files, but execution raises `TypeError`. -The candidate suppresses incompatible interface overrides; green type checking -alone therefore does not establish substitutability. - -The same runtime failure occurs in the benchmark's reference implementation. -Its full-source type check additionally reports three errors in two files. -This is a benchmark coverage limitation, not evidence that codeaf alone introduced -this kind of defect. These supplemental probes do not rewrite the frozen score. - -The committed supplemental check is `bench/deepswe/validated-contract.py`. -It requires both rejection of an incompatible caller and successful type checking -and execution of a compatible caller, alongside clean library source checking. -It fails for both frozen Validated implementations. A control using the existing -Result implementation passes all four conditions, including rejection at the -negative call site. These controls run without network or model calls. - -## Runtime fixes and remaining work - -The current wave repairs task opening and transcript refresh, missed deadlines -while workers are silent, malformed shell arguments becoming unreadable permission -cards, and false switching status when no second model request starts. Automatic -routing now carries declared verification commands through the existing task -contract. Changed requests, composed commands and partial handoffs do not acquire -execution rights from stale or whole-request checks. Worker receipts remain -evidence, not permission to repeat arbitrary commands. - -Kept-branch completion messages now tell the parent to inspect and test the work -there and preserve the requested branch and review workflow. The protected-branch -sentence no longer invites an unsolicited merge or claims shell access cannot -change that branch. This is guidance to the model, not a general Git restriction; -live compliance remains to be established. Real repository tests cover the kept -branch and inconclusive-review paths. The final session/manual make-check passes -(185.713 and 1.982 seconds), with vet, formatting, packed manual, build and size. - -Task workers and short forks already instantiate the same Agent core. The simpler -direction is to retain one owner and one request, use ordinary tools directly, -and give independent work a child only when useful. Further nested tool managers -would not fix the observed failures. Overlapping handoff/routing policies remain -a consolidation target after behavior is dependable. - -Still required before a production claim: - -- Preserve the user's branch/review instructions through task completion and - parent follow-up; validate actual behavior on a live run. -- Exercise public interface contracts beyond happy-path acceptance tests, without - blanket typing suppressions, and add more substantial repository cases. -- Repeat end-to-end work on the corrected verification handoff, including failure, - cancellation, correction and recovery paths. -- Verify the updated terminal build in the user's actual task. That existing - terminal has not been restarted or changed by this work. - -## Evidence locations and comparison limits - -On Spark, `~/bench-artifacts/af653-native-recovery-20260905` contains the frozen -manifest, `codeaf-recovery/door.json`, delivered patch, primary and supplemental -grades, `delivered-public-check`, `type-check-delivered`, and the -`type-contract-resolved-{delivered,gold}` diagnostics. The candidate never receives -the hidden tests or reference implementation. Prepared image and binary digests -are recorded in the manifest and the chronological takeover report. - -All 202 recovery-trial calls settled and used the pinned model; only 192 have -prices. Total cost is unknown. The earlier blocked native trial remains failed -at its 45-minute cap, with 0/159 acceptance and 61/61 regression checks. Prior -emulated comparisons and an unlanded-child diagnostic remain separate evidence. -No cost advantage, production success rate or Pi performance ranking is claimed. diff --git a/docs/design/conversation-runtime/REAL-WORK-EVALUATION.md b/docs/design/conversation-runtime/REAL-WORK-EVALUATION.md deleted file mode 100644 index 771b89a079..0000000000 --- a/docs/design/conversation-runtime/REAL-WORK-EVALUATION.md +++ /dev/null @@ -1,97 +0,0 @@ -# Real work, with an independent judge - -Inventory and decisions checked on 2026-09-05. This supplements the execution plan; -it is not a new result table. - -## Use the two existing rigs - -`bench/conversation` already compares codeaf, Pi and OMP through print and real -tmux conversation doors, with frozen campaigns and attributable model billing. -Its small fixtures are good diagnostics but cannot establish performance on large -repository changes. - -`bench/deepswe` already supplies containerized repository tasks and a separate -network-disabled grader. Spark has all 113 task directories at -`/swe-pro/tools/deepswe-bench/tasks`, in repository revision -`ecf7638b3705c2e08d2b10e38fcca8c70ee73f70`. Absence on the Mac is not absence of -the corpus. Open PR #634 adds a chat door to this rig; inspect and reuse that work -before implementing a second container/chat adapter. - -The integration gap is comparison adapters and comparable conditions. Add Pi and -OMP behind the existing container execution boundary, retaining the conversation -rig's inference guard and receipt accounting. Do not copy the corpus's hidden tests -or reference patches into any agent container. Freeze the task files and actual -image digest, not just a mutable image tag. - -## Two complementary sources - -Actual GitHub defects supply clear provenance and realistic reports. The initial -five local-repository candidates cover jobs on reopen, spending display, usage -frames, named-file scope and JSON error reporting. They remain diagnostics from -one Go repository. A twelve-line fix is a real issue but not evidence of complex -parallel execution. codeaf working on its own code may have product-specific -advantages; disclose them and include unrelated repositories. - -DeepSWE supplies broader repository feature work. These are benchmark tasks with -repository provenance, not automatically original GitHub issues: require the -source issue URL before labeling a task an actual GitHub issue. Initial candidates -include: - -| Repository task | Frozen source base | Why evaluate it | -| --- | --- | --- | -| `unjs/ofetch`: per-origin circuit breaker | `dfbe3ca4ef8a22fc023fca5a5ef530e525f5e523` | Stateful behavior, failure handling and regression control | -| `vadimdemedes/ink`: grid box layout | `0cea59169ef0f3f83e4aa7fbedbff9d165646472` | Layout semantics and integration across components | -| `dry-python/returns`: error-accumulating Validated | `41607fae1289de2787523c452d75212206b9c7c0` | Interface compatibility and interacting laws | -| `vitest-dev/vitest`: duration sharding | `647e6ade3b99523e3a0387a65fccfe918c331236` | Scheduling behavior and existing test-runner expectations | - -These choices cover different forms of coupling, not a guarantee that splitting -them is profitable. Predeclare selection before comparing harness scores. Include -an easy control, moderate multi-file work and substantial coupled work rather than -selecting only the previous rig's five easiest cases. - -## Controls before inference - -1. Inspect the exact model-visible issue or instruction. Some GitHub reports - contain a complete suggested fix; copying the body verbatim can turn an - investigation benchmark into transcription. Store provenance separately and - record any derived prompt and its checksum. A lexical leak check is lint, not - proof that the task contains no answer. -2. Check that the starting environment works. Run the independent tests on the - base and reference. Classify behavioral failures separately from unavailable - dependencies, emulator crashes and missing reports. A missing public API that - the feature explicitly asks for differs from a test demanding the reference - fix's private helper name. -3. Protect test and grader material. Do not protect the implementation files that - the issue requires changing. Run grading outside the candidate process and - account for modified harness hooks, tests and dependencies. -4. Validate alternate implementations where feasible. A test can pass the gold - patch and still overfit its internal structure. Existing tests supply evidence, - not automatic authority over the required behavior. -5. Freeze machine architecture, toolchain, resource limits, images and network - policy for every arm. Spark is arm64; existing corpus images use amd64 - emulation. The rig has an emulation workaround, which must be recorded and - revalidated rather than silently treated as native performance. -6. Establish all-arm startup and cost capture before a scored run. A model guard - and a harness that cannot use the same capabilities is an infrastructure - problem, not a quality zero. Judge runtime and harness runtime are distinct. - -The corpus's current verifier builder uses tags and its positive-control script -does not itself prove the negative control. Neither is a sufficient freeze for a -new comparison campaign without the checks above. Cached images for ofetch, ink -and returns were found on Spark; vitest's image was not cached at inspection. -Availability is not a fresh base/reference validation. - -## Promotion to a comparative result - -Run a small all-arm preflight, then paired randomized repetitions under a fixed -cap appropriate to each complexity tier. Keep the original issue outcome primary; -test counts, patch size and number of lanes are diagnostic. Include a chat variant -with a clear mid-work requirement change and an unrelated question while work -continues. Follow-up text must add the same information through each harness's -supported interaction door. - -Report quality, billed cost, time to a usable accepted result, correct follow-up -latency and revision adoption separately. Preserve timeouts even when some files -look correct. Unknown billing stays unknown. The next optimization should address -the measured critical path or quality failure, followed by a held-out check; -increased parallel activity is not by itself an improvement. diff --git a/docs/design/conversation-runtime/RECURSIVE-CORE-20260905.md b/docs/design/conversation-runtime/RECURSIVE-CORE-20260905.md deleted file mode 100644 index 90762ba46f..0000000000 --- a/docs/design/conversation-runtime/RECURSIVE-CORE-20260905.md +++ /dev/null @@ -1,97 +0,0 @@ -# One recursive agent loop - -Decision note, September 5, 2026. This narrows the execution plan; it is not a -claim that the policy simplification below has shipped. - -The user's analogy is substantially right: a Pi-like agent loop can start other -instances of that loop. codeaf already uses `newAgent` for both durable workers -(`task_run.go`, `newTaskAgentOn`) and short forks (`fork.go`). We do not need to -replace that core, or introduce separate agent implementations for every role. - -```mermaid -flowchart TD - U[Person] <--> P[Conversation owner] - P -->|ordinary tool call| T[Files, commands, search] - P -->|spawn, steer, cancel| C[Child: same agent loop] - C -->|result and evidence| P - C -->|independent work when useful| G[Child: same agent loop] - G -->|result and evidence| C -``` - -## The small core to keep - -An agent reads messages, calls a model, executes allowed tools and continues. -Starting a child is another tool with a clear assignment. A child uses the same -loop, a selected context, actual capabilities and an execution budget. The -parent remains responsible for the outcome it promised, including integrating -and checking returned work. - -The runtime owns facts: identity, parentage, assignment revision, workspace, -execution receipts, messages, cancellation, budget and result delivery. It does -not need a second opinion to know that a command is still running or that a -child has unread results. Waiting for either should not consume model turns. - -Short forks and durable tasks may share this core without pretending their -lifetimes are identical. An isolated durable task survives reopening, can be -entered and steered, and owns its worktree. A short fork returns to its caller; -if it writes in a shared directory, scope and stable combined checks still need -enforcement. A shell process is an owned operation, not another agent. - -## What currently adds decisions around the core - -| Current path | Question it answers | Simplification target | -| --- | --- | --- | -| `route_judge.go` router and confirmation | Should this become a task? | The current capable agent chooses the existing delegation tool. Avoid a mandatory two-reader gate. | -| `writeseam.go` two-file/five-write trigger | Has direct work become substantial? | Remove file-count-based ownership transfer. File count does not establish independence, lifetime or unfinished work. | -| `checkpoint.go` sketches, ceiling and handoff writer | What remains, should it move, and how should the next worker be briefed? | One current assignment plus attributable evidence. Transfer a concrete remainder only when needed; do not repeatedly reconstruct it through separate model calls. | -| `checkpointReopen` | Is the user's request actually complete? | Keep the owner accountable to acceptance evidence and outstanding work; avoid an unconditional second model reading of every final answer. | -| `task_child_run.go` and `task_audit.go` | Is work progressing and should it land? | Keep hard limits and real checks. Use independent review when the task's risk or uncertainty warrants it, rather than automatically rebuilding a miniature management stack for every child. | - -These are deletion candidates, not proof that every listed call is unnecessary. -Some were added after real premature-completion and runaway-work failures. -Removing their behavior requires retaining those regression cases. Do not add a -new generic controller alongside them or a matrix of permanent feature flags. - -## Product behavior that must survive simplification - -- Entering a task opens its actual conversation. It does not create another - agent or depend on whether the connection has a remote hostname. -- A correction has a recipient and a receipt. Refresh, reconnect and changing - views must not lose or silently redirect the person's words. -- Stopping a parent has a defined effect on its children and processes. A - completed child does not imply the whole request is complete. -- An unavailable provider produces a readable failure and bounded recovery. - A failure must not be relabeled as completed work. -- A worker returns its actual patch, tests and unresolved issues. The parent - verifies the integrated result against the current request. A passing test - report is evidence with provenance, not a universal correctness certificate. -- Ordinary tools stay direct. The current single specialist-discovery level - is sufficient; loading a schema does not need an agent, a planner or another - task hierarchy. - -## Order of work - -First repair and exercise real task interactions and provider failure paths. -The September 5 task-click and refresh fixes remove one incorrect distinction -between local and remote windows while preserving the existing engine protocol. - -Then replace ownership decisions one at a time, starting with the arbitrary -write-count promotion rule. Retain substantial-work delegation, interruption, -revision and completion tests. Measure a capable owner directly invoking tasks -before removing the router pair; the question is whether the requested work -still completes and remains steerable, not how many calls disappear. - -Use realistic repository tasks to evaluate this change. The current Validated -pilot exposed a grader collection conflict and a compressed-error proxy defect; -it cannot justify a wholesale architecture rewrite or an efficiency ranking. -The corrected-proxy codeaf trial retains the original runtime so that those -experimental conditions are kept separate from future policy changes. It reached -the 30-minute deadline with implementation in the child worktree but no feature -integrated into the main workspace. The child was awaiting a full test run. -The delivered patch scored 0/159 acceptance and 61/61 regression assertions. -This supports investigating ownership timing, test execution and integration; -it does not establish that deleting a particular policy would have passed. - -The target is one recursive loop with a reliable task lifecycle. The remaining -engineering is mostly about deleting duplicate decisions while preserving the -facts and behavior that make that loop dependable. diff --git a/docs/design/conversation-runtime/TAKEOVER-STATUS-20260905.md b/docs/design/conversation-runtime/TAKEOVER-STATUS-20260905.md deleted file mode 100644 index c8097eea18..0000000000 --- a/docs/design/conversation-runtime/TAKEOVER-STATUS-20260905.md +++ /dev/null @@ -1,545 +0,0 @@ -# Conversation execution takeover — 2026-09-05 - -The work continues from the “codeaf clean” Codex thread, in isolated worktree -`/private/tmp/af-runtime-next`, on `codex/conversation-execution`. Draft PR #653 -targets `dev`. The shared development checkout is not this worktree. - -For the latest results and remaining gaps, read [the quality checkpoint](QUALITY-20260905.md). - -## Product direction - -One owner remains responsible for the requested outcome. Direct tools handle -ordinary actions; short forks handle independent slices; durable tasks own work -that needs its own lifetime. Loading a specialist tool is a local registry -operation, not another agent, a new task, or an inference-based routing decision. - -Use one discovery level. Keep everyday file, shell, job, task, manual and web -tools directly available. Chat may defer media, settings and saved-procedure -schemas until requested. Retain original tool names and the existing execution, -approval and accounting paths. Append loaded tools without reordering the active -list. Task workers and reduced fork/auditor belts keep their existing contracts. - -The tradeoff is explicit: less repeated schema data on ordinary requests, one -additional model request on first use of a specialist group, and a changed -provider prefix when that group loads. Byte savings alone do not establish -lower latency, lower bills, or equal success. - -## Implemented before tool discovery - -- Replacement windows receive pending questions even while the old window is - still attached. Initial welcome and replay deliver the question once, from one - coherent snapshot. Reconnect still delivers questions missed during a gap. -- A completion claim can decline handoff without a second reader explicitly - agreeing. A sketch with independent work remaining still vetoes the claim. - Every accepted claim is charged once per request, including corroborated ones; - ten more rounds of work bring the ceiling back. A new user direction changes - the request. Existing running commands retain their owner and completion wake. -- Canceling before or during handoff preparation prevents task admission; a - canceled reader does not fall through to a worker receiving the original ask. - Regression tests failed before these changes and passed afterward. - -## Prior live evidence retained - -At source `70888a56b`, the two-repeat revision diagnostic recorded one codeaf -pass and one timeout; Pi passed both. The timed-out codeaf attempt still has an -unsettled charge after a read-only reconciliation attempt. Its correct files do -not override its missed deadline. - -In the successful diagnostic pair, codeaf made 18 admitted calls and sent -910,727 request bytes; Pi made six calls and sent 56,038 bytes. codeaf's main -requests carried 34 tool definitions. These are one pair's observations, not -population estimates or an attribution of all overhead to tool schemas. - -## Still separate work - -- Automatic handoff can still trigger after two distinct files or five writes. - Completion changes reduce one bad outcome of that rule; they do not remove it. -- A stale sketch that names independent parts can still veto a current - completion claim. Consolidating ownership decisions remains unfinished. -- Wider real-repository, chat and interruption comparisons remain necessary; - a tiny fixture battery cannot establish a Pareto frontier. - -## Integrated specialist discovery - -Source `9f0ce1619` includes the reviewed discovery implementation and the other -fixes above. `load_capability` lists actual available tool names in three possible -groups; it adds no classifier or model-routing call. Errors remain tool errors, -loading preserves existing permissions, and normal tools execute under their -original names. Reopen restores load calls still present in the saved transcript; -compacted load records may need loading again. There is no separate persistence -store and no nested execution wrapper. - -Serialized measurements from complete JSON tool blocks: - -| Configuration | Before | After | Saving | -| --- | ---: | ---: | ---: | -| Standard fixture, prompt + tools | 47,606 bytes | 44,347 bytes | 3,259 bytes (6.8%) | -| Fully enabled chat, tools only | 40,595 bytes | 26,740 bytes | 13,855 bytes (34.1%) | - -The fully enabled loader itself is 708 bytes. The standard fixture includes the -heavier prompt wording for discovery; the prefix budget stays 48,000 bytes. These -are encoded bytes, not provider-token or billing measurements. The larger saving -requires the richer configured tool set and is not the default-chat saving. - -## Offline validation record - -The full `make check` run at `9f0ce1619` completed 85 passing packages, including -`cmd/codeaf`, `internal/remote`, `internal/enginehost` and the full `internal/tui3` -suite (464.828 seconds). Four tests failed across three packages: the worker -vocabulary law, the chat-manual vocabulary law, and two media-schema fixtures -that inspected the initial tool list without loading media. Those failures are -retained in `takeover-final-check.log`. - -Commit `eccef810e` corrects the wording and makes both schema fixtures load the -media group before inspecting the real active schemas. All four focused -regressions pass. The three affected packages then passed in full through -`make check PKGS='./internal/exec ./internal/manual ./internal/session'`: execution -106.290 seconds, manual 1.514 seconds, and session 183.534 seconds. Vet, formatting, -packed-manual tests, release build and size ratchet also passed. The binary is -50,552,226 bytes against a 54,600,000-byte cap. The repository's one existing -known-red exclusion was retained; no new exclusions were added. This is a full -suite with corrected affected-package reruns, not a claim that the initial -full-suite invocation exited successfully. - -The first remote CI run at `b255c23c4` passed the light gate and seven of eight -touched packages, including the full terminal suite (466.187 seconds). The -session package exposed one more fixture dependency: the load-and-call acceptance -test expected a media group without providing media, relying on a locally -installed FFmpeg. Running that test with `/usr/bin:/bin` as PATH reproduced the -failure on the Mac. The fixture now provides scripted media explicitly; the -separate absent-capability tests retain their checks of unavailable groups. This -is a test-environment correction, with no runtime change to the frozen complex -comparison build. The failed CI log is retained and the corrected gates must pass. - -## Live tool discovery smoke - -On `eccef810e`, a fresh hosted test profile received: "What is my daily budget -set to? Read the current setting and report its value. Do not change any setting -or file." The model loaded `settings`, called the original `settings` tool with -`search: "daily budget"`, and reported the returned $500 value in the same turn. -The workspace remained empty. This is the isolated fixture's setting, not an -inspection of the user's personal configuration. - -The answer used three model requests and took 18.940 seconds in the session's -usage record. Tool counts were 25 → 27 → 27. Two further requests handled -background work; one lost its usage block when the host closed. A read-only -generation lookup settled that missing bill without repeating inference. All -five admitted requests are now priced: **$0.00186847322 total**, including -**$0.00180473322** for the answer's three requests. Every request used the exact -DeepSeek pin. This proves the loading/continuation path; it is not a comparison -against another harness. - -## Current revision regression - -The frozen `takeover-revision-v1` campaign at `eccef810e` completed all four -planned cells. Both codeaf and Pi passed both repetitions, including adopting a -mid-work requirement change while preserving the already-running command. Every -assertion passed and every cell has attributable upstream billing. - -| Arm | Passes | Mean wall time | Mean billed cost | -| --- | ---: | ---: | ---: | -| codeaf | 2/2 | 85.504 s | $0.002696575 | -| pi | 2/2 | 90.343 s | $0.000496991 | - -This is regression evidence from a small fixture, not a quality ranking. Provider -endpoints varied between calls; it is a native-harness comparison, not an isolated -tool-discovery ablation. The planned synthetic four-defect pipeline campaign was -not started: the next evaluation is substantial work in an unrelated repository. - -## Complex repository evaluation in progress - -The first paired pilot is the DeepSWE `dry-python/returns` Validated feature: -container interfaces, applicative error accumulation, converters, decorators, -pointfree dispatch and Hypothesis integration. It has 159 acceptance tests and -61 regression tests. It is a benchmark feature request with repository provenance, -not a published GitHub issue. Its prompt includes implementation guidance, so this -measures implementation and integration rather than unaided defect discovery. - -Base and reference controls scored respectively 0/159 and 159/159 acceptance -tests, with 61/61 regression tests in both. The cached independent grader image -ID still matches the negative-control record. Candidate and comparator will use -the same frozen runtime on Spark: two CPUs, 8 GiB, linux/amd64 emulation on arm64, -and the documented Go emulation workaround. Each gets 1,800 seconds, the exact -DeepSeek pin, low requested reasoning effort, an isolated profile, and its own -credential-holding guard outside the container. Both use the existing tmux -conversation driver. Hidden tests and the reference implementation never enter -the candidate container. No quality result is claimed before independent grading. - -## Correctness-first task interaction pass - -The user's priority is now dependable developer workflows before efficiency; -small-task costs remain diagnostics. The default local window supplied a -`TaskRoom` reader over the engine connection but no remote host label. The UI -mistakenly required that label before using the reader, so clicking a live task -reported that the session had no task rooms. The regression failed on the old -condition and passes when the UI selects the available capability directly. - -A second regression reproduced accepted corrections disappearing on the next -journal refresh. Refresh now retains unchanged entries, preserves open/full state -when content grows, and keeps accepted corrections until their journal entries -arrive. Repeated identical corrections remain distinct. The same loading wording -works on a local engine and on a remote machine. - -A raw-input test sends SGR mouse clicks, typed text, Enter and Escape through -Bubble Tea's actual input decoder/program loop, with a real loopback engine -client. It verifies opening the intended task, reading its journal, delivering a -correction once to task 7, and returning to the main conversation. The fixture -provides the roster and scripted engine; it does not establish model quality or -claim to control the user's existing terminal window. The focused interaction -and hosted-room tests passed under the race detector. - -The media-fixture correction's full session check passed (191.714 seconds). -CI at `3768a93ce` then exposed a separate ordering assumption in the failed-handoff -fixture: its background task was not guaranteed to fail before the closing -response. The fixture now explicitly waits for graph completion. Thirty race -runs passed. That CI run's full terminal package also passed (467.450 seconds). -The new task interaction changes require a fresh package check. - -### Task interaction validation outcome - -The full terminal package passes with these changes (466.970 seconds), and the -manual package passes. The session package's run failed only its optional live -compaction smoke: the returned marker was `ROUND-04`, instead of the complete -`ROUND-04-UNIQUE-TAIL`. That test allowed only 64 output tokens, shared with -reasoning. Its allowance is now 512 and failure output includes the finish -reason; three fresh pinned-model runs pass (5.078 seconds total). No compaction -runtime behavior was changed to accommodate that model response. The final -offline session gate is run without provider credentials; live checks are -reported separately rather than making ordinary package checks depend on a -provider response. - -### Complex pilot outcome and diagnostic grade - -Both original terminal runs ended at their ready screens: Pi in 1,232 seconds, -codeaf in 1,282 seconds. Those times include startup and the quiet window; they -are not times to equivalent completed work. Both original grades are zero. - -Pi produced an implementation. Its added test package collided with the hidden -suite's module names, so the original acceptance collection never ran. codeaf's -only task failed with an upstream HTTP 404, and its final main-workspace patch -is empty. No unlanded branch was substituted to improve that score. - -A supplementary diagnostic excludes **all candidate `tests/` changes** from each -saved patch, preserves every other change and runs the same frozen verifier -without network or additional inference. Results: - -| Saved implementation | Acceptance | Regression | -| --- | ---: | ---: | -| Pi, with candidate tests excluded | 159/159 | 61/61 | -| codeaf, with candidate tests excluded | 0/159 | 61/61 | - -This diagnostic does not replace the original failed run. It establishes that -Pi's implementation passes the assertions once the collection collision is -removed; it does not repair codeaf's missing implementation. The original -patches, grades and logs remain intact. The diagnostic script and its patch -hashes are retained under `af653-complex-20260905/diagnostic-test-isolation` on -Spark; the invocation script is at that artifact root. - -The codeaf task's unreadable refusal begins with the gzip signature. The -benchmark proxy preserved compressed error bytes but dropped `Content-Encoding` -and retry headers, unlike its success path. An offline compressed-404 regression -fails before the proxy fix and passes after it. Error headers now retain the -same end-to-end contract as successful responses. This does **not** explain away -the upstream 404, prove it is fixed, or turn the failed task into a success. -The corrected proxy was not used retroactively in either scored run. - -Every admitted inference used `deepseek/deepseek-v4-flash-0731`: Pi made 76 calls; -codeaf made 120. Read-only reconciliation prices all Pi calls at $0.048953800644. -codeaf has 116 priced calls out of 120, so its total remains **unknown**. No -cost ratio is reported. Emulation also drove codeaf's UI and engine close to the -8 GiB container cap; this run used the declared `GOGC=off` workaround and does -not establish native runtime memory demand. One pair with these failures cannot -establish quality equivalence, a performance ranking or a frontier. - -The temporary credential file used by this pilot's guard was removed after both -runs and reconciliation ended. Further complex-work validation should first -resolve the provider refusal with readable diagnostics, then repeat under a -predeclared test-isolation policy. The practical priority remains opening, -steering and recovering real work, ahead of schema or small-task cost tuning. - -Final gate for the task UI wave: the full terminal suite passed above; the -credential-free `make check` for session/manual passes (session 181.771 seconds), -including vet, formatting, packed manual, build and binary-size ratchet. The -binary is 50,552,770 bytes, below 54,600,000. All 156 conversation benchmark -Python checks pass (44.738 seconds). A new complex codeaf trial will retain the -original runtime/model/limits and change the proxy's error-header behavior; -its result will be recorded separately from the original paired pilot. - -### Read failures and the recursive-core review - -A further task-page regression reproduced temporary read failures presented as -missing history. Finished tasks stopped polling after that failed first read. -Opening and refreshing now share one captured reader, show a steady retry message -on failure and retain any transcript already displayed. The focused running and -finished recovery cases pass with the local task/wire tests under the race -detector. A fresh terminal package check follows this change. - -The user's proposed “Pi spawning Pis” model is already the implementation's core: -workers and forks create the same Agent. `RECURSIVE-CORE-20260905.md` inventories -the surrounding duplicate decisions and defines the smaller target. It is a -proposal for removing policy layers, not a claim that the two-file/five-write -promotion rule or the extra routing readers have already been deleted. - -The task-read recovery gate passes: `make check` for terminal/manual, with the -full terminal suite at 467.541 seconds, manual at 1.541 seconds, packed manual, -vet, formatting, build and size all passing. The binary is 50,552,722 bytes. -Remote CI for the preceding `120806c17` task-click/refresh wave also passes both -halves (run 33984222248); the recovery follow-up is a later commit. - -An important limit on the architecture inference: the original complex run's -40-round ceiling recorded `dropped:work-already-out`. The existing ownership -check prevented another handoff there. We do not attribute that run's failure -to a forced handoff that never happened. Its recorded request traffic was -71,984,610 bytes across 120 calls for codeaf and 17,282,734 across 76 for Pi; -19 codeaf requests carried no tools. These are workload observations with failed -outcomes, not proof that every helper call is unnecessary or a cost comparison. - - -### Corrected-proxy complex retry: deadline without delivery - -The diagnostic retry completed at the 1,800-second limit (`door.json` reports -1,801 seconds and `ended: cap`). Its runtime binary hash is unchanged from the -original pilot, `593d7400b068b3182cbad72ae73842b8b06615ce8f94edd87d32be6db1b2a7aa`; -only the proxy correction was introduced. Both the original verifier and the -predeclared test-isolation diagnostic returned **0/159 acceptance, 61/61 -regression**. This is a failed delivery, not an efficiency result. - -The final main-workspace patch contains a QEMU ruff core dump, with no feature -implementation. The sole child, task 2, has implementation and test edits in its -isolated worktree, but they were not integrated before the deadline. Those -unlanded files were not substituted into either score and have not received an -independent correctness grade. The final task record is paused for resumption -after the harness stopped the engine; this does not constitute completion. - -The main conversation investigated for 10 minutes 26 seconds and 58 tool calls -before the runtime moved the work into that task. At 18:51:02 UTC the child -started `python -m pytest tests/ -q -p no:cacheprovider`, piped through `tail -30`, -with a 600-second timeout. It became background job 4 after 30 seconds. Its -result had not reached the child journal when the harness stopped at about -19:00:48 UTC. Toolchain preparation also encountered missing test plugins and a -QEMU ruff crash. These observations identify time spent and an undelivered -result; they do not isolate policy overhead from model behavior, environment -setup or emulated execution speed. - -All 126 admitted calls used the exact allowed model; 126 have settlement -records, but only 122 have prices after read-only reconciliation. Total cost -therefore remains **unknown**. Recorded request traffic is 56,331,950 bytes. The -retry is not a new paired comparison, and it cannot establish a cost ratio. -Its evidence is retained on Spark under -`/af653-complex-guard-retry-20260905`, including the -manifest, native screen, complete saved state, both grades and reconciled -receipts. The owned temporary guard credential was removed at job completion. - -The next architectural change should shorten the path from assignment to a -stable owner, preserve that owner's context, and make test progress and result -integration dependable. Additional nested wrappers for ordinary tools would -not address the failure observed here. The UI fixes in this wave have passed -their local gates, but this complex run prevents claiming that substantial -end-to-end developer work is reliable yet. - - -CI run 33985211705 passed the full terminal suite (468.315 seconds) and session -suite (162.085 seconds), but failed a reconnect fixture that required the new -window's arrival snapshot to report zero other attachments. Client socket close -does not acknowledge server EOF processing: the replacement may arrive before -the old connection detaches, as the server's detach contract explicitly allows. -The fixture now retains conversation identity, completed work and eventual lane -cleanup checks without asserting that unsupported arrival order. Separate -simultaneously attached-window tests still check the attachment count. - -The old reconnect assertion reproduced 20 failures in 300 local runs. With the -first window explicitly subscribed, all 300 revised runs pass under the race -detector (35.177 seconds). The full enginehost package passes (13.865 seconds), -with vet, formatting, packed manual, build and size gates. An accidentally -started all-package gate was stopped and is not counted as validation; the -intended enginehost gate completed successfully. - - -## Production-readiness continuation - -The preceding reconnect fix passed both CI halves (run 33986235213). The next -pass keeps correctness ahead of cost and does not change model routing, prices, -automatic handoff thresholds or task allowances. - -The sole unlanded child from the corrected-proxy retry received an offline -implementation diagnostic. Using a separate Git index, it preserves the saved -worktree and excludes all candidate tests changes, then applies the resulting -patch in the original frozen verifier. It passes **159/159 acceptance and 61/61 -regression assertions**. This does not replace the failed delivered-workspace -score. It narrows the observed failure to timely completion and delivery; the -saved implementation itself passes these assertions. Evidence is under -`af653-native-quality-20260905/child-diagnostic` on Spark. - -A native ARM64 correctness environment now uses the same repository base and -feature request, with the public Poetry-lock development dependencies and -check-laws/compatible-mypy extras installed before the run. The native verifier -passes its controls: base 0/159 acceptance and 61/61 regression; reference 159/159 -and 61/61. It retains both the raw delivered-workspace grade and a supplementary -grade excluding candidate tests, with a predeclared 45-minute trial allowance. -Different architecture, environment preparation and allowance mean this is not -a paired performance comparison with the earlier emulated runs. - -Preparing the native binary exposed a cross-build defect: `make build` inherited -GOOS/GOARCH into the manual generator, attempting to execute a Linux generator -on macOS. Generation now clears those target variables, like the existing -furrow fetcher. The final binary still uses the requested target. The old build -failed with `exec format error`; the revised Linux ARM64 build completed and -`file` confirms a statically linked AArch64 executable. - -Separately, a worker's deadline was only checked after completed tool events. -A silent model request, command or promoted job could therefore defer the -checkpoint until another event arrived. The runner now arms one timer for its -current deadline, rearms on an allowed extension and disarms when stopped or -finished. The existing progress decision, cancellation and delegated-parts pause -rules remain. A deterministic silent-worker test fails before the change. A -real-shell test confirms that an owed command can remain parked through one -renewal, with no extra worker request, then stop when renewal is refused. - -The first full session/manual check found only the structural complexity gate: -the combined timer/event drain exceeded its function ceiling. Event observation -and accounting are now a separate method from waiting on events and the clock. -No complexity allowance was raised and no test was skipped. Fresh final checks -follow this refactoring. - - -Final deadline/build gate passes: full session 181.020 seconds, manual 1.515 -seconds, packed manual 1.660 seconds, vet, formatting, build and binary size. -The focused deadline/renewal/delegated-pause/complexity cases also pass under the -race detector (three repetitions, 7.582 seconds). The binary remains under its -54,600,000-byte ceiling. No new skips or expanded complexity allowances. - -The native public baseline passed 823 tests and 124 subtests with 6 expected -failures in 55.87 seconds, but exited 1 because its default 100% coverage gate -reported 76.4% on the unmodified repository. The trial therefore discloses this -pre-existing failure and installed dependencies in a model-visible environment -note, while retaining the original feature requirements and the repository's -coverage settings. This preparation/prompt difference is explicit in the -manifest; no hidden-test or reference-solution information is given to the -candidate. - - -### Native trial found malformed calls becoming permission questions - -The first launch preflight ended before inference because the copied executable -had lost its execute permission. Review also found the initial verifier Dockerfile -retained a second FROM line pointing to the corpus image; a corrected verifier -uses exactly the prepared native base. The corrected verifier checks Python's -build platform (`aarch64 linux-aarch64`), not just Docker's architecture label, -and its fresh `native-control-base` and `native-control-gold` controls pass at -0/159 plus 61/61, and 159/159 plus 61/61. The earlier image/build logs remain. -The paid native trial started at 21:28:07 UTC on runtime `8ae9d135e` using that -corrected verifier. Its own manifest and rig snapshots identify all inputs. - -That frozen trial reached a permission card for a model-emitted bash call whose -arguments were only `{`. An unreadable command cannot be matched by approval, -so the defensive policy converted allow to prompt. This is safe against execution -but creates a product dead end: the person has no actual command to authorize. -The trial remains untouched, with that blocked state retained as evidence. - -The next runtime rejects malformed/missing/empty/wrong-type bash commands before -asking approval, returns the shared actionable argument-error wording and counts -it as a model input failure. It never executes or guesses the missing command. -A corrected command still goes through the existing pre-action and approval -chain. The regression drives an invalid call, model correction, real shell run -and receipt of its output, without a consent event; the old code fails all five -invalid-input cases. Approval policy rules themselves are unchanged. - -The native trial also exposed an overbroad environment hint: clearing every -pytest addopt disabled this repository's doctest/import setup and broke public -test collection. A new public baseline run verifies the narrower command -`python -m pytest tests/ -q -p no:cacheprovider --no-cov`, preserving repository -options and executing 823 passing tests, 6 expected failures and 124 passing -subtests in 33.49 seconds (exit 0). A later trial must disclose the corrected -hint as another experimental difference, not silently rewrite the frozen prompt. - -The invalid-shell fix passes the final full session/approval/manual make-check -gate (181.859, 0.541 and 1.477 seconds), plus vet, formatting, packed manual, -build and size. Recovery, control-byte scrubbing and consent/denial/floor tests -pass three race repetitions (12.264 seconds). The first full run exposed one -old raw-control-byte fixture expecting invalid JSON to reach a consent card; -it now asserts no consent, no execution and a safe actionable error. Valid JSON -with an escaped control byte still exercises the existing card-scrubbing path. -The preceding deadline/build revision `8ae9d135e` passed both CI halves -(run 33992899226). The first native trial remains frozen at its permission card. - -### Recovery trial and truthful waiting status - -The separate native recovery trial started at 21:55:01 UTC on `79b2d48b0`, -with binary SHA-256 `a3b59d2b5fb2919d14042f266bf6a87f7c9127ed30f739e533455c1e1d1bcbb0`. -Its environment note gives the verified `--no-cov` command. Source, prompt, -images, model and controls are frozen in -`~/bench-artifacts/af653-native-recovery-20260905/manifest.json` on Spark. -It is unattended and graded only after delivery or the declared deadline. - -Read-only observation exposed a separate presentation defect: repeated -`still no answer — trying another lane` notices occurred on calls whose own -receipts recorded `action: report`, with no rescue arm. The first-prompt report -shared the rescue wording and switching phase even when no alternative could -start. A wire-level regression with exactly one available provider reproduces -that false claim. The next runtime says `still waiting for an answer · /model -switches` and remains in the waiting phase in that case; actual rescue requests -retain their existing notice and switching phase. Provider selection, allowances -and retries are unchanged. This presentation fix is not injected into the -running recovery trial. - -The waiting-status fix passes the full provider/manual make-check gate -(51.922 and 1.760 seconds), packed manual, vet, formatting, build and size. -The real-wire wait/rescue/last-resort cases pass ten race repetitions -(11.983 seconds). The no-alternative regression fails before the change. - - -### Native recovery outcome and verification contract - -The recovery trial finished at 22:20 UTC: its terminal driver ended idle after -1,488 seconds with no consent questions. Delivered code passes 159 acceptance -and 61 regression checks under both the primary and candidate-test-exclusion -policies. Independent public tests pass 823 cases and 124 subtests with six -expected failures; independent mypy accepts 118 source files. The complete -outcome and additional type-contract failure are in QUALITY-20260905.md. - -The trial is not a product-quality pass. Automatic review had no executable -checks and its reading window expired. The task branch was retained, but the -parent merged it into main without a request to do so. The extra typed interface -probe is accepted by the checker and crashes at runtime; the reference has the -same defect, exposing a limitation in the original benchmark's quality coverage. -All 202 calls settled, but only 192 are priced. Both completed trials' temporary -guard credentials were removed. Frozen candidates and scores remain untouched. - -The automatic route verdict previously had no checks field. It now carries the -same declared verification contract as an explicit task proposal, bound to the -request read by the router and validated by the same command parser. Whole-request -checks are dropped when custody narrows the handoff to a remainder. This does not -scrape worker receipts or acceptance prose into executable permissions. - -Both automatic entrances fail before the fix and pass with it, including a real -shell verification command and refusal of an undeclared action. The full session -make-check gate passes (183.408 seconds), manual (1.551), packed manual (1.590), -vet, formatting, build and size. Final scope regressions pass ten race repetitions (3.263 seconds), and the -partial-handoff test fails with its guard removed. A final manual make-check -passes (1.475 seconds; packed 1.595) after the section heading was improved. - - -### Branch follow-up guidance and a stronger public quality check - -Kept-branch notices now explicitly preserve the person's requested branch and -review workflow. The protected-branch sentence describes automatic task landing, -removing its invitation to merge whenever ready. This is model guidance; it does -not claim to prevent arbitrary shell writes. Real repository regressions cover -protected-branch completion and an inconclusive review. An initial regression -caught the use of `cameHome`, whose meaning includes a correctly kept branch; -the new guidance now tests the actual kept/conflicted/aborted outcomes instead. -The final full session/manual make-check passes (185.713/1.982 seconds), packed -manual 1.589, vet, formatting, build and size. Focused branch/routing/settlement -cases pass three race repetitions (7.603 seconds). - -The portable supplemental diagnostic `bench/deepswe/validated-contract.py` -requires clean source checking, rejection of the invalid caller at its argument -boundary, and successful type checking plus execution of the tuple-aware caller. -It fails for both frozen Validated implementations. A positive control aliases -the established Result implementation into the same probes and passes all four -conditions. This control validates the diagnostic, not a repaired Validated. -Fleet jobs 20260905-225653-000281 and 20260905-225915-000282 retain the output in -`type-contract-paired-{delivered,gold}` and `type-contract-result-control` under -the native recovery artifact root. No model calls or network were used inside -these verification containers. The new diagnostic never replaces the frozen -159+61 scores. diff --git a/docs/design/conversation-runtime/TASK-EXPERIENCE-PROTOTYPE.md b/docs/design/conversation-runtime/TASK-EXPERIENCE-PROTOTYPE.md deleted file mode 100644 index ef892abeec..0000000000 --- a/docs/design/conversation-runtime/TASK-EXPERIENCE-PROTOTYPE.md +++ /dev/null @@ -1,141 +0,0 @@ -# Conversation and tasks: product architecture and prototype - -Status: interaction prototype, approved direction from September 5, 2026. The -runtime changes below are proposals. The prototype does not execute models, Git, -commands, or network requests. Existing runtime fixes remain on PR #653. - -## Product boundary - -The main conversation holds discussion, decisions and continuity. A task holds a -recognizable outcome, its own conversation, and the work needed to deliver it. -Both use the same agent core. The user addresses work, not an agent topology. - -Ordinary work may stay inline. Create a task when an assignment benefits from -sustained execution, a separate workspace or a separately reviewable result. -Size, number of files and elapsed tokens alone are insufficient. A user can -explicitly choose “Run as task.” Automatic handoff needs a visible receipt naming -the task and what moved, and must continue the existing assignment without -repeating completed work. The automatic policy is not simulated in this prototype. - -Opening a task changes the view, not who owns execution. Leaving it, hiding the -rail or closing a client does not cancel work. Persistence requires the execution -host to remain running; local work cannot continue while the machine is off. - -## One durable identity, several views - -Reuse the existing task identity, request revisions, journal, delivery receipts, -check declarations and result references. Do not create another task database or -rename every internal child node into a user-visible task. - -| Concern | Authority | UI projection | -| --- | --- | --- | -| Conversation | Existing conversation ID and journal | Main discussion and its task list | -| Task | Existing stable task reference, owning conversation/project | Named outcome and task conversation | -| Assignment | Current request/revision, scope, checks, workspace and requested delivery | Inspectable assignment; acknowledged directions | -| Execution | Existing attempt/lifecycle and internal work graph | Running phase, dependencies and collapsed activity | -| Result | Artifact location, source revision, check evidence and unresolved issues | Reviewable result with precise delivery status | -| View state | Client state keyed by conversation/task reference | Recipient, draft, scroll anchor, expanded entries | - -A task can use internal helpers. Their events contribute to the task's activity -and aggregate state. They become separate top-level tasks only when the user -commissions a separate outcome or explicitly splits the assignment. Preserve the -internal graph for cancellation, dependencies, resource ownership and diagnostics; -change its presentation rather than deleting its semantics. - -## Messages have a recipient and a receipt - -Opening a task pins its composer to that task. From the main conversation, an -explicit recipient selection can send a direction without changing views. Natural -language routing must name its interpreted recipient in the acknowledgement. If -several tasks match, retain the unsent message and clarify that recipient only. - -Do not broadcast a casual sentence to every task. Do not treat discussion as a -scope revision automatically. A material direction advances the existing request -revision, invalidates stale acceptance evidence, and reaches the executing agent -at a safe boundary. A question can remain a question in the task conversation. - -The transport must distinguish pending send, received and applied. An optimistic -bubble is not acknowledgement. Retries carry a message ID; reconnect replays -unacknowledged sends and events after the client's cursor, with deduplication. -A failed read keeps the last content and an explicit retry action. Opening a task -from another window must address the same durable task, including when it is idle. - -Drafts and scroll anchors are keyed by recipient/view, not stored in one shared -composer variable. Escape returns to the main conversation, preserving both -views' drafts. Navigation never discards text or stops execution. - -## Keep the conversation useful - -Progress updates replace the existing task row. Tool output and internal helper -transcripts remain inside the task. Main context gets a bounded current task -summary and evidence references; detailed history is read on demand. - -Questions and results have stable notification identities. Reconnect cannot -announce a completion twice. A finished piece does not finish the whole request. -The main assistant still owes the original outcome, including requested synthesis -or integration, and can consult task results when continuing that request. - -A background event must not seize keyboard focus or replace a draft. Show a task -needing input in the rail and one actionable notice. Answering it in either view -resolves that same question. Batch unrelated completions rather than generating a -new main-model response for every internal event. Do not report a stale result -as satisfying a newer direction. - -## State and delivery are separate facts - -User states: Running, Needs your input, Done, Stopped, and Interrupted. Checking -is a running phase. A provider wait remains running with a truthful explanation; -a terminal/provider failure offers recovery with retained work. - -“Done” means the requested outcome is delivered in the requested form. A branch -and commit can be a complete result while main stays unchanged. A work product -that lacks required evidence should say what remains, rather than merely display -a success color. Opening or reading a result does not accept, merge or publish it. -Those operations remain bounded by the user's request and existing permissions. - -Stop applies to the task's execution, owned commands and internal helpers. It -preserves their partial work. Stopping a main response does not stop all tasks. -An explicit “Stop all work” action would require scope to be visible. Continuing -a completed outcome uses a new execution attempt/revision and retains earlier -results; an unrelated outcome becomes a new task. - -Concurrent writes remain isolated or serialized by the existing workspace rules. -A calm UI cannot conceal two tasks writing incompatible changes to one checkout. - -## Prototype scope - -The inline fragment at `prototype/conversation-tasks.html` demonstrates: - -- Main conversation and a flat list of outcome-based tasks. -- Opening and returning, including Escape, with per-recipient drafts. -- Direct task messages and explicit routing from the main composer. -- Starting a task without leaving the main conversation. -- A running task, a decision that pauses it, acknowledgement and continuation. -- Stop/resume and a completed result with its branch, changes and checks. -- Collapsed activity, responsive task navigation and keyboard controls. - -“Advance demo task” produces deterministic simulated background events. The host's -optional design controls compare a right rail and a compact task strip. All task -content, checks and artifacts are illustrative. Sending text uses a local receipt, -not an AI response or semantic intent classifier. Client reconnection, durable -storage, scrolling restoration, old result history, provider recovery and real -work execution remain outside this prototype. - -## Implementation order and proof - -1. Introduce a task presentation model over existing events and IDs. Reuse task - opening, message delivery and stored results. Keep the current executor. -2. Make recipient and per-view draft state explicit. Verify mouse and keyboard - navigation through the actual local and remote terminal connections. -3. Project progress into task rows; deduplicate questions and result notices. - Verify reconnect, lost acknowledgement, late completion and revised requests. -4. Only then replace file-count handoffs with the current capable agent choosing - the existing task tool. Retain cancellation and premature-completion regressions. - -Acceptance is a substantial repository issue while the user keeps discussing -another topic. The user opens and steers the task, returns with their draft intact, -answers one question from the main view, reconnects, stops/resumes work and receives -one accurately located result. Required outcomes: no wrong-recipient sends, no -lost directions or drafts, no orphaned execution, no duplicate completion and no -unrequested merge. Grade the resulting code independently, including public type -contracts; a pleasant task flow cannot substitute for a correct result. diff --git a/docs/design/conversation-runtime/VALIDATION.md b/docs/design/conversation-runtime/VALIDATION.md deleted file mode 100644 index 2699bfd172..0000000000 --- a/docs/design/conversation-runtime/VALIDATION.md +++ /dev/null @@ -1,210 +0,0 @@ -# Validation and remaining product work - -## Current measured candidate - -The later candidate `ec9f5a0c2` passed the full repository check and completed a -frozen 36-cell codeaf/Pi/OMP calibration on the exact DeepSeek model. The detailed -protocol, failures, costs and next decisions are in -[CALIBRATION-20260905.md](CALIBRATION-20260905.md). Small print requests all passed, -but codeaf cost more than Pi on every observed print slice. codeaf passed one of -two side-question cases and one of two revision cases; the other revision timed -out. Steering handoff improved, but a Pareto or production-reliability claim is -not established. The sections below retain the earlier wave's evidence. - -Post-calibration source `3cb79f3f8` also passed `make check`, plus 92 Python tests -and all 112 offline benchmark checks. It includes bounded cache-credit lengths, -request-specific provider observations, and the reviewed multi-module fixture. -It has not been substituted for the measured candidate in the comparison. -The full provider package under `-race` still exposes an allocation-count test -failure reproduced on unchanged baseline `389ab7594`; focused concurrency race -tests passed and no skip was added. See the calibration report for exact scope. - -This is a local development wave, not a release or a claim that the harness is on a -universal cost/time/quality frontier. The integration branch is -`a local working branch`; the shared `dev` checkout and the prior consolidated -QA branch were not used as mutable workspaces. Independent Claude Opus lanes were -reviewed and merged locally. After the Opus session limit, the owner authorized OMP -with GLM 5.3 for the remaining delegated review. All live comparison calls use -`deepseek/deepseek-v4-flash-0731`, including auxiliary roles. Guards refuse other -models before forwarding an inference request. - -## Deterministic evidence - -The code carries behavioral regressions for message provenance, delivery/reopen, -result preservation, task scope at admission, leaf capabilities, assignment revisions, -publication races, check authority, foreground handoff, host detach, status projection, -workspace identity and compaction. Focused tests, race checks and build checks were -run at lane integration. Final whole-tree validation is recorded below when complete. - -Worker openings and landing instructions now use the runtime-note door. Previously, -a child admission could quote its parent's composed brief as a fresh statement by the -person. The new regression covers live recording, journal classification and actual -person corrections; composed instructions no longer become person quotes. - -## Live observations - -The manual terminal run `host-live-02` demonstrated foreground availability and work -continuing after terminal closure. It also found action replay by the checker, wrong -causality on a task-result wake, and relative paths misread as absolute. Each received -a bounded code fix and regression tests. - -The nested run `host-live-03` tested source candidate `b8e868d85` through the normal -hosted terminal. Task 1 created task 2, the prepared one-minute action ran once, and -the parent produced the marker and correct service count. Explicit checks did not -repeat the action. Admission took about 77 seconds, including two missing-deliverable -refusals, one invalid piped check and provider stall rescue. The child still did some -unassigned counting and read a contradictory manual passage; that passage and the -composed-brief provenance defect were fixed afterwards. - -The main conversation answered during the build in about 8.7 seconds, but the saved -model response reversed the checksum incorrectly. This is a quality failure. The -parent result was then recorded, but its automatic narration encountered a provider -internal-markup cut; no substantive final answer appeared during observation. A task -card and a stored artifact do not by themselves satisfy the conversational delivery -requirement. The guard recorded interrupted and unresolved calls, so a complete cost -for this diagnostic run is unknown. - -The first integrated paired-run attempt was invalid: a long evidence path exceeded -codeaf's Unix socket path limit, causing in-process fallback and setup. The rig now -uses a short owned alias to the same cell state. Those two unsupported cells are not -product failures or successful tests. - -## Comparison limits - -The small print-task pilot was not consistently faster or cheaper than Pi or OMP. -Its two orderings and a few tasks do not establish a frontier. Early cost metering -could omit interrupted requests; current metering reconciles every admission and -reports unknown cost when usage is missing. - -Peer interactive evidence uses one repetition per case with a 60-second prepared -command. Pi answered the follow-up after the command finished. OMP displayed the wrong -word. Both peers honored the output-format correction. OMP's installed MCP discovery -was still present despite disabled skills, rules and extensions, weakening isolation. -These facts are observations, not a ranking of the harnesses in general. - -## Corrected hosted comparison, candidate 0c76ea5ca - -The corrected run used the short state alias, the same 60-second fixture and the -same exact DeepSeek model as the peers. Both codeaf cells failed. Follow-up: 86s total, -incorrect `RABBANIC` in the model response and no build-result narration on screen. -Revision: 183s, valid CSV produced but superseded Markdown also left behind. The -fixed session goal was visibly trying to restore the Markdown requirement after the -person changed it. Multiple build starts overwrote a phase marker and also spoiled -its ordering witness. The fixture now counts invocations and keeps the first start. -Both cells have unknown total cost due missing upstream usage; token receipts were -155,235 input / 2,560 output and 772,819 input / 23,523 output respectively. - -These failures prompted two additional bounded repairs: subscribing hosted sessions -to runtime-started turns, and separating interactive conversation policy from a frozen -headless goal. Their final validation belongs below once integrated; the failed raw -runs remain evidence and are not relabeled as passes. - -## Hosted stream follow-up, candidate f7e341d65 - -The wake subscription alone did not pass the live follow-up case: 148s, $0.005210415, -259,894 input and 5,122 output tokens. Unlike the preceding run, the journal contains -the correct `RABANNIC` answer at 01:05:24Z and the complete build result at 01:06:29Z. -Neither survived visibly in the captured chat. Hosted turn adoption reused the -previous turn number and could discard a new stream while the old tail was draining. -The fixed-goal controller also ordered a second invocation of the prepared action. -This is a failed product run; correct hidden model output is not a passed chat test. - -The display repair routes hosted turns through the existing local follow-up/wake -queue. Two deterministic regressions fail before the change and pass afterward: -preserving a preceding answer across a hosted wake, and retaining a new hosted stream -while the previous one drains. The focused TUI suite passed in 2.888s. The remote -wake subscription passed its full package under the race detector in 20.497s, plus -structural laws and the full manual package. Full integrated validation follows. - -## Interactive policy candidate 1e1756b6f - -The mid-work revision case passed: the action ran once, the change arrived while it -ran, all three CSV rows were correct, and the superseded Markdown file was absent. -Wall time was 104s. Total cost is unknown (36 of 40 calls priced); the receipt records -399,746 input and 15,909 output tokens. This fixes the observed reversion behavior, -not a general proof of correct steering across arbitrary tasks. - -The follow-up case still failed: 94s, $0.002813903988, 138,511 input / 3,746 output. -The action ran once. The model first returned an incorrect word, then corrected it -after the work finished. Its build answer was hidden by the later correction inside -the same turn. The trace exposed a remaining causal gap: background command outcomes -had no task result tag, so the completion reader fell back to the latest unrelated -question. Explicit background outcome provenance now survives note batching and -selects its own reply duty. Untagged control wakes retain their prior behavior. - -The shared prompt also now applies to general conversation and artifact work, keeps -engineering guidance conditional on coding, and asks for deterministic computation -when exact calculations or transformations matter. Redundant uncertainty wording was -shortened to keep the existing fixed-prefix budget unchanged. The follow-up rerun -below evaluates these changes; this failed candidate remains in the evidence. - -## Latest follow-up candidate 1209ea81e - -The remaining follow-up rerun took 88s. The prepared action ran exactly once, the -build finished, and its `BUILD-OK marker=QUARTZLINE` result remained visible in chat. -The completion turn did not reopen the unrelated question. These are direct live -observations of the delivery and causal-reply repairs. - -The full scenario still **failed**: the model returned `RABBANNIC` instead of -`RABANNIC`. It replied during the work window, but an incorrect reply does not pass -the correct-answer timing check. The prompt's deterministic-computation guidance did -not make the model use a tool in this instance. Do not call this a renderer failure -or hard-code this particular string as a harness remedy. - -Total cost is unknown: 13 of 16 admitted calls were priced. The receipt records -117,039 input and 2,432 output tokens. The earlier revision pass was on 1e1756b6f; -it has not been relabeled as a pass on this later prompt revision. All live model -calls in these comparison cells were guarded to the exact DeepSeek V4 Flash 0731 -slug. OMP coding delegates used a separate GLM5.3-only guard. - -## Validation scope and local delivery - -The full repository check is recorded in -`/private/tmp/af-conversation-ops/final-make-check-02.log`: **PASS on 8cf8c62cf**. -This includes vet, formatting, the whole test suite under the unchanged known-red -ledger, packaged manual tests and the binary-size check. Full `internal/session` -took 185.807s, `internal/remote` 18.621s and `internal/tui3` 465.307s. The packaged -manual passed in 1.547s. The binary was 50,501,634 bytes against a 54,600,000-byte -limit. The checkout remained unchanged throughout this successful run. - -The first full run is preserved separately as -`final-make-check.log`: it found a missing conditional-tool fragment registration -and lost continue-task wording. Both were fixed, and their focused regression suite -passed (1.629s). That run also detected a documentation edit I made while its checkout -was being monitored; the fresh run keeps the worktree unchanged throughout. No test -or known-red exception was added for these failures. Documentation-only commits may -follow the final source candidate. The configured known-red -ledger has not grown. Focused budget tests passed under the race detector (2.207s), -as did causal-wake and prefix regressions (10.353s). The unchanged fixed-prefix -budget remains 48,000 bytes. - -Work is integrated locally on `a local working branch` in -`/private/tmp/af-conversation`, with its own `bin/codeaf`. The original QA checkout -and shared `dev` were not modified by this wave. Owned merged delegate worktrees, -CLI profile and GLM forwarding process were removed; raw evidence and its short -state aliases are retained. The binary's dirty marker reflects an unrelated tracked -`internal/.DS_Store` change, which has been preserved outside our commits. - -## Priorities after this wave - -1. Make result delivery an explicit user-facing obligation through provider failures: - retained result, attempted narration, visible retry/failure, and a clear way to - recover. Model text ending is not proof that the person received an answer. -2. Reduce admission overhead using measured traces. Keep a small typed assignment; - avoid requiring title, summary and prose restatements as independent model-authored - sources of truth. Measure malformed-tool recovery and model calls before work starts. -3. Replace command-count heuristics for foreground work with a policy that accounts - for duration and reversibility. A long single command can occupy chat as effectively - as a hundred edits; a quick commit still should not need a worktree ceremony. -4. Finish scoped clarification and revision propagation to dependencies. A person can - address one task now; broadcasting to all descendants would be wrong when only one - deliverable changed. Questions need correlated owners and affected work needs explicit - revision dependencies. -5. Extract larger packages only after the interfaces settle. Delivery, provenance, - admission, verification authority and status have single owners, but the session - package still has direct internal access. Landing/workspace disposition and evidence - evaluation remain the next useful module boundaries, with compatibility tests. -6. Run a broader randomized and repeated battery: research, writing, data analysis, - multiple repositories, long context, interruption, crashes, revisions and return - after absence. Report quality gates, time to first useful result, total time, actual - billed cost and failed-call cost. Do not optimize only the successful cells. diff --git a/docs/design/conversation-runtime/local-change-notes/await-own-command.md b/docs/design/conversation-runtime/local-change-notes/await-own-command.md deleted file mode 100644 index f4fd2d2d78..0000000000 --- a/docs/design/conversation-runtime/local-change-notes/await-own-command.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -kind: fixed -title: a reply whose only remainder is a command it started for you no longer becomes a task -surface: [chat, engine, docs] -invalidates: - - "A turn that had finished every artifact you asked for and was left only waiting on a background command it started could still be converted into a task whose whole brief was \"wait for the command, then verify the files\". It is not converted any more: the handoff ask now offers the conversation's own live background commands and fork hands by number, the model may answer that the remainder is only those endings, and the runtime verifies the numbers were offered, are still running, and that the request has not moved. A running job on its own still suppresses nothing — a remainder with real work in it hands over exactly as before." ---- - -Local lane note; root folds this behaviour into its own 653 entry. No pull -request has been opened for it. - -calibration-02 cell 018-revision-midwork-codeaf: the write seam fired after -report.csv was written and report.md removed, the mark reader drew `(waiting)`, -and task 2 was admitted to wait for `./slow-build.sh` — which woke the -conversation with the right answer 42 seconds later anyway. The task spawned a -repair child and an audit child and the cell hit its 180-second cap. diff --git a/docs/design/conversation-runtime/local-change-notes/pareto-measurement.md b/docs/design/conversation-runtime/local-change-notes/pareto-measurement.md deleted file mode 100644 index b116d17d60..0000000000 --- a/docs/design/conversation-runtime/local-change-notes/pareto-measurement.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -kind: fixed -title: Conversation comparisons require matched attempts and complete billing -surface: [build, docs] -invalidates: - - "The conversation report scored quality by fraction of assertions, pooled scenarios, and dropped missing-cost failures. Success is now binary, exact scenario/door/model/effort slices stay separate, and lost billing prevents cost claims without erasing attempts." - - "OMP personal MCP discovery was described as impossible to disable. Its documented discovery-source setting now lives in each benchmark-owned profile." - - "A count of admissions versus settlements was treated as complete billing. Request identities must reconcile exactly; cancelled streams can use matching provider generation receipts without replaying inference." ---- - -Local integration note; no pull request has been opened. - - -A frozen campaign manifest records executable and rig hashes, expected arms, random block order, and operating conditions. Legacy pilot output remains descriptive; small samples and incomplete ledgers do not establish a Pareto frontier. Work is local and no publication is authorized by this change. - -Calibration also found that the driver equated an idle composer with completed work and could close a terminal while its requested background action still ran. It now waits for the fixture's completion witness or the cap. An idle-background regression fails against the previous driver and passes after the fix. Campaigns can stop between cells through an owned STOP file. diff --git a/docs/design/conversation-runtime/local-change-notes/prefix-cache-credit-bound.md b/docs/design/conversation-runtime/local-change-notes/prefix-cache-credit-bound.md deleted file mode 100644 index 1a3975637a..0000000000 --- a/docs/design/conversation-runtime/local-change-notes/prefix-cache-credit-bound.md +++ /dev/null @@ -1,126 +0,0 @@ -# The prefix note carries a length - -Local change note. Not a changelog entry, not a PR description. - -## What was wrong - -`internal/lane`'s prefix memory stored `{prefix, at}` — which conversation a lane -last served, and when. It stored no length. `cachedTokens` therefore answered -"how much of this request does that lane hold?" with the **whole current -prompt**, whenever the lineage matched inside `PrefixHold`. - -A conversation grows between requests. A lane that answered at 14,972 prompt -tokens was credited the full 15,502 of the next request: the ~530 tokens -appended since had never been on the wire to it and could not be in its cache. -The frontier priced that phantom discount into the lane's score -(`frontier.go:197`, `PriceWithCache`). - -This is an accounting error that needs no cache behaviour at all to be wrong. It -is independent of what any endpoint reports about cache reads. - -## What changed - -- `prefixNote` gains `tokens int` — the prompt length the lane was last **seen** - at (`internal/lane/choose.go`). -- `RememberPrefix(id, prefix, tokens, at)` takes it. Negative is clamped to zero. -- `cachedTokens` returns `min(note.tokens, req.PromptTokens)`, and returns 0 when - `note.tokens <= 0`. -- The length fed in is the **settlement's** `usage.prompt_tokens`, threaded - `client.go` → `noteVelocity` → `noteLane` → `RememberPrefix`, via a nil-safe - `promptTokensOf`. It is deliberately **not** `ask.prompt`, the adapter's - pre-send four-characters-a-token estimate that sits two lines away in - `noteLane` — a discount granted on a guessed schema size is a discount no - usage frame agreed to. -- `RememberedPrefix(id)` is added as the exported read half, for the same reason - `ForgetPrefixes` is exported: the transport that writes these notes is in - another package. Nothing in the routing path calls it. - -## The attribution the length had to travel with (root review) - -The first cut of this change read the lineage at settlement out of -`askFor(model)`, backed by `laneAsks` — a last-ask-per-model map written at -encode time. That map answers **"which request encoded most recently"**, which -is a different question from "whose answer is this". - -One `Client` serves several requests at once — a fan-out's leaves, an errand -beside a turn, the two halves of a hedge. Encode A, encode B, and let A settle -first: the map hands A's answer B's lineage, and B's conversation is credited -with a prompt cache that A wrote. The mutex around the map made that -misattribution free of data races; it did not make it true. Correcting the -*length* while leaving the *lineage* looked up at the end would have filed a -newly-accurate number against the wrong conversation. - -So the observation is now a small typed value, `settled{lineage, prompt, -cached}`, built at the settlement site by `settledFrom(ctx, usage)`: - -- the lineage is `CacheKeyFrom(ctx)` — the settling request's own context, which - is the only thing that knows whose call it is. This is the same read - `noteEndpointAffinity` already does, so the affinity pin and the prefix note - can no longer disagree about which conversation an answer belonged to; -- it replaces two trailing positional ints rather than adding a third; -- `laneAsks`, `lanesState`, `rememberAsk` and `askFor` are **deleted**. Nothing - else read them, so there is no second, staler answer left behind. - -One consequence, stated plainly: `Sighting.PromptTokens` is now the settlement's -observed prompt length instead of the pre-send estimate. It feeds -`lane.promptNoise`, a three-bucket weight on how much noise a TTFT reading -carries. A settlement whose frame reported no length now weights as the quietest -bucket — which is exactly what the field already produced whenever no ask had -been remembered for that model, and is a better failure than an estimate -belonging to a different request. Output and generation timings keep their -existing path untouched. - -Both settlement paths — the whole-response epilogue and the streamed one, -including its no-first-token branch — build the observation the same way and -still record the served identity read off the decode. - -Unchanged on purpose: `PrefixHold`, lineage isolation, the 256-entry bound and -its oldest-first eviction, affinity release (`provider/affinity.go`), the -sampler, provider fallback (`AllowFallbacks`), prices, and exact-model pins. -`Sighting.PromptTokens` still carries the pre-send estimate — a different -consumer, out of scope here. - -## Honest limits - -**This is still an estimate, not a receipt.** A matching lineage inside the hold -says a lane answered these bytes recently — not that it still holds them. The -lane may have evicted the prefix. Only the next answer's `cached_tokens` -settles it, and nothing here reads that number back as proof. - -**Compaction and rewrite are bounded, not solved.** They can change bytes -*under* an unchanged lineage, so a prompt of the same or shorter length may be a -*different* prompt that shares no prefix. The `min` keeps such a case from -earning credit for tokens it does not send; it cannot detect that the shared -prefix itself is gone. A lineage key that changed when the transcript was -rewritten would be the real fix, and that is not this change. - -**Observed cache reads are not the cacheable prefix.** The last answer's -`cached_tokens` is deliberately not used as the note's length: a first cold call -reads zero and still *populates* the cache, so treating a cache-read count as -the ceiling would deny the discount exactly where it was earned. - -**No cost claim.** The completed 36-cell benchmark ran on the preceding source. -Nothing here has been measured against it and no saving is claimed. - -## Evidence - -Two ablations, each run in the session that made this change rather than left in -the tree: - -- restoring `cachedTokens` to `return req.PromptTokens` fails - `TestAGrownPromptEarnsNoCreditForTokensTheLaneNeverSaw` with "a grown prompt - was credited 15502 cached tokens, want the 14972 the lane actually saw", and - `TestALengthNobodyReportedEarnsNoDiscount`; -- restoring the lineage-from-last-encode attribution fails - `TestAnAnswerIsAttributedToItsOwnRequestWhenTwoOverlap` with "the last answer - was A's and its prefix note says \"conversation-B\"". - -The overlap regression drives two real completions through the lanestub door on -one client with two lineages. The ordering is FORCED, not timed: a `requestGate` -in `Config.HTTPClient` holds A at its send until B has encoded, been answered -and settled, and only then releases it. There are no sleeps. Every wait is -bounded by one 10-second test context, and a single cleanup — registered after -the server's, so it runs first — opens the gate, cancels both calls and waits -for A to leave the wire before the listener closes. Both settlement paths are -covered: the streamed epilogue, and the whole-body branch, reached by having the -gate rewrite `stream` to false so the router answers in one piece. diff --git a/docs/design/conversation-runtime/local-change-notes/prompt-shape-composition.md b/docs/design/conversation-runtime/local-change-notes/prompt-shape-composition.md deleted file mode 100644 index 9eaa10edb2..0000000000 --- a/docs/design/conversation-runtime/local-change-notes/prompt-shape-composition.md +++ /dev/null @@ -1,76 +0,0 @@ ---- -kind: fixed -title: Prompts follow standing availability and rendered prefixes are measured -surface: [chat, engine] -invalidates: - - "`# Things that keep working after this window` was in prompts/system.md for everybody, 2.4KB opening with the words \"When your tool list carries `stand`\" — the page saying in its own first clause that it was written for readers who may not have the verb. A task room is one of those: a node is handed no standing store (task_run.go), and neither is `--once` or a firing's headless session. The section is now a beltFact composed from [Config.mayStand], the predicate tools_standing.go builds the tool from, and the absent case is one sentence that names no verb." - - "The session facts' clock bullet told every shape to reach for `stand`'s `when.in` when a minute mattered. On a belt without `stand` that is the exact defect beltfacts.go exists to prevent, so the bullet is composed from the same predicate and the absent case says to read the clock as needed instead." - - "`## 2. Decompose` was one line pointing at `# Planning`, which already says to write the plan note first. The pointer is gone and the Workflow phases renumber; the law is unchanged and lives once." - - "The fixed-prefix budget weighed [widestPage] plus one door's tool block, so nothing measured what a shape actually renders — a task room carries prompts/worker.md and the fan-out page on top of the page, and pays that prefix on every round of every step. prefixshape_test.go now renders all four shapes and prints the bill; the budget stays the one ratchet." ---- - -Local integration note; no pull request has been opened. - - -Measured, before → after, page plus marshalled tool block as the shape's own door -renders it (prefixshape_test.go): - -| shape | before | after | delta | -| --- | --- | --- | --- | -| chat | 56,184 | 55,945 | -239 | -| yolo (`--yolo`) | 54,616 | 54,377 | -239 | -| task room, may fan out | 55,013 | 52,487 | -2,526 (-4.6%) | -| task room, floor | 42,335 | 39,809 | -2,526 (-6.0%) | - -The budget's own figure went 47,965 → 47,725 against an unchanged 48,000: the -widest page is by definition the one that HAS `stand`, so the ratchet sees only -the sentence the composed section no longer needs. - -WHAT IS NOT DONE, with the evidence for it. The conversation's own prefix is -~14k tokens and 36,660 of it is the tool block, so the remaining reduction is -mostly there and not in the page: `propose_task` is 6,041 bytes on its own, -`tasks` 2,202, `fork` 1,958, `edit_video` 1,873. On the page, `# When working on -code` and `# Workflow` are 2.1KB of engineering-shaped recipe every non-coding -turn also pays for, and their laws are pinned by proportion_test.go and -task_escalation_test.go — which is where a rewrite has to start reading, not a -place to cut in a hurry. - -## The task family's prose, 2026-09-05 - -Second bounded pass, on `propose_task` and `tasks` only. Every field, type, -default, required entry and enum is unchanged; what came out is duplication and -worked examples. - -- `propose_task`'s description: the elsewhere line said "nothing is blocked or - queued and your task has started" where the sentence before it had already - said the id returns at once. The planner clause STAYS — task_divide_test.go - pins it against a measured live run that announced an adaptive research run - for wide work, and the page's own "THERE IS NO PLANNER" paragraph is the - second layer rather than the only one. -- `brief`: the three worked examples (prose / code / research) came out and the - principle over them — name the lazy but plausible-looking answer and forbid - it, in the terms this kind of work is judged by — is what carries them. It - also stops saying "files and symbols", which is a coding shape for a field a - literature review fills in too. The mid-work findings contract, the - cannot-ask-you contract and the failed-task carry are unchanged. -- `expects` (shared with `divide_work`, so it pays twice): the four-item example - list came out; `path`, `holds`, `absent` and `fact` each say what they are - directly underneath it. -- `no_progress`: "before its first edit" → "before it produces anything". -- `tasks`: the description's "'continue task N' uses continue" is the `continue` - field's own first sentence, in the same block of text; the tail restated the - four operations the sentence before it had just named. - -| shape | after pass 1 | after pass 2 | total vs base | -| --- | --- | --- | --- | -| chat | 55,945 | 55,627 | -557 (-1.0%) | -| yolo | 54,377 | 54,059 | -557 (-1.0%) | -| task room, may fan out | 52,487 | 52,225 | -2,788 (-5.1%) | -| task room, floor | 39,865 | 39,865 | -2,470 (-5.8%) | - -WHAT REMAINS UNPROVEN. This bounded edit found only small savings; it does not -establish a minimum possible prompt size or prove that every remaining sentence -is necessary. The conversation still carries a large tool block. A larger -experiment should compare a concise equivalent contract and demand-loaded tool -definitions, preserving access to capabilities and measuring discovery latency, -correctness and cost together. Neither design has been validated by this patch. diff --git a/docs/design/conversation-runtime/local-change-notes/steer-young-bash-grace.md b/docs/design/conversation-runtime/local-change-notes/steer-young-bash-grace.md deleted file mode 100644 index 8bac251f69..0000000000 --- a/docs/design/conversation-runtime/local-change-notes/steer-young-bash-grace.md +++ /dev/null @@ -1,116 +0,0 @@ ---- -kind: fixed -title: A correction typed into a young bash no longer waits for the command or the clock -surface: [chat, build, docs] -invalidates: - - "A steer that found every foreground bash younger than 3 seconds was measured once and never again, so the correction waited for the command's own ending — or, in a live chat, for the 30-second background-after clock. The age is now re-read once, at the moment the youngest running call crosses the same 3 seconds, through the one adoption path the steer itself uses." - - "`waiting for the running step` implied a wait of unknown length for a bash call. The HANDOFF is now bounded at about three seconds — not the model's reply, which still waits on the rest of the batch and on the request itself; the clause is unchanged because it was true when it was sent, and the transcript's own record of the correction is updated to say which of the two landings happened." - - "An explicit stop phrase used to wait out the same 3-second grace as an ordinary correction when the command was young. It now reaches a foreground command at any age and cancels the armed watch, and a superseded stop left on the queue no longer holds authority over the delayed look." ---- - -Local implementation note; no pull request has been opened. - -## What was wrong - -`Agent.Steer` → `steerRunningBashLocked` skipped every foreground call younger -than `steerBashAge` (3s) and scheduled nothing. The only other road to a step -boundary was `bashPromotion.Started`'s background-after clock — -`DefaultBashBackgroundAfter`, 30 seconds in livechat — or the command's own end. -So a side question typed 0.5s into a 60-second command reached the model at ~30 -seconds. An earlier live calibration answering a side question at ~34 seconds is -consistent with this, though that run also failed on quality and cannot on its -own separate the two causes. - -Measured on this branch, `internal/session`, with the background clock set to 20s -and a `sleep 7` foreground command, steering ~0ms after the call started: - -| | when the model received the correction | -|---|---| -| before | 7.03s — the command's own ending (the 20s clock never fired) | -| after | 3.03s — `steerBashAge` plus a 10ms margin | - -An explicit `stop` typed at a half-second-old command: unbounded before (it -waited for the command), and 0.02s now, measured inside `Agent.Steer` itself. - -These are figures for the HANDOFF — the moment the command is let go of and the -next request can carry the words. The model's reply is not bounded by them. - -## The mechanism - -`internal/session/steer_grace.go`. One `time.AfterFunc` per steer at most, armed -under `a.mu` for the instant the LAST call in the batch ripens, replacing (and -stopping) any watch already there. On firing it re-checks, under `a.mu`, the -identities it captured — `turnSeq`, the exact `*bare.BashCall` pointers, and a -freshly read steering queue — and then walks `steerRunningBashLocked`, which now -takes its calls as an argument so the watch can restrict it to what it armed for. -There is no second adoption rule, no second tool sentence and no second kind of -job. - -- An explicit stop never enters this file: `steerRunningBashLocked` ignores the - age gate for a stop phrase, so it reaches a command of any age the instant it - is typed, and `Steer` cancels any armed watch on that road. The delayed look - therefore acts for the LATEST still-pending correction only — a superseded - stop is a directive the person moved on from, not standing authority, and - both lines reach the model in order for it to decide. -- `Interrupt` stops the watch INSIDE its own `a.mu` block, before it releases - the lock to call cancel. That gap is a real interval — the turn is still - running and the command's context still alive — and a firing inside it would - hand the foreground command to the job registry, where a job deliberately - survives an interrupt: the person who pressed stop would be left with the - command detached and running. The later cleanup stops it again, idempotently. -- The watch is stopped by the turn's cleanup, by `Abandon` and by `Close`, and - a firing whose watch the agent is no longer holding returns at once — a - replacement inside one turn shares the turn number and the calls, so that is - the only check that tells the two apart. -- The firing goroutine takes `a.mu` directly and calls the job registry with it - held, exactly as `Steer` does — it is never reached through a registry - callback, which is the lock order that deadlocked a previous attempt. - -## Tests - -`internal/session/steer_grace_test.go`: the bounded handoff with the process -adopted once and its exit still arriving; a quick command left alone with an -inert late timer; an explicit stop reaching a half-second-old command at once -and leaving no watch armed; the newest-direction rule including a superseded -stop; a watch replaced inside its own turn firing inertly; a firing into -`Interrupt`'s own unlock-then-cancel window, gated by wrapping `a.cancel`; -the armed-for -identities (wrong turn, foreign calls, a repeat firing); and an interrupted -turn leaving no job and no consumed steer. - -## Root review, second pass - -Two defects found in the first commit and fixed here: - -1. `steerGraceFired` cleared `a.steerGrace` only when it matched and then acted - regardless, so a watch replaced by a second steer inside the SAME turn could - still adopt — the different-turn test could not see it. There is now an - explicit return, with `TestAReplacedSteerGraceIsInertInItsOwnTurn` as the - counterexample (it fails against the first commit with - `a watch the agent had already let go of adopted a command`). -2. The delayed look took any queued stop phrase as standing authority. An - explicit stop now acts immediately at any age instead, which removes the - question rather than answering it; no classifier and no resume vocabulary - were added. -3. `Interrupt` released a.mu before cancelling and never stopped the watch, so - a firing in that window could detach the very command the person had just - stopped. `TestASteerGraceCannotDetachWorkInsideAnInterrupt` holds Interrupt - at the gap by wrapping `a.cancel`, and fails against the second commit with - `a stop the person asked for detached the command instead: "job 1 · running - · 0ms · sleep 30"`. - -A live functional probe (root, `grace-live-before/ADJUDICATION.md`) measured the -original accepted → consumed gap at 30.620791s, which is the background-after -clock and matches the mechanism above, separately from that run's wrong answer. - -## Limits - -- The `EventSteerAccepted` clause a person already read is not rewritten, because - the surface folds a repeat acceptance into the row it already drew - (`internal/tui3`'s `app.steerAccepted`). Only the journal's `SteerMark` is - corrected. A surface that wanted the live clause to change would need an event - of its own. -- The bound is `steerBashAge` plus scheduling, so a correction can still wait - ~3 seconds behind a bash call; short non-bash tools are unchanged and still - finish first. Nothing here bounds what the batch or the provider then cost. -- No live-model run was made from this lane. diff --git a/docs/design/conversation-runtime/prototype/README.md b/docs/design/conversation-runtime/prototype/README.md deleted file mode 100644 index 0f0dca9ce2..0000000000 --- a/docs/design/conversation-runtime/prototype/README.md +++ /dev/null @@ -1,28 +0,0 @@ -# Conversation and tasks prototype - -`conversation-tasks.html` is a self-contained interactive fragment. Open it in a -browser, or render it with the Codex inline visualization host. It has no model, -server, storage, external assets or network dependencies. Refresh starts a fresh -simulation. The optional host design control switches task navigation between a -right rail and a compact strip. - -Try this sequence: - -1. Type a draft in the main conversation, then open the authentication task. -2. Type a task draft and press Escape. The main draft is still there. Choose the - authentication task in the recipient menu to see and send its draft. -3. Refresh, then use **Advance demo task**. The task needs a compatibility decision. - Open it, stop and resume it: the unanswered decision remains. -4. Answer, then advance again. Review the branch result without merging it. -5. Back in the conversation, describe another assignment and choose **Run as task**. - The task appears while the conversation stays open. - -All changes, checks and results shown are fixtures. The walkthrough does not -interpret natural language or perform actual work. The architecture and omitted -runtime behavior are documented in `../TASK-EXPERIENCE-PROTOTYPE.md`. - -Validation: headless browser interactions cover recipient routing, independent -drafts, stop/resume, retained questions, completion, task creation, escaped input -and mobile navigation. Layout checks cover 1024, 736 and 360 pixels in light and -dark appearances. Prototype JavaScript runs without browser errors. The design -detector reports no findings. This does not establish production integration. diff --git a/docs/design/conversation-runtime/prototype/conversation-tasks.html b/docs/design/conversation-runtime/prototype/conversation-tasks.html deleted file mode 100644 index aa8fa3368d..0000000000 --- a/docs/design/conversation-runtime/prototype/conversation-tasks.html +++ /dev/null @@ -1,190 +0,0 @@ -
- -
Interactive concept · simulated work
-
-
codeaf / acme-apiConversation + tasks
-
-
- -
-
ConversationEnter send · Shift+Enter newline · Esc back to conversation
-
- -
From 8761f37b341112d9f1eb138de0cd20c31a12b82c Mon Sep 17 00:00:00 2001 From: santoshkumarradha Date: Wed, 16 Sep 2026 16:33:25 -0400 Subject: [PATCH 3/8] chore: neutralize the remaining bench-host and work-machine references A wider sweep found the internal bench host ("Spark"), its secrets path (fleet/secrets.env), and a scratch ops directory still named in live docs, bench READMEs and bench scripts that the earlier redaction had not touched. Rename the host to "benchhost"/"the bench host", drop the secrets path, and replace "the laptop" as the owner's own machine with a neutral "work machine" where it meant the internal one. The manual's first-person "the laptop you are sitting at" is public-facing and left alone. Co-Authored-By: codeaf --- PERF.md | 2 +- bench/census/README.md | 4 ++-- bench/conversation/lib/adapters.sh | 6 +++--- bench/conversation/notes/pilot-02-03.md | 6 +++--- bench/prompt-diet/run.sh | 4 ++-- docs/design/home-mission-control/DESIGN.md | 10 +++++----- docs/design/quick-task/DESIGN.md | 10 +++++----- docs/design/recovery/DESIGN.md | 2 +- docs/design/task-start/DESIGN.md | 4 ++-- docs/rules/ci.md | 2 +- 10 files changed, 25 insertions(+), 25 deletions(-) diff --git a/PERF.md b/PERF.md index 97584346d1..4c23e879ef 100644 --- a/PERF.md +++ b/PERF.md @@ -1637,7 +1637,7 @@ number rather than one per kind for a reason worth writing down, because the obvious change was made and taken back out (#846). A person's keystroke on a question is a call too — internal/tui3's `answerQuestion` asks its door straight from Update rather than from a command — so a longer window for an act would be -a longer time the terminal can sit without drawing. Measured on the Spark: +a longer time the terminal can sit without drawing. Measured on the bench host: eighteen copies of the questions e2e, six at a time, and with a thirty-second act window two of them took 53 seconds where every other copy took 22, both losing the receipt because nothing repainted. Twelve copies of the parent commit diff --git a/bench/census/README.md b/bench/census/README.md index 1b1057584b..60998ae68f 100644 --- a/bench/census/README.md +++ b/bench/census/README.md @@ -19,11 +19,11 @@ make census TOP=40 DAYS=7 MIN=10 CHAINS=20 # widen what it prints It is pure Go with no dependencies, and it never writes to the log it reads. -## Nightly, on the Spark +## Nightly, on the bench host The measurement is the point: five waves of recovery work each move a number in that table, and without a run that happens without anybody asking, every one of -them is an argument about anecdotes. So the Spark syncs the laptop's log and +them is an argument about anecdotes. So the bench host syncs the work machine's log and runs the same target once a night, keeping one file per day: ```sh diff --git a/bench/conversation/lib/adapters.sh b/bench/conversation/lib/adapters.sh index 2c8ffc7c10..d5f0f9123a 100644 --- a/bench/conversation/lib/adapters.sh +++ b/bench/conversation/lib/adapters.sh @@ -507,7 +507,7 @@ arm_tui_argv() { # 2026-09-10 — after the seven-panel home landed — a fresh screen drew the # state word at the right edge of the status row with nothing in front of # it, so the needle matched nothing and every interactive codeaf cell - # recorded `unsupported` at the ready wait. Measured on the Spark that + # recorded `unsupported` at the ready wait. Measured on the bench host that # day: `followup-while-working` and `work-result-recalled` both ended # `noready` after 91s against a pane whose last line read `idle`. # @@ -520,7 +520,7 @@ arm_tui_argv() { ARM_READY_RE='(· idle|idle[[:space:]]*$)' ARM_BUSY_RE='(· (working|interrupted)|(working|interrupted)[[:space:]]*$|[0-9]+ running)' ARM_ASK_RE='waiting · your call' - ARM_DOOR_NOTE='markers from internal/tui3/render.go (stateWord, waitingWord), matched against a live pane in /private/tmp/af-conversation-ops/host-live-01' + ARM_DOOR_NOTE='markers from internal/tui3/render.go (stateWord, waitingWord), matched against a live pane in a scratch bench-ops dir (host-live-01)' # Hosted, like any other conversation: the interactive door is the one # place this suite can measure the product's actual default, and passing # --no-host here would quietly measure something else. The cell's own @@ -542,7 +542,7 @@ arm_tui_argv() { # whichever provider is in use, so a rig that wired pi to the guard and # then waited for "(openrouter)" waits forever: a live pane on a guarded # run reads "(guard) deepseek/deepseek-v4-flash-0731 • low" - # (/private/tmp/af-conversation-ops/live-interactive-peers-01/followup-while-working-pi/screen.txt). + # (a scratch bench-ops dir: live-interactive-peers-01/followup-while-working-pi/screen.txt). ARM_READY_RE="\\($(arm_provider_name)\\)" ARM_BUSY_RE='Working\.\.\.' ARM_ASK_RE='(\[y\]|approve|Allow\?)' diff --git a/bench/conversation/notes/pilot-02-03.md b/bench/conversation/notes/pilot-02-03.md index c6a31491ab..185d2867d1 100644 --- a/bench/conversation/notes/pilot-02-03.md +++ b/bench/conversation/notes/pilot-02-03.md @@ -3,9 +3,9 @@ Three live runs the owner made against codeaf `4d9d91fb0`, pinned to `deepseek/deepseek-v4-flash-0731` throughout: - /private/tmp/af-conversation-ops/live-pilot-02 print door, 4 scenarios × 3 arms - /private/tmp/af-conversation-ops/live-pilot-03 the same, repeated - /private/tmp/af-conversation-ops/live-interactive-peers-01 interactive door, pi and omp + scratch bench-ops dir: live-pilot-02 print door, 4 scenarios × 3 arms + scratch bench-ops dir: live-pilot-03 the same, repeated + scratch bench-ops dir: live-interactive-peers-01 interactive door, pi and omp This file adjudicates the non-pass cells in those runs. **Nothing in the raw evidence has been touched**: no receipt is rewritten, no verdict is edited in diff --git a/bench/prompt-diet/run.sh b/bench/prompt-diet/run.sh index 5515fdd935..49dcb1a161 100755 --- a/bench/prompt-diet/run.sh +++ b/bench/prompt-diet/run.sh @@ -20,8 +20,8 @@ # rather than two prompts. # # IT RUNS ON THE BENCH HOST. Layer B alone is seventeen minutes of a real binary in a -# real terminal against a real model, and the owner's standing order is that no -# full suite runs on the laptop. Nothing here refuses to run elsewhere — a +# real terminal against a real model, and the full suite is not meant to run on a +# work machine. Nothing here refuses to run elsewhere — a # refusal keyed off a hostname would be a lie on the next machine — but every # recipe in docs/design/prompt-diet/BENCH.md is spelled for `ssh benchhost`. # diff --git a/docs/design/home-mission-control/DESIGN.md b/docs/design/home-mission-control/DESIGN.md index 4b04ea274f..09ed8db5bb 100644 --- a/docs/design/home-mission-control/DESIGN.md +++ b/docs/design/home-mission-control/DESIGN.md @@ -240,7 +240,7 @@ lines would read as the screen running out of room (owner, 2026-09-10). ## 5. Lanes Worktrees off `origin/dev` at `~/af-`, pull requests against `dev`, full -suites on the Spark, Opus for the mechanical lanes, one change entry per PR. +suites on the bench host, Opus for the mechanical lanes, one change entry per PR. | Lane | Owns | Depends on | Size | |---|---|---|---| @@ -273,10 +273,10 @@ useful. ## 7. Where the work runs -Lanes E, G and K run as Claude Code instances on the Spark under fleet +Lanes E, G and K run as Claude Code instances on the bench host under fleet (`af-home-e`, `af-home-g`, `af-home-k`; worktrees `~/af-home-` on the -Spark, branches `home/mc-`, briefs in `spark:~/af-home-briefs/`, logs -`spark:~/af-home-.log`, reports `spark:~/af-home-.report.md`). +bench host, branches `home/mc-`, briefs in `benchhost:~/af-home-briefs/`, logs +`benchhost:~/af-home-.log`, reports `benchhost:~/af-home-.report.md`). The integrator merges each into `home/mission-control`, builds `bin/codeaf` for the owner after every viewable step, and runs the full suites once, at -the end, on the Spark. +the end, on the bench host. diff --git a/docs/design/quick-task/DESIGN.md b/docs/design/quick-task/DESIGN.md index 03d0f62bbe..92aa579221 100644 --- a/docs/design/quick-task/DESIGN.md +++ b/docs/design/quick-task/DESIGN.md @@ -169,7 +169,7 @@ Everything else is new files: `task_quick.go`, `task_quick_test.go`, | C ceiling | `~/af-quick-c`, branch `quick/ceiling` | `launchRouteTask`: a write-free turn's sketch becomes a quick node's items; test | | F fork | `~/af-quick-f`, branch `quick/fork-off` | `fork` off the belt, belt bullet, manual sections, probes, actioncategory; keep the scope functions | -Merge order K → S → M → C → F, one PR against `dev`, suites on the Spark. +Merge order K → S → M → C → F, one PR against `dev`, suites on the bench host. ## Acceptance, end to end first @@ -183,18 +183,18 @@ Merge order K → S → M → C → F, one PR against `dev`, suites on the Spark reaches the task's worker, not the conversation. 5. `internal/tui3`: a `TaskNotice{Kind: quick, Doing: "quick · 1/3 · …"}` draws a rail row with that doing line and no branch or merge word; a done card shows the result body. -6. Real model, `deepseek/deepseek-v4.1-flash`, person-style in tmux on the Spark: ask the +6. Real model, `deepseek/deepseek-v4.1-flash`, person-style in tmux on the bench host: ask the chat to compare four files in parallel and summarise; expect four `quick` rows at once, each done within a minute, the summary as a reply, no branch anywhere. ## Testing law for every lane -NO FULL SUITE ON THE LAPTOP. `go build ./...`, `go vet`, gofmt, `make test-quick`, and +No full suite on the work machine. `go build ./...`, `go vet`, gofmt, `make test-quick`, and single named tests (`make test-focus PKGS=./internal/session RUN='^TestX$$'`) run here. -Every package suite runs on the Spark: +Every package suite runs on the bench host: ```sh -ssh spark 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/.config/fleet/secrets.env; set +a; \ +ssh benchhost 'export PATH=$HOME/.local/bin:$PATH; set -a; . ~/secrets.env; set +a; \ git -C ~/src/codeaf fetch -q origin && \ git -C ~/src/codeaf worktree add -f --detach /tmp/af- origin/ && \ go -C /tmp/af- build ./... && go -C /tmp/af- test ./internal/session/ -count=1 -timeout 20m 2>&1 | tail -60; \ diff --git a/docs/design/recovery/DESIGN.md b/docs/design/recovery/DESIGN.md index edc21eb61f..53e947a931 100644 --- a/docs/design/recovery/DESIGN.md +++ b/docs/design/recovery/DESIGN.md @@ -303,7 +303,7 @@ duration bound. Five waves, each landing green on `dev` and each reducing a number in §1. The first is measurement, because two of the twelve problems are that we cannot see. -Every wave runs its suites on the Spark, never on the laptop. +Every wave runs its suites on the bench host, never on a work machine. | wave | state | lands | number it moves | | --- | --- | --- | --- | diff --git a/docs/design/task-start/DESIGN.md b/docs/design/task-start/DESIGN.md index b22671d613..d12337b95c 100644 --- a/docs/design/task-start/DESIGN.md +++ b/docs/design/task-start/DESIGN.md @@ -142,7 +142,7 @@ existing, with no model call in it, and the snapshot that followed took 0.8 seco Nothing said so anywhere, and nothing stopped the next node paying again. `Fork` was also the one furrow call on that road with no bound at all. -The laptop's own cause is not reproducible on Linux: on the Spark the same furrow +The work machine's own cause is not reproducible on Linux: on the bench host the same furrow 0.1.0 forks the same shapes of repository and the rung succeeds (eight of eight recorded nodes). One cause shape was proven there, though. The fork sealed its world with `git checkout -b` and `git commit`, which run the repository's commit hooks and @@ -185,7 +185,7 @@ without the ignored files, and every such task's log says so. Retrying on every the fifteen seconds this section was written to end, and retrying on a timer is a knob with no measured value behind it. -Measured on the Spark (real furrow, four nodes per shape, per-node ground time): +Measured on the bench host (real furrow, four nodes per shape, per-node ground time): | ground | before | after | | --- | --- | --- | diff --git a/docs/rules/ci.md b/docs/rules/ci.md index 2be1166f10..0c03066a8e 100644 --- a/docs/rules/ci.md +++ b/docs/rules/ci.md @@ -183,7 +183,7 @@ report, not a line to add back — the ledger does not return. It is a bug report, and it never goes on the ledger above. Reproduce it under load — a focused `-count=50` at `GOMAXPROCS=2` with a few `yes > /dev/null` -beside it, **on Spark and never on the laptop**, since the point is to starve +beside it, **on the bench host and never on a work machine**, since the point is to starve the machine — and fix the CAUSE, which is almost always that the test measured the SCHEDULER and called it the road: From fd84adfcfe64a9dc9bd1b35e15a62edf2288cfe2 Mon Sep 17 00:00:00 2001 From: santoshkumarradha Date: Wed, 16 Sep 2026 16:36:20 -0400 Subject: [PATCH 4/8] chore: fix review findings in the public-cleanup diff - PERF.md: drop a private tracker issue reference (#846). - PARETO.md: restore a complete sentence dropped when the calibration link was removed, and align tense with the remaining present-tense claim. - home-mission-control/DESIGN.md: drop the internal "fleet" system name and an owner-framing phrase, consistent with the secrets-path scrub elsewhere. Co-Authored-By: codeaf --- PERF.md | 2 +- docs/design/conversation-runtime/PARETO.md | 4 ++-- docs/design/home-mission-control/DESIGN.md | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/PERF.md b/PERF.md index 4c23e879ef..0ac5b3a2bd 100644 --- a/PERF.md +++ b/PERF.md @@ -1634,7 +1634,7 @@ the pointer crosses. **The deadline is `callDeadline`, ten seconds, for every call**, and it is one number rather than one per kind for a reason worth writing down, because the -obvious change was made and taken back out (#846). A person's keystroke on a +obvious change was made and then taken back out. A person's keystroke on a question is a call too — internal/tui3's `answerQuestion` asks its door straight from Update rather than from a command — so a longer window for an act would be a longer time the terminal can sit without drawing. Measured on the bench host: diff --git a/docs/design/conversation-runtime/PARETO.md b/docs/design/conversation-runtime/PARETO.md index 8596cd8094..ee54857540 100644 --- a/docs/design/conversation-runtime/PARETO.md +++ b/docs/design/conversation-runtime/PARETO.md @@ -2,8 +2,8 @@ ## Current evidence -Calibration runs (36 frozen cells across codeaf, Pi and OMP) recorded at -calibration time. codeaf was not yet at the target: the observed small-task cost gap +Calibration runs (36 frozen cells across codeaf, Pi and OMP) were recorded at +calibration time. codeaf is not yet at the target: the observed small-task cost gap remains, and each chat slice passed only one of two attempts. The next priority is unnecessary handoff/checking and the rendered request size. Bounded cache accounting and request-specific attribution landed after those measurements; diff --git a/docs/design/home-mission-control/DESIGN.md b/docs/design/home-mission-control/DESIGN.md index 09ed8db5bb..be7306f475 100644 --- a/docs/design/home-mission-control/DESIGN.md +++ b/docs/design/home-mission-control/DESIGN.md @@ -273,10 +273,10 @@ useful. ## 7. Where the work runs -Lanes E, G and K run as Claude Code instances on the bench host under fleet +Lanes E, G and K run as Claude Code instances on the bench host (`af-home-e`, `af-home-g`, `af-home-k`; worktrees `~/af-home-` on the bench host, branches `home/mc-`, briefs in `benchhost:~/af-home-briefs/`, logs `benchhost:~/af-home-.log`, reports `benchhost:~/af-home-.report.md`). The integrator merges each into `home/mission-control`, builds `bin/codeaf` -for the owner after every viewable step, and runs the full suites once, at +after every viewable step, and runs the full suites once, at the end, on the bench host. From b2af09b701899836d9291bf59bf513f96874615d Mon Sep 17 00:00:00 2001 From: Abir Abbas Date: Wed, 16 Sep 2026 17:18:10 -0400 Subject: [PATCH 5/8] docs/design: the two records live code still cites stay in the tree plan-gate-doe/REPORT.md is the measured evidence behind the plan-gate default that twelve comments in cmd/codeaf, internal/config, internal/session and internal/splitgate point at, and turn-wall-share-doe is the run internal/session/turnwall.go says it was added from. CLAUDE.md keeps the record on purpose for exactly this reason, and neither folder names a machine, a secret or a private address. Co-Authored-By: Claude Fable 5.1 --- docs/design/plan-gate-doe/REPORT.md | 122 ++++++++++++++++++++++ docs/design/turn-wall-share-doe/README.md | 120 +++++++++++++++++++++ 2 files changed, 242 insertions(+) create mode 100644 docs/design/plan-gate-doe/REPORT.md create mode 100644 docs/design/turn-wall-share-doe/README.md diff --git a/docs/design/plan-gate-doe/REPORT.md b/docs/design/plan-gate-doe/REPORT.md new file mode 100644 index 0000000000..3b88a64f39 --- /dev/null +++ b/docs/design/plan-gate-doe/REPORT.md @@ -0,0 +1,122 @@ +# The plan-gate experiment — planner arm × split-gate mode + +*On 2026-09-02 the owner chose planner B with the split gate OFF from this front. +The gate had shipped armed, with a six-item floor under every division, and +[#418](https://github.com/Agent-Field/codeaf/issues/418) said the floor folds +real divisions; #384 proposed three planner repairs. Neither question was going +to be settled by argument, so both were made factors of one designed experiment +and the winner was taken from the measurement. What landed in PR #435 is the +decision: unpinned, the gate has no say; `CODEAF_SPLITGATE=1` puts the shipped +count back and `judgment` asks the plan's own sizing. The `lanes` counting arm +lost and its code is gone.* + +*What follows is the report as it was written, and it is +planner arm × split-gate mode, plan door, deepseek-v4-flash (kimi-k3 on R1d), 2026-09-02.* + +## DECISION + +**Planner B (stages), split gate off.** From the front table below, on the +pre-registered tiebreak of quality > cost > wall, with unrecoverable draws as +the disqualifier — the report's own front table, quoted: + +## The front (arm × gate) +| arm | gate | draws | quality | unrecoverable | cost $ | wall s | front | +|---|---|---|---|---|---|---|---| +| D | lanes | 21 | 0.71 | 12/21 | 0.0277 | 25 | YES | +| C | lanes | 21 | 0.71 | 18/21 | 0.0209 | 22 | YES | +| B | off | 21 | 0.71 | 4/21 | 0.0129 | 27 | YES | +| B | lanes | 21 | 0.61 | 6/21 | 0.0186 | 25 | YES | +| A | lanes | 21 | 0.60 | 4/21 | 0.0142 | 21 | YES | +| D | off | 21 | 0.56 | 12/21 | 0.0143 | 28 | | +| C | off | 21 | 0.56 | 15/21 | 0.0136 | 28 | | +| B | shipped | 14 | 0.50 | 3/14 | 0.0105 | 24 | YES | +| A | off | 14 | 0.48 | 5/14 | 0.0194 | 28 | | +| A | shipped | 14 | 0.43 | 1/14 | 0.0175 | 22 | YES | +| B | judgment | 14 | 0.36 | 1/14 | 0.0138 | 22 | YES | +| A | judgment | 14 | 0.33 | 4/14 | 0.0144 | 28 | | +| D | judgment | 14 | 0.29 | 5/14 | 0.0141 | 26 | | +| D | shipped | 14 | 0.29 | 6/14 | 0.0160 | 25 | | +| C | shipped | 14 | 0.29 | 6/14 | 0.0255 | 21 | | +| C | judgment | 14 | 0.21 | 3/14 | 0.0120 | 19 | YES | + + +Three cells tie at the top on quality, 0.71. **B off** carries the fewest +unrecoverable draws of the three — 4 of 21, against 12 and 18 for the two +`lanes` cells that reach the same quality — and the lowest cost, $0.0129 against +D lanes' $0.0277 for no more quality. `shipped`, the gate as it was, is on the +front only at 0.50 and 0.43, below the same arm's off cell in both cases. + +The gate's own main effect is **not consistent across planners**: `lanes` beats +`off` under planner A (0.60 vs 0.48) and loses to it under planner B (0.61 vs +0.71). An interaction that changes sign is not an improvement to ship, and it is +why the counting arm was deleted rather than kept as a fourth pin. `judgment` is +kept as a pin because it is one-directional — it can only add keeps to the count +— and because nothing in the front argues it does harm where somebody wants a +floor with an escape. + +**What this decision does not say.** No cell satisfied the pre-registered null on +every brief, and n=3 differences under ~0.15 are coins (planner A under `lanes` +fell from 0.76 to 0.60 on one more draw per brief). This picked the best front +cell on the responses that were measured; it did not prove the gate harmful. +Tier 2 — full runs on real issues — was never run. + +Pre-registered before any cell ran: the factors, the briefs, the responses, the tiebreak. Every draw was judged +blind from the graph JSON alone, against a lanes-matched rule written for each brief before its first plan. +Draws: 16 cells × 7 briefs × 2, plus a third draw on the seven cells within reach of the n=2 front (273 judged draws, ≈$5). +Tiebreak stated in advance: quality > cost > wall; unrecoverable = oversized-undivided / no graph / dead fan-out. + +## The front (arm × gate) +| arm | gate | draws | quality | unrecoverable | cost $ | wall s | front | +|---|---|---|---|---|---|---|---| +| D | lanes | 21 | 0.71 | 12/21 | 0.0277 | 25 | YES | +| C | lanes | 21 | 0.71 | 18/21 | 0.0209 | 22 | YES | +| B | off | 21 | 0.71 | 4/21 | 0.0129 | 27 | YES | +| B | lanes | 21 | 0.61 | 6/21 | 0.0186 | 25 | YES | +| A | lanes | 21 | 0.60 | 4/21 | 0.0142 | 21 | YES | +| D | off | 21 | 0.56 | 12/21 | 0.0143 | 28 | | +| C | off | 21 | 0.56 | 15/21 | 0.0136 | 28 | | +| B | shipped | 14 | 0.50 | 3/14 | 0.0105 | 24 | YES | +| A | off | 14 | 0.48 | 5/14 | 0.0194 | 28 | | +| A | shipped | 14 | 0.43 | 1/14 | 0.0175 | 22 | YES | +| B | judgment | 14 | 0.36 | 1/14 | 0.0138 | 22 | YES | +| A | judgment | 14 | 0.33 | 4/14 | 0.0144 | 28 | | +| D | judgment | 14 | 0.29 | 5/14 | 0.0141 | 26 | | +| D | shipped | 14 | 0.29 | 6/14 | 0.0160 | 25 | | +| C | shipped | 14 | 0.29 | 6/14 | 0.0255 | 21 | | +| C | judgment | 14 | 0.21 | 3/14 | 0.0120 | 19 | YES | + +## Per brief, front cells against the two nulls (mean; draws; u = unrecoverable draws) +| brief | A shipped | A off | A lanes | B off | B lanes | D lanes | C lanes | +|---|---|---|---|---|---|---|---| +| R1a | 1.00 (1.00/1.00) u0 | 0.00 (0.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u0 | 1.00 (1.00/1.00/1.00) u0 | 0.33 (0.00/0.00/1.00) u0 | 0.67 (0.00/1.00/1.00) u2 | 0.33 (0.00/1.00/0.00) u1 | +| R1b | 0.50 (0.00/1.00) u0 | 1.00 (1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u0 | 0.67 (1.00/0.00/1.00) u0 | 1.00 (1.00/1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u3 | +| R1c | 0.50 (1.00/0.00) u1 | 0.50 (0.00/1.00) u2 | 0.11 (0.33/0.00/0.00) u1 | 0.33 (0.00/0.00/1.00) u1 | 0.67 (0.00/1.00/1.00) u1 | 0.00 (0.00/0.00/0.00) u0 | 0.67 (1.00/0.00/1.00) u2 | +| R1d | 1.00 (1.00/1.00) u0 | 0.50 (1.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u1 | 0.56 (0.00/1.00/0.67) u1 | 0.33 (0.00/1.00/0.00) u3 | 1.00 (1.00/1.00/1.00) u1 | 0.33 (0.00/1.00/0.00) u3 | +| H1 | 0.00 (0.00/0.00) u0 | 0.38 (0.75/0.00) u0 | 0.42 (0.25/0.00/1.00) u0 | 0.50 (0.00/1.00/0.50) u0 | 0.25 (0.50/0.00/0.25) u1 | 1.00 (1.00/1.00/1.00) u3 | 0.75 (0.25/1.00/1.00) u3 | +| H2 | 0.00 (0.00/0.00) u0 | 0.50 (1.00/0.00) u0 | 1.00 (1.00/1.00/1.00) u1 | 0.89 (0.67/1.00/1.00) u0 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u3 | 1.00 (1.00/1.00/1.00) u3 | +| H3 | 0.00 (0.00/0.00) u0 | 0.50 (1.00/0.00) u1 | 0.67 (1.00/1.00/0.00) u1 | 1.00 (1.00/1.00/1.00) u2 | 1.00 (1.00/1.00/1.00) u0 | 0.67 (1.00/0.00/1.00) u2 | 0.89 (0.67/1.00/1.00) u3 | + +## Reading +- No cell satisfies the pre-registered null on every brief: the original #252 brief on the default model (R1c) is a coin for + every arm (A shipped 0.50, B off 0.33, D lanes 0.00, C lanes 0.67), and R1d (kimi-k3) stays at 0.56–1.00. +- Among the 0.71 tie, B off has the fewest unrecoverable draws (4/21) and the lowest cost ($0.013); D lanes and C lanes + reach the same quality only by carrying the measurement flag on 12/21 and 18/21 draws (see finding 2), and D costs 2×. +- Gate main effect is not consistent across arms (interaction): lanes beats off in A (0.60 vs 0.48) and loses to off in + B (0.61 vs 0.71), because B under lanes collapses R1a on 2 of 3 draws (one atomic node, "no two pieces could be named"). + Shipped and judgment fold every held-out brief in every arm (judgment's fallback count is the shipped one). +- Planner main effect at a fixed gate: B ≥ A (off 0.71 vs 0.48; shipped 0.50 vs 0.43); C and D are dominated by their flag mass. +- Unrecoverable-draw recovery (#384 null): A off 5/14 (R1a, R1c×2, R1d, H3) → B off 4/21 (R1c, R1d, H3×2): R1a's collapse is + recovered, R1c/R1d are not. +- n=3 is still a coin: A lanes fell from 0.76 (n=2) to 0.60 with one more draw per brief. Differences under ~0.15 are not decisions. + +## Findings for other owners +1. Plan door verdict vs tests (#428 family, for codeaf-3f): the door exits 0 with a graph on 103 of 273 draws that hold an + oversized node carrying a refusal word (A 14, B 13, C 42, D 34); cell ids and node ids are with the raw draws. +2. Measurement defect in #425 (arms C, D): a per-lane node still names the whole file, so "its named material exceeds what one + worker holds" is stamped on correctly divided lanes (R1a 3/3 lanes, R1b, H2, H3), and expansion cannot divide a lane over one + file further. The measure needs the lane's share of the file, or must not veto a node whose summary scopes a region. +3. #424 under lanes: a stage that holds three lanes is sized atomic with no parts and refused as unnamed, so the stage question + is never asked (B lanes R1a 2/3 draws). The atomic-until-proven pin and the measured window disagree on a 144 KB file. +4. Panel copies: C draws a three-identical-pass Panel on R1c 6/8 and R1b 3/8 — the ensemble hook fires where A splits. + +## Tier 2 (full runs on real issues) not run: the pool has one anchor plus isort-2562, and the WIP freeze holds new work. diff --git a/docs/design/turn-wall-share-doe/README.md b/docs/design/turn-wall-share-doe/README.md new file mode 100644 index 0000000000..81253e5540 --- /dev/null +++ b/docs/design/turn-wall-share-doe/README.md @@ -0,0 +1,120 @@ +# The share of the wall a turn may spend inline: the experiment (2026-09-03) + +This folder is the record of the run that settles `turnWallShare` +(internal/session/turnwall.go), the fraction of an unattended run's wall that ONE +turn may spend working inline before what is left of it moves onto a task +([#546](https://github.com/Agent-Field/codeaf/issues/546), landed as the +constant at 3). The law and the mechanism are on the issue and in the code; this +is the evidence for the number. + +## The question + +Under a steward with a wall, a turn hands over once its inline stretch passes +`Wall / turnWallShare`. Everything else about the seam is fixed — the road, the +line, the once-per-turn claim, the boundaries it is asked at — so the only free +parameter is the divisor, and it trades one failure against the other: + +- **Too large a share** (the divisor too small) is the measured failure that + opened #546: the tox-4031 cell under `--yolo` on a 900 s wall spent five sixths + of it reading and running tests inline, moved the work with 147 s left, spent 60 + of those opening the task's working copy, and hit the wall with three changed + files, no commit, no check and no landing. +- **Too small a share** (the divisor too large) converts turns that were about to + finish. The road can decline — the running model and the mark's own reader have + to agree nothing is left — but each ask is two model calls, and a seam that + fires early on every run pays for them on every run. + +## The two arms + +| Arm | `turnWallShare` | Inline stretch allowed on a 900 s wall | +| --- | --- | --- | +| A | 3 (shipped) | 300 s | +| B | 2 | 450 s | + +Arm A is the branch as merged, byte for byte. Arm B is a one-line scratch commit +on top of it changing the constant and nothing else; the note's wording is not a +factor and is left as it is in both. + +## The cells + +`attrs`, `tox` and `reef` from the canary pool, each run through the chat door +under `--yolo` with a wall, at both arms. The rig, the pool and the driver are the +gatekeeper's (issue [#407](https://github.com/Agent-Field/codeaf/issues/407)); +`tox` is the cell the defect was measured on and is the one that must move. + +## The method + +**Pareto on ends-done-before-wall, then cost.** An arm that ends more cells at +done before the wall wins outright; arms that tie on that are separated by cost. +Wall time is recorded but is not a tiebreak — a run that ends early because it +gave up is not a better run — and `files` is recorded because the failure this +seam exists to prevent shows up there as changed files with nothing landed. + +## What the first pass found: the rig gave the run no wall + +The first pass of both arms produced no `ran long` row in any cell, and reading +the cells' launch lines said why. The canary's chat driver starts +`codeaf chat -yolo -one-model -model … -max-cost ` and passes no +`-max-hours`, so `Budget().Wall` was zero in every chat cell and the share seam, +which is asked only under a wall, never fired. Every "wall" the canary had recorded +for a chat cell so far was the tmux clock killing the pane from outside, never the +run's own wall — which also means the tox-4031 stretch that opened #546 was +measured against a clock the run could not see. + +The pass was rerun through a wrapper binary that appends `-max-hours 0.25` when +the first argument is `chat` (a 900 s wall, matching the table above), with the +rig itself untouched. Those are the rows below. The first, wall-less pass is kept +beside them on the rig as the control: with no wall, both arms are the shipped +seam with the share switched off. + +## Results + +Both arms, three cells each, chat door under `--yolo`, a 900 s wall (2026-09-03, +runs `20260903T204602Z-b07c3dd51-doe546-wall-share3` and +`20260903T211119Z-15a2b6ef2-doe546-wall-share2` on the canary rig, tables posted +on #407). `seam` says which door moved the work out of the turn, if any. + +| share | cell | ends done before wall | wall s | cost | files | seam | +| --- | --- | --- | --- | --- | --- | --- | +| 3 | reef | no | 903 | $0.085 | 2 | split at 230 s; task back at 589 s; **share at 894 s** on a turn begun at 593 s | +| 3 | attrs | no | 903 | $0.130 | 0 | parts at 239 s, three tasks; one still running at the wall | +| 3 | tox | no | 902 | $0.096 | 2 | split at 199 s; task done 462 s later; its check still running at the wall | +| 2 | reef | yes | 249 | $0.012 | 1 | none; inline, done at 202 s | +| 2 | attrs | no | 903 | $0.091 | 1 | split at 287 s; task done 360 s later; its second check cut by the wall | +| 2 | tox | yes | 202 | $0.010 | 1 | none; inline, done at 181 s | + +The control pass with no wall (same binaries, same cells, the rig as it stands): +arm A ended 0 of 3 before the tmux clock, arm B 1 of 3 (reef, 114 s), and the +share seam fired in none of the six. + +## The read: the number was not what this run measured + +Counted by the method, arm B wins 2–0 on ends-done-before-wall. But neither of +those two cells reached EITHER arm's share: they ended inline at 181 s and 202 s, +under the 300 s that arm A allows, and the difference between the arms is the +running model's variance on a cell, not the constant. **`turnWallShare` stays at +3.** The run is a null result on the number and a measured result on three other +things: + +1. **The split seam gets there first, and that is the rig's limit.** Every cell + that ran long moved its work at 199–287 s through writeseam.go's door, before a + third of a 900 s wall. On a short wall the share is close to unreachable; the + wall it was written for is the long one, where a turn can read and run tests + for an hour without crossing the write seam's file count. The canary's cells + run on a fifteen-minute wall, so the share's own case is outside what this rig + can measure, and the null result on the number is a null result at that wall. +2. **The share is taken off the whole wall from the turn's own start, so a turn + that begins late gets a third the wall no longer has.** The one firing in six + cells was reef: a turn that began with 310 s left ran its full 300 s and handed + over at 894 s, six seconds before the wall — the exact shape #546 opened with. + The seam now hands over only what can still be checked before the wall: past + the share it fires only while at least a task's setup and check (`taskAllowance` + in turnwall.go, the two limits a task is already held to) is left, and a turn + with less runs to the wall inline with nothing spent asking. Landed in the same + pull request with reef's 894 s as the before in + `TestATurnThatCannotBeCheckedBeforeTheWallIsNotMoved`. +3. **What actually ran to the wall was the task's check.** In three of the six + wall cells the task had finished and the check that reads it — the full suite, + run once by the task and again by each check, 150 s a time on these projects — + was still running when the wall came down. The share buys a task time to be + checked; on a 900 s wall the check itself is what there is not time for. From 45e10534a2dd0ff10b3d9ddee8f754abf0b93a09 Mon Sep 17 00:00:00 2001 From: Abir Abbas Date: Wed, 16 Sep 2026 17:18:11 -0400 Subject: [PATCH 6/8] test/ux: the revived journey suite goes back out #991 removed it because it drove the v1 and v2 surfaces, and the revival did not change that: its journal verb reads graph.db tables (messages, nodes, usage, commands) that the v3 chat never writes, lib.sh still waits for the resident, nothing runs it from the Makefile or a workflow, and its only evidence was bash -n. docs/JOURNEY.md says internal/e2e is the one suite that drives the binary, and that stays true. A suite that has been run green against bin/codeaf can come back on its own pull request with a target that runs it. Co-Authored-By: Claude Fable 5.1 --- test/ux/journeys/01-ask-small.sh | 24 -- test/ux/journeys/02-commission.sh | 56 --- test/ux/journeys/03-status-read.sh | 35 -- test/ux/journeys/04-steer.sh | 67 ---- test/ux/journeys/05-correct.sh | 47 --- test/ux/journeys/06-answer-question.sh | 48 --- test/ux/journeys/07-teach.sh | 36 -- test/ux/journeys/08-charter.sh | 64 ---- test/ux/journeys/09-consent.sh | 58 --- test/ux/journeys/10-money.sh | 44 --- test/ux/journeys/11-what-did-you-learn.sh | 41 -- test/ux/journeys/12-leave-return.sh | 60 --- test/ux/journeys/13-second-window.sh | 52 --- test/ux/journeys/14-craft.sh | 57 --- test/ux/journeys/15-fanout.sh | 79 ---- test/ux/journeys/16-model-words.sh | 57 --- test/ux/journeys/17-attachments.sh | 63 ---- test/ux/journeys/18-take-out.sh | 52 --- test/ux/journeys/19-self-pitch.sh | 39 -- test/ux/judge.py | 104 ------ test/ux/lib.sh | 336 ----------------- test/ux/quality/20-coding.sh | 115 ------ test/ux/quality/21-research.sh | 69 ---- test/ux/quality/22-quicker.sh | 59 --- test/ux/quality/23-learning-applied.sh | 60 --- test/ux/quality/24-shape.sh | 106 ------ test/ux/quality/25-writing.sh | 56 --- test/ux/quality/26-data.sh | 61 --- test/ux/quality/27-errand.sh | 54 --- test/ux/run.sh | 435 ---------------------- test/ux/tier-matrix.sh | 134 ------- 31 files changed, 2568 deletions(-) delete mode 100755 test/ux/journeys/01-ask-small.sh delete mode 100755 test/ux/journeys/02-commission.sh delete mode 100755 test/ux/journeys/03-status-read.sh delete mode 100755 test/ux/journeys/04-steer.sh delete mode 100755 test/ux/journeys/05-correct.sh delete mode 100755 test/ux/journeys/06-answer-question.sh delete mode 100755 test/ux/journeys/07-teach.sh delete mode 100755 test/ux/journeys/08-charter.sh delete mode 100755 test/ux/journeys/09-consent.sh delete mode 100755 test/ux/journeys/10-money.sh delete mode 100755 test/ux/journeys/11-what-did-you-learn.sh delete mode 100755 test/ux/journeys/12-leave-return.sh delete mode 100755 test/ux/journeys/13-second-window.sh delete mode 100755 test/ux/journeys/14-craft.sh delete mode 100755 test/ux/journeys/15-fanout.sh delete mode 100755 test/ux/journeys/16-model-words.sh delete mode 100755 test/ux/journeys/17-attachments.sh delete mode 100755 test/ux/journeys/18-take-out.sh delete mode 100755 test/ux/journeys/19-self-pitch.sh delete mode 100644 test/ux/judge.py delete mode 100644 test/ux/lib.sh delete mode 100755 test/ux/quality/20-coding.sh delete mode 100755 test/ux/quality/21-research.sh delete mode 100755 test/ux/quality/22-quicker.sh delete mode 100755 test/ux/quality/23-learning-applied.sh delete mode 100755 test/ux/quality/24-shape.sh delete mode 100755 test/ux/quality/25-writing.sh delete mode 100755 test/ux/quality/26-data.sh delete mode 100755 test/ux/quality/27-errand.sh delete mode 100755 test/ux/run.sh delete mode 100755 test/ux/tier-matrix.sh diff --git a/test/ux/journeys/01-ask-small.sh b/test/ux/journeys/01-ask-small.sh deleted file mode 100755 index 048e289aa4..0000000000 --- a/test/ux/journeys/01-ask-small.sh +++ /dev/null @@ -1,24 +0,0 @@ -#!/usr/bin/env bash -# J1 · Ask a small thing -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J1 Ask a small thing — "what'"'"'s 2^32?" → direct answer in thread, no job ceremony' - -expect_nodes 0 -since="$(mark)" - -say "what is 2 to the power 10?" -assert_screen '1024' 'the thread answers with the number' 90 -snap answered - -# No job ceremony: a small question must not become a work order. -assert_journal_is "select count(*) from commands where seq > $since and kind='splice'" 0 \ - 'no splice command — a question is not a commission' -assert_journal_is "select count(*) from nodes where created_seq > $since and origin='user'" 0 \ - 'no user-origin task node was created' -assert_journal_is "select count(*) from messages where seq > $since and role='agent'" 1 \ - 'exactly one agent message for the turn' -assert_journal "select count(*) from messages where seq > $since and role='agent' and body like '%1024%'" \ - 'the durable agent message holds the answer' - -dump_turn "$since" -finish diff --git a/test/ux/journeys/02-commission.sh b/test/ux/journeys/02-commission.sh deleted file mode 100755 index 77eda16505..0000000000 --- a/test/ux/journeys/02-commission.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# J2 · Commission work -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J2 Commission work — "build me X" → work order journaled, tasks on the board, deliverable lands in the thread answer-first' - -expect_nodes 1 -since="$(mark)" - -say "write a haiku about rivers into a file and show me the haiku" - -# The deliverable is answer-first: the words, not a path to go and read. -assert_journal "select count(*) from commands where seq > $since and kind='splice' and status='applied'" \ - 'a splice command was journaled and applied' 60 -assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ - 'the commissioned node reached done' "$UX_JOB_TIMEOUT" - -node="$(journal "select id from nodes where created_seq > $since and origin='user' order by created_seq limit 1")" -record 'task node' "${node:-}" -record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since and origin='user'")" - -# "I'm on it — I'll have a haiku about rivers written to a file" also contains -# the word "rivers" and is longer than forty characters. It is a promise, not a -# deliverable, and an answer-first check that a promise can satisfy is not a -# check. So this asks for the message the finished NODE produced. -assert_journal "select count(*) from messages where seq > $since and node_id='$node' and lower(body) like '%river%'" \ - 'the finished job posted its own message into the thread' 90 -assert_journal "select count(*) from messages where seq > $since and node_id='$node' and length(body) > 60 and (body like '%' || char(10) || '%')" \ - 'that message carries the haiku itself — several lines of it — not only a path' 30 - -deliverable="$(journal "select body from messages where seq > $since and node_id='$node' order by seq desc limit 1")" -record 'the deliverable message' "$(printf '%s' "$deliverable" | tr '\n' ' / ' | head -c 400)" -if printf '%s' "$deliverable" | grep -Eqi "i'?m on it|i will|i'll have|when it'?s done"; then - _check no 'the deliverable is the work, not another promise' \ - 'the haiku, not a sentence about writing one' "$(printf '%s' "$deliverable" | head -c 120)" -else - _check yes 'the deliverable is the work, not another promise' \ - 'the haiku, not a sentence about writing one' 'no promise language in the delivered message' -fi - -assert_screen 'river' 'the thread shows the haiku on screen' 30 -snap delivered - -# Nothing else happened: no question left dangling, no failed node. -assert_journal_is "select count(*) from agent_questions where seq > $since and status in ('pending','asked')" 0 \ - 'no orphaned question was left open' -assert_journal_is "select count(*) from nodes where created_seq > $since and status='failed'" 0 \ - 'no node failed' - -# Keep the haiku's file path for the journeys that correct it later. -journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')" \ - > "$UX_DIR/deliverable.txt" - -judge_this 'write a haiku about rivers into a file and show me the haiku' "$(journal "select group_concat(body, char(10)) from messages where seq > $since and node_id='$node'")" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/03-status-read.sh b/test/ux/journeys/03-status-read.sh deleted file mode 100755 index b88a7611de..0000000000 --- a/test/ux/journeys/03-status-read.sh +++ /dev/null @@ -1,35 +0,0 @@ -#!/usr/bin/env bash -# J3 · Watch progress -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J3 Watch progress — "how'"'"'s it going" is a read of the live graph, never a refusal' - -expect_nodes 0 -since="$(mark)" - -say "did you finish the haiku?" -assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied at all' 120 -sleep 6 -snap replied - -reply="$(journal "select group_concat(lower(body), ' ') from messages where seq > $since and role='agent'")" -record 'reply' "$(printf '%s' "$reply" | head -c 400)" - -# The read must reflect the graph. The haiku node is done, so the honest answer -# says so — and must never be a refusal to look. -if printf '%s' "$reply" | grep -Eq 'done|finish|complete|wrote|written|created|yes'; then - _check yes 'the status read reflects the finished haiku' 'a reply naming the work as done' "$(printf '%s' "$reply" | head -c 160)" -else - _check no 'the status read reflects the finished haiku' 'a reply naming the work as done' "$(printf '%s' "$reply" | head -c 160)" -fi - -if printf '%s' "$reply" | grep -Eq "i can'?t|cannot|unable to|don'?t have access|no access|i do not have"; then - _check no 'status is a read, not a refusal' 'no refusal language' "$(printf '%s' "$reply" | head -c 160)" -else - _check yes 'status is a read, not a refusal' 'no refusal language' 'none found' -fi - -assert_journal "select count(*) from messages where seq > $since and role='agent'" \ - 'the question got exactly one reply in the thread' - -dump_turn "$since" -finish diff --git a/test/ux/journeys/04-steer.sh b/test/ux/journeys/04-steer.sh deleted file mode 100755 index fa4d80bc9e..0000000000 --- a/test/ux/journeys/04-steer.sh +++ /dev/null @@ -1,67 +0,0 @@ -#!/usr/bin/env bash -# J4 · Steer running work -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J4 Steer running work — words reach running workers; words arriving after landing are called out, not eaten' - -expect_nodes 1 -# Whether a steer becomes a journaled redirect COMMAND or is absorbed by the -# head into a plain reply is a model decision, and it flips between runs on the -# same surface. So a red verdict on this journey is evidence of nothing until -# it repeats — read it beside a re-run before calling it a product gap. -note 'this journey is head-nondeterministic: the redirect command appears on some runs and not others' -since="$(mark)" - -say "write a 6-verse poem about mountains, one verse at a time, into a file" -sleep 12 # let the splice land and a worker start -snap running - -steer_at="$(mark)" -say "make it about the sea instead" - -assert_journal "select count(*) from commands where seq > $steer_at and kind in ('redirect','steer')" \ - 'a redirect command was journaled' 90 - -status="$(journal "select status from commands where seq > $steer_at and kind in ('redirect','steer') order by seq limit 1")" -result="$(journal "select result from commands where seq > $steer_at and kind in ('redirect','steer') order by seq limit 1")" -record 'redirect status' "${status:-}" -record 'redirect result' "${result:-}" - -sleep 20 - -# Only the head's own reply to the steer is evidence. The deliverable that -# lands afterwards is not: a poem is free to contain the word "sea" without -# anyone having steered anything, and grading on that would let the journey -# pass on a coincidence. -reply="$(journal "select lower(body) from messages where seq > $steer_at and role='agent' order by seq limit 1")" -record 'the head'"'"'s reply to the steer' "${reply:-}" -record 'everything said afterwards' "$(journal "select group_concat(substr(body,1,120),' | ') from messages where seq > $steer_at and role in ('agent','system')" | head -c 600)" - -if printf '%s' "$reply" | grep -Eq "couldn'?t queue|could not queue|try again"; then - _check no 'the redirect produced a reply saying what happened to the words' \ - 'either "passed to the workers" or an honest too-late line' \ - "the generic command-failure line: \"$reply\"" - record 'outcome' 'REFUSED — the head returned its generic "I couldn'"'"'t queue that change — try again" instead of saying the work had already landed' -elif printf '%s' "$reply" | grep -Eq 'passed|reached|worker|redirect|steer|now working|carrying it|took it'; then - _check yes 'the redirect landed and one reply says what was done with the words' \ - 'a reply naming where the words went' "$reply" - record 'outcome' 'LANDED — the words reached running work' -elif printf '%s' "$reply" | grep -Eq 'already|too late|finished|had landed|completed before|done before|by the time'; then - _check yes 'the redirect arrived after landing and was called out, not eaten' \ - 'an honest too-late line' "$reply" - record 'outcome' 'TOO LATE — called out honestly (the mechanism works either way)' -else - _check no 'the redirect produced a reply saying what happened to the words' \ - 'either "passed to the workers" or an honest too-late line' "${reply:-}" - record 'outcome' 'SILENT — the cardinal UX sin' -fi - -assert_journal "select count(*) from messages where seq > $steer_at and role in ('agent','system')" \ - 'the steer was not eaten in silence' - -settle 60 -snap after - -judge_this 'write a 6-verse poem, one verse at a time, into a file — then, while it ran: make it about the sea instead' "$(deliverable_since "$since")" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/05-correct.sh b/test/ux/journeys/05-correct.sh deleted file mode 100755 index 19da01e6db..0000000000 --- a/test/ux/journeys/05-correct.sh +++ /dev/null @@ -1,47 +0,0 @@ -#!/usr/bin/env bash -# J5 · Correct delivered work -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J5 Correct delivered work — "that'"'"'s wrong" spawns a revision beside the original, inheriting its workspace' - -expect_nodes 1 -since="$(mark)" - -say "that haiku is wrong — the haiku must mention a heron. fix it and show me the new haiku" - -# Either shape counts as the mechanism: a surgery command against the original, -# or a correction node spliced beside it. The report says which one landed. -assert_journal "select count(*) from commands where seq > $since and kind in ('splice','revise','surgery','correct','redirect')" \ - 'a correction command was journaled' 90 - -kind="$(journal "select group_concat(distinct kind) from commands where seq > $since")" -record 'command kinds' "${kind:-}" - -assert_journal "select count(*) from nodes where created_seq > $since" \ - 'a revision node was created beside the original' 90 - -parent="$(journal "select parent_id || ' / retry_of=' || retry_of from nodes where created_seq > $since order by created_seq limit 1")" -record 'revision node lineage' "${parent:-}" - -assert_journal "select count(*) from nodes where created_seq > $since and status='done'" \ - 'the revision reached done' "$UX_JOB_TIMEOUT" - -assert_journal "select count(*) from messages where seq > $since and role in ('agent','system') and lower(body) like '%heron%'" \ - 'the new deliverable mentions a heron' 90 - -# The word "heron" is on screen the moment the user types it, so the screen -# check has to exclude the user's own sentence or it grades itself. -snap corrected -if pane | grep -i heron | grep -qvi 'must mention'; then - _check yes 'the corrected haiku is on screen, in the agent'"'"'s words not the user'"'"'s' \ - "a heron line that is not the user's own request" "$(pane | grep -i heron | grep -vi 'must mention' | head -1 | sed 's/^ *//')" -else - _check no 'the corrected haiku is on screen, in the agent'"'"'s words not the user'"'"'s' \ - "a heron line that is not the user's own request" "only the user's own sentence mentions a heron" -fi - -record 'the head'"'"'s reply' "$(journal "select group_concat(substr(body,1,200),' | ') from messages where seq > $since and role in ('agent','system')" | head -c 600)" - -judge_this 'that haiku is wrong — the haiku must mention a heron. fix it and show me the new haiku' "$(deliverable_since "$since")" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/06-answer-question.sh b/test/ux/journeys/06-answer-question.sh deleted file mode 100755 index a91e4b4ec8..0000000000 --- a/test/ux/journeys/06-answer-question.sh +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env bash -# J6 · Answer its questions -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J6 Answer its questions — type the number; the answer appears as the user'"'"'s message and a receipt lands in the thread' - -expect_nodes 0 - -since="$(mark)" - -# The cheapest reliable question generator is a standing rule: its proposal -# always asks to be ratified. -say "remind me every morning at 8am to drink water" - -assert_journal "select count(*) from agent_questions where seq > $since and status in ('asked','pending')" \ - 'the agent asked a question' 120 -# The question draws its options numbered, inline in the prose — the shape the -# store composes ("Should I keep watching this when you're not here? -# ▸ 1 yes, always · ▸ 2 only while I'm around"). This asserts exactly that. -assert_screen '▸ 1 ' 'numbered options render in the thread' 30 -snap asked - -qseq="$(journal "select seq from agent_questions where seq > $since order by seq limit 1")" -record 'question seq' "$qseq" -record 'question' "$(journal "select replace(substr(text,1,200),char(10),' / ') from agent_questions where seq=$qseq")" - -# A bare digit with an empty composer submits itself — no Enter; see lib.sh. -answer_number 1 -sleep 8 -snap answered - -assert_journal_is "select status from agent_questions where seq=$qseq" 'answered' \ - 'the question is recorded answered' -assert_journal "select answer_message_seq from agent_questions where seq=$qseq" \ - 'the question carries the answering message seq' - -ans="$(journal "select answer_message_seq from agent_questions where seq=$qseq")" -assert_journal_is "select role from messages where seq=$ans" 'user' \ - "the answer is journaled as the user's own message" -record 'resolution' "$(journal "select resolution from agent_questions where seq=$qseq")" -record "user's message" "$(journal "select body from messages where seq=$ans")" - -assert_journal "select count(*) from messages where seq > $ans and role in ('agent','system')" \ - 'a receipt landed in the thread after the answer' 60 -assert_screen 'stands?|standing|stood|ratif|watch' 'the receipt is visible on screen' 30 - -dismiss_standing_watch -dump_turn "$since" -finish diff --git a/test/ux/journeys/07-teach.sh b/test/ux/journeys/07-teach.sh deleted file mode 100755 index e22f8b6bf6..0000000000 --- a/test/ux/journeys/07-teach.sh +++ /dev/null @@ -1,36 +0,0 @@ -#!/usr/bin/env bash -# J7 · Teach a lesson -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J7 Teach a lesson — a stated lesson lands in the notebook and future jobs see it' - -expect_nodes 0 -since="$(mark)" - -say "always run what you build once before telling me it's done — remember that" -assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 120 -sleep 5 -snap taught - -# The notebook is the durable half. A stated lesson is channel=stated; the -# learning loop writes it asynchronously, so the window is generous. -assert_journal "select count(*) from facts where seq > $since" \ - 'something landed in the notebook' 180 - -assert_journal "select count(*) from facts where seq > $since and channel='stated'" \ - 'it was recorded as STATED, not inferred — the user said it out loud' -assert_journal "select count(*) from facts where seq > $since and kind in ('lesson','preference')" \ - 'it was filed as a lesson or a preference' -assert_journal "select count(*) from facts where seq > $since and (lower(body) like '%run%' or lower(body) like '%verif%')" \ - 'the fact body carries the substance of the lesson' - -record 'facts written' "$(journal "select group_concat(kind || '/' || channel || ': ' || substr(body,1,140), char(10)) from facts where seq > $since")" - -reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" -if printf '%s' "$reply" | grep -Eq "got it|noted|remember|i'?ll|will do|understood|reflected"; then - _check yes 'the thread confirms the lesson was taken' 'a confirming reply' "$(printf '%s' "$reply" | head -c 160)" -else - _check no 'the thread confirms the lesson was taken' 'a confirming reply' "${reply:-}" -fi - -dump_turn "$since" -finish diff --git a/test/ux/journeys/08-charter.sh b/test/ux/journeys/08-charter.sh deleted file mode 100755 index 1142f534a7..0000000000 --- a/test/ux/journeys/08-charter.sh +++ /dev/null @@ -1,64 +0,0 @@ -#!/usr/bin/env bash -# J8 · Stand up a rule -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J8 Stand up a rule — charter drafted and priced, ratified with one answer, and stood down just as sayably' - -expect_nodes 0 - -since="$(mark)" - -say "remind me every day at 9am to stretch" - -assert_screen 'when:|fires:' 'the drafted charter shows its cadence' 120 -assert_screen 'costs:' 'the draft carries a price line — per firing and per day' 30 -snap drafted -record 'price line' "$(pane | grep -i 'costs:' | head -1)" - -assert_journal "select count(*) from charters where created_seq > $since" \ - 'a charter row was written' 60 -charter="$(journal "select id from charters where created_seq > $since order by created_seq limit 1")" -record 'charter' "$charter" -assert_journal_is "select status from charters where id='$charter'" 'proposed' \ - 'the charter starts proposed, not active — nothing stands up unasked' - -# Answer only once the question is actually open in the journal. A digit typed -# at a draft that has not yet been asked lands in the composer instead, and the -# journey would then be measuring the suite's timing rather than the product. -assert_journal "select count(*) from agent_questions where seq > $since and status in ('pending','asked')" \ - 'the ratification question is open' 120 -qseq="$(journal "select seq from agent_questions where seq > $since and status in ('pending','asked') order by seq limit 1")" -answer_number 1 -wait_journal "select count(*) from agent_questions where seq=$qseq and status='answered'" 60 -sleep 8 -snap ratified -record 'ratification question' "$(journal "select status || ' → ' || resolution from agent_questions where seq=$qseq")" - -assert_journal_is "select status from charters where id='$charter'" 'active' \ - 'ratifying with one answer made the charter active' -assert_journal "select count(*) from commands where seq > $since and kind='charter_ratify' and status='applied'" \ - 'the ratification is journaled as an applied command' -record 'next due' "$(journal "select next_due from charters where id='$charter'")" - -dismiss_standing_watch - -# Stand-down is the other half of the journey, and it is said, not navigated. -down_at="$(mark)" -say "stand down the stretch reminder" -sleep 25 -snap stood-down - -status="$(journal "select status from charters where id='$charter'")" -record 'status after stand-down' "$status" -record 'commands after stand-down' "$(journal "select group_concat(kind || '/' || status, ', ') from commands where seq > $down_at")" -record 'reply to stand-down' "$(journal "select group_concat(substr(body,1,200),' | ') from messages where seq > $down_at and role in ('agent','system')")" - -if [ "$status" = "retired" ] || [ "$status" = "paused" ]; then - _check yes 'saying "stand down" retired the charter in the journal' \ - "charters.status in (retired,paused)" "$status" -else - _check no 'saying "stand down" retired the charter in the journal' \ - "charters.status in (retired,paused)" "$status — the thread may claim it, but the journal does not" -fi - -dump_turn "$since" -finish diff --git a/test/ux/journeys/09-consent.sh b/test/ux/journeys/09-consent.sh deleted file mode 100755 index 73eaa34d82..0000000000 --- a/test/ux/journeys/09-consent.sh +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env bash -# J9 · Approve spend (cheap variant) -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J9 Approve spend — price shown before purchase, consent asked, refusal honored' - -expect_nodes 1 - -# The real journey needs a plan estimated over $3, which is exactly the money a -# test suite may not spend. CODEAF_PLAN_CONSENT is the same gate with a movable -# line, so the cheap variant drops the line to a cent and asks the identical -# question about a job costing cents. Same code path, 1/300th of the money. -note 'variant: CODEAF_PLAN_CONSENT lowered to $0.0002 — below the estimate of a cents-scale job, so the same gate fires' - -ux_kill "$UX_SESSION" -sleep 2 -# A cent is still above what these jobs are estimated to cost, so the gate -# never fires at $0.01 — the line has to go below the estimate, not below the -# real $3 rail. -ux_launch "$UX_SESSION" "CODEAF_PLAN_CONSENT=0.0002" - -since="$(mark)" -say "research the three most common causes of bridge collapses and write me a short report file" - -assert_journal "select count(*) from agent_questions where seq > $since and category='plan-consent'" \ - 'the consent question was asked before the work was bought' 150 -snap asked - -qseq="$(journal "select seq from agent_questions where seq > $since and category='plan-consent' order by seq limit 1")" -record 'consent question' "$(journal "select replace(substr(text,1,300),char(10),' / ') from agent_questions where seq=$qseq")" - -if pane | grep -Eqi '\$[0-9]'; then - _check yes 'the price is shown before the purchase' 'a dollar figure on screen' "$(pane | grep -Eo '\$[0-9][0-9.]*' | head -3 | tr '\n' ' ')" -else - _check no 'the price is shown before the purchase' 'a dollar figure on screen' 'no dollar figure in the ask' -fi - -held_before="$(journal "select count(*) from nodes where created_seq > $since and status in ('running','done')")" -record 'nodes running or done while consent is outstanding' "$held_before" - -# Refuse it. -answer_number 2 -sleep 12 -snap refused - -assert_journal_is "select status from agent_questions where seq=$qseq" 'answered' 'the refusal was recorded' -record 'resolution' "$(journal "select resolution from agent_questions where seq=$qseq")" - -assert_journal "select count(*) from nodes where created_seq > $since and held=1" \ - 'refusal is honored — the plan is held, not run' 30 -assert_journal_is "select count(*) from nodes where created_seq > $since and status='done'" 0 \ - 'nothing was executed after the refusal' -record 'hold reason' "$(journal "select group_concat(distinct error) from nodes where created_seq > $since")" - -# Put the suite back on the ordinary rail for the journeys that follow. -ux_kill "$UX_SESSION"; sleep 2; ux_launch "$UX_SESSION" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/10-money.sh b/test/ux/journeys/10-money.sh deleted file mode 100755 index 08daeeac89..0000000000 --- a/test/ux/journeys/10-money.sh +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env bash -# J10 · Ask about money -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J10 Ask about money — per-job and windowed spend readable in the thread, and always-on on the foot' - -expect_nodes 0 - -since="$(mark)" -actual="$(spend)" -record 'journal spend so far' "\$$actual" - -say "what have you spent so far today?" -assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 120 -sleep 6 -snap answered - -reply="$(journal "select group_concat(body,' ') from messages where seq > $since and role in ('agent','system')")" -record 'reply' "$(printf '%s' "$reply" | head -c 400)" - -if printf '%s' "$reply" | grep -Eq '\$[0-9]|[0-9]+ ?¢|cent'; then - _check yes 'the reply carries a real money figure' 'a dollar or cent amount' "$(printf '%s' "$reply" | grep -Eo '\$[0-9.]+|[0-9]+ ?¢' | head -3 | tr '\n' ' ')" -else - _check no 'the reply carries a real money figure' 'a dollar or cent amount' "$(printf '%s' "$reply" | head -c 200)" -fi - -# The always-on half of the same fact — the money you never had to ask for. -# -# v3 carries the bill on the frame's foot row, beside the cache and context -# readings ("$0.27 · ⟲ saved $0.0038 · 58% cached … 66.8k/1.3M · 5% …"), and on -# the status deck's first row beside the session name. The foot is the row the -# frame always draws, so this reads the foot — the same fact, at the address -# the frame gives it. -foot="$(pane | tail -3 | grep -E '\$' | head -1)" -record 'foot row' "$(printf '%s' "$foot" | sed 's/^ *//')" -if printf '%s' "$foot" | grep -Eq '\$[0-9]'; then - _check yes 'the foot carries spend without being asked' \ - 'a dollar figure on the foot row' "$(printf '%s' "$foot" | grep -Eo '\$[0-9.]+[KM]?' | head -1)" -else - _check no 'the foot carries spend without being asked' \ - 'a dollar figure on the foot row' "${foot:-no foot row with a dollar figure}" -fi - -dump_turn "$since" -finish diff --git a/test/ux/journeys/11-what-did-you-learn.sh b/test/ux/journeys/11-what-did-you-learn.sh deleted file mode 100755 index 2e641d6f25..0000000000 --- a/test/ux/journeys/11-what-did-you-learn.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash -# J11 · Ask what it learned -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J11 Ask what it learned — a read over the notebook: lessons, beliefs, retractions' - -expect_nodes 0 -since="$(mark)" -record 'notebook holds' "$(journal "select count(*) from facts where status='active'") active facts" - -say "what have you learned about how I like work done?" -assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 150 -sleep 8 -snap answered - -reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" -record 'reply' "$(printf '%s' "$reply" | head -c 600)" - -# The proof it read the notebook rather than improvising is that J7's lesson — -# run what you build before calling it done — comes back in substance. -if printf '%s' "$reply" | grep -Eq 'run |verif|before .*done|test'; then - _check yes "the answer draws on J7's lesson in substance" \ - 'the run-it-before-done lesson, in its own words' "$(printf '%s' "$reply" | head -c 220)" -else - _check no "the answer draws on J7's lesson in substance" \ - 'the run-it-before-done lesson, in its own words' "$(printf '%s' "$reply" | head -c 220)" -fi - -if printf '%s' "$reply" | grep -Eq "i haven'?t learned|nothing yet|no lessons|i don'?t have|cannot recall"; then - _check no 'it does not refuse a question about itself' 'a substantive answer' "$(printf '%s' "$reply" | head -c 200)" -else - _check yes 'it does not refuse a question about itself' 'a substantive answer' 'no refusal' -fi - -length="$(journal "select coalesce(max(length(body)),0) from messages where seq > $since and role in ('agent','system')")" -record 'longest reply (bytes)' "$length" -[ "${length:-0}" -gt 60 ] \ - && _check yes 'the answer has substance' 'more than a shrug' "$length bytes" \ - || _check no 'the answer has substance' 'more than a shrug' "$length bytes" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/12-leave-return.sh b/test/ux/journeys/12-leave-return.sh deleted file mode 100755 index debfe6571e..0000000000 --- a/test/ux/journeys/12-leave-return.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env bash -# J12 · Leave and return -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J12 Leave and return — close the terminal, come back: the same thread resumes and a brief covers what landed' - -expect_nodes 0 - -before_messages="$(journal "select count(*) from messages")" -record 'messages before leaving' "$before_messages" -snap before-leaving - -# Leaving means the window dies. The store does not. -ux_kill "$UX_SESSION" -sleep 5 - -assert_cmd 'the journal survived the terminal closing' test -f "$UX_DB" -since="$(mark)" - -ux_launch "$UX_SESSION" -sleep 6 -snap returned - -# alive_re: v3 signs its composer prompt (see lib.sh). -assert_screen "$(alive_re)" 'codeaf came back up' 60 - -# The one thing that must be true on return: the conversation is still there. -assert_screen 'heron|haiku|river|stretch|water|learned' \ - 'the prior conversation is on screen — the thread resumed, it did not restart' 45 - -assert_journal_is "select count(*) from messages where seq <= $since" "$before_messages" \ - 'no prior message was dropped by the restart' - -session_now="$(journal "select session_id from messages where session_id != '' order by seq desc limit 1")" -sessions="$(journal "select count(distinct session_id) from messages where session_id != ''")" -record 'session id in use' "$session_now" -record 'distinct sessions in the journal' "$sessions" - -# Resumption is either the same session id continuing, or a new one attached -# with a brief. Both are the journey; the report says which happened. -say "are you still there?" -sleep 10 -after_session="$(journal "select session_id from messages where role='user' order by seq desc limit 1")" -record 'session after returning' "$after_session" - -if [ "$after_session" = "$session_now" ]; then - _check yes 'the same session resumed' 'session id unchanged across the restart' "$after_session" -else - brief="$(journal "select count(*) from messages where seq > $since and body like 'While you were away%'")" - if [ "${brief:-0}" != "0" ]; then - _check yes 'a new session attached with an arrival brief' 'a "While you were away" line' "$brief brief(s)" - else - _check no 'the return either resumes the session or brings a brief' \ - 'same session id, or a "While you were away" brief' "new session $after_session with no brief" - fi -fi - -assert_journal "select count(*) from messages where seq > $since and role='agent'" \ - 'the returned window answers — the brain came back, not just the frame' 120 -dump_turn "$since" -finish diff --git a/test/ux/journeys/13-second-window.sh b/test/ux/journeys/13-second-window.sh deleted file mode 100755 index 8961d18763..0000000000 --- a/test/ux/journeys/13-second-window.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# J13 · One brain -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J13 One brain — a second window on a conversation this machine already has open is told where the conversation lives, and the first window never blinks' - -expect_nodes 0 - -# One brain means one composer. The watcher machinery (watching.go) exists for -# conversations on FAR machines; a conversation open in a second window on this -# one is refused at the journal with the session-busy word. That refusal is the -# product keeping its law — two windows must not race one composer — and this -# journey holds it open. -snap a-before -assert_screen "$(alive_re)" 'window A is alive' 30 - -# B opens onto the same brain. It must not become a second front door. -ux_launch "$UX_SESSION_B" -snap b-opened "$UX_SESSION_B" - -if wait_screen 'open in another window' 30 "$UX_SESSION_B"; then - _check yes 'the second window is refused with the session-busy word' \ - '"open in another window — go there, or start a new conversation here"' \ - "$(pane "$UX_SESSION_B" | grep -i 'open in another window' | head -1 | sed 's/^ *//')" -else - _check no 'the second window is refused with the session-busy word' \ - '"open in another window — go there, or start a new conversation here"' \ - "$(pane "$UX_SESSION_B" | sed -n '1p;$p' | tr '\n' ' ')" -fi - -# And A never blinked: still drawing its own frame. -if pane "$UX_SESSION" | grep -q 'CODEAF-EXITED'; then - _check no 'the first window survived the second opening' 'A still running' 'A exited' -else - _check yes 'the first window survived the second opening' 'A still running' 'A alive' -fi - -# The one brain answers from the window that owns it. -since="$(mark)" -say "what is 7 times 6?" "$UX_SESSION" -if wait_screen '42' 120 "$UX_SESSION"; then - _check yes 'the first window still answers — one brain, undisturbed' '42 on A' 'found' -else - _check no 'the first window still answers — one brain, undisturbed' '42 on A' \ - "$(pane "$UX_SESSION" | tail -12 | tr '\n' ' ')" -fi -snap a-answered "$UX_SESSION" - -ux_kill "$UX_SESSION_B" -sleep 3 - -dump_turn "$since" -finish diff --git a/test/ux/journeys/14-craft.sh b/test/ux/journeys/14-craft.sh deleted file mode 100755 index 44ddee89fb..0000000000 --- a/test/ux/journeys/14-craft.sh +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env bash -# J14 · Repeat a workflow — EXPLORATORY -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J14 Repeat a workflow — craft learned from the first run compiles into the second' - -expect_nodes 2 - -# This one is reported, not graded. Craft is forged by the distiller only when -# the model judges a job SHAPE reusable, and the prompt that asks it says -# outright that "most jobs have none" (cmd/codeaf/chat.go, the distiller's -# craft clause). Two toy jobs are exactly the case it is told to decline. The -# thresholds on the other side — recognition — are constants: -# CraftDecisiveScore = 1.5 × craft.MatchFloor (use a known-good craft) -# CraftOverwhelmingScore = 4 × craft.MatchFloor (use an unproven draft) -# in internal/resident/craftmind.go. So: run the two jobs, look, say what was -# actually there. -observed 'craft forging is a model judgment with an explicit "most jobs have none" instruction; two toy jobs are reported, not graded' - -since="$(mark)" -craft_dir="$UX_STATE/craft" - -before_commits="$(git -C "$craft_dir" rev-list --count HEAD 2>/dev/null || echo 0)" -record 'craft repo commits before' "$before_commits" - -say "make a file listing 3 prime numbers" -wait_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" 240 -first_done="$(mark)" -sleep 20 -snap first-job - -say "make a file listing 3 square numbers" -wait_journal "select count(*) from nodes where created_seq > $first_done and origin='user' and status='done'" 240 -sleep 30 -snap second-job - -after_commits="$(git -C "$craft_dir" rev-list --count HEAD 2>/dev/null || echo 0)" -record 'craft repo commits after' "$after_commits" -record 'craft repo log' "$(git -C "$craft_dir" log --oneline -10 2>/dev/null | tr '\n' ' ' || echo '')" -record 'craft files' "$(ls "$craft_dir" 2>/dev/null | tr '\n' ' ' || echo '')" -record 'nodes carrying a craft' "$(journal "select count(*) from nodes where craft != ''")" -record 'craft named on nodes' "$(journal "select group_concat(distinct craft) from nodes where craft != ''")" - -first_cost="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since and seq <= $first_done")" -second_cost="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $first_done")" -first_nodes="$(journal "select count(*) from nodes where created_seq > $since and created_seq <= $first_done")" -second_nodes="$(journal "select count(*) from nodes where created_seq > $first_done")" -record 'first job' "$first_nodes nodes, \$$first_cost" -record 'second job' "$second_nodes nodes, \$$second_cost" - -if [ "$after_commits" -gt "$before_commits" ]; then - record 'OBSERVED' "craft WAS forged: $((after_commits - before_commits)) new commit(s) in the craft repo" -else - record 'OBSERVED' 'no craft was forged from two toy jobs — consistent with the distiller being told most jobs have none' -fi - -dump_turn "$since" -finish diff --git a/test/ux/journeys/15-fanout.sh b/test/ux/journeys/15-fanout.sh deleted file mode 100755 index 17a465ce28..0000000000 --- a/test/ux/journeys/15-fanout.sh +++ /dev/null @@ -1,79 +0,0 @@ -#!/usr/bin/env bash -# J15-fanout · Parallelism lives in the graph, not in terminal tabs -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Fan-out — "say five things and five jobs run": one message asking for three independent things must become three jobs, not one queue' - -# docs/JOURNEY.md: "Parallelism lives in the graph, not in terminal tabs: say -# five things and five jobs run; the board shows them side by side." Three is -# enough to tell splitting from serialising, and cheap enough to say twice. -expect_nodes 3 - -since="$(mark)" -started="$(date +%s)" - -say "three separate things please: write a file with the first 5 prime numbers, write a file with the capitals of France Japan and Peru, and write a file with a two-line poem about rain" - -assert_journal "select count(*) from commands where seq > $since and kind='splice' and status='applied'" \ - 'the ask was journaled as work' 90 - -# The fan-out: three independent jobs, not one node doing three things. -wait_journal "select case when count(*) >= 3 then 1 else 0 end from nodes where created_seq > $since and origin='user'" 240 -jobs="$(journal "select count(*) from nodes where created_seq > $since and origin='user'")" -record 'jobs formed from one sentence' "$jobs" -record 'job titles' "$(journal "select group_concat(title, ' · ') from nodes where created_seq > $since and origin='user'")" - -if [ "${jobs:-0}" -ge 3 ]; then - _check yes 'one sentence naming three things became three jobs' 'three or more user jobs' "$jobs" -else - _check no 'one sentence naming three things became three jobs' 'three or more user jobs' \ - "$jobs — the fan-out did not split the ask" -fi - -wait_journal "select case when count(*) = 0 then 1 else 0 end from nodes where created_seq > $since and origin='user' and status not in ('done','failed','cancelled')" "$UX_JOB_TIMEOUT" -wall=$(( $(date +%s) - started )) -sleep 10 -snap delivered - -assert_journal_is "select count(*) from nodes where created_seq > $since and origin='user' and status != 'done'" 0 \ - 'every one of the jobs finished' - -# Concurrency factor: the work the graph did, divided by the time the person -# waited. At 1.0 the jobs were a queue with extra steps; above it, the graph is -# the parallelism the product promises instead of terminal tabs. -busy="$(journal " - select cast(coalesce(sum( - (julianday(finished_at) - julianday(started_at)) * 86400.0), 0) as int) - from nodes where created_seq > $since and origin='user' - and started_at is not null and finished_at is not null")" -record 'wall time the person waited' "${wall}s" -record 'summed job time the graph did' "${busy}s" -factor="$(python3 -c " -wall = max(1.0, float('${wall:-1}')) -print('%.2f' % (float('${busy:-0}') / wall))")" -record 'CONCURRENCY FACTOR (summed job seconds / wall seconds)' "$factor" -printf '%s\n' "$factor" > "$UX_DIR/concurrency" - -if python3 -c "import sys; sys.exit(0 if float('$factor') >= 1.3 else 1)"; then - _check yes 'the jobs actually ran side by side' 'concurrency factor at or above 1.3' "$factor" -else - _check no 'the jobs actually ran side by side' 'concurrency factor at or above 1.3' \ - "$factor — the graph held three jobs but ran them one after another" -fi - -# Each ask must be answered on its own terms, not merged into one blob. -answers="$(journal "select group_concat(lower(body), char(10)) from messages where seq > $since and role in ('agent','system')")" -# ERE, not BRE: grep -E reads \| as a literal pipe, and the earlier spelling -# searched for the six-character string "lima|peru" and never found it. -for want in 'prime' 'lima|peru|tokyo|paris' 'rain'; do - if printf '%s' "$answers" | grep -Eqi "$want"; then - _check yes "the thread answers the '$want' part of the ask" "a deliverable mentioning $want" 'found' - else - _check no "the thread answers the '$want' part of the ask" "a deliverable mentioning $want" 'absent' - fi -done - -judge_this 'three separate things: a file with the first 5 primes, a file with the capitals of France Japan and Peru, and a file with a two-line poem about rain' \ - "$(journal "select group_concat(body, char(10)) from messages where seq > $since and node_id != ''")" - -dump_turn "$since" -finish diff --git a/test/ux/journeys/16-model-words.sh b/test/ux/journeys/16-model-words.sh deleted file mode 100755 index 8529e44bde..0000000000 --- a/test/ux/journeys/16-model-words.sh +++ /dev/null @@ -1,57 +0,0 @@ -#!/usr/bin/env bash -# J16 · Choose models in words -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J16 Choose models in words — "boost this" routes the call and is recorded on usage' - -work_model="$(python3 -c "import json;print(json.load(open('$UX_STATE/settings.json')).get('task_model',''))")" -boost_model="$(python3 -c "import json;print(json.load(open('$UX_STATE/settings.json')).get('boost_model',''))")" -record 'work model' "$work_model" -record 'boost model' "$boost_model" - -expect_nodes 1 -since="$(mark)" -say "use the boost model for this: write one sentence about mountains" - -assert_screen 'mountain' 'the sentence came back' 150 -sleep 8 -snap answered - -models="$(journal "select group_concat(distinct model) from usage where seq > $since")" -record 'models on usage rows for this turn' "${models:-}" - -boost_bare="${boost_model#\~}" -if [ -n "$models" ] && printf '%s' "$models" | grep -qF "$boost_bare"; then - _check yes 'the words routed the call — usage names the boost model' \ - "a usage row on $boost_model" "$models" - record 'outcome' 'ROUTED ON USAGE — the boost model appears on the turn' -elif [ -n "$models" ] && [ "$models" != "$work_model" ] && [ "$models" != "${work_model#\~}" ]; then - _check yes 'the words routed the call — usage differs from the default work model' \ - "a usage model other than $work_model" "$models" - record 'outcome' 'ROUTED ON USAGE — a different model than the default' -else - cmd="$(journal "select group_concat(kind || ':' || substr(instruction,1,60), ' | ') from commands where seq > $since")" - record 'commands this turn' "${cmd:-}" - if printf '%s' "$cmd" | grep -qi 'boost'; then - _check yes 'the words are carried on the command even though usage kept the default model' \ - 'a command carrying boost' "$cmd" - record 'outcome' 'CARRIED ON COMMAND — the boost word is journaled but usage stayed on the work model' - else - _check no 'the model words reached either the usage row or the command' \ - "$boost_model on usage, or boost on a command" "usage=[${models:-none}] commands=[${cmd:-none}]" - record 'outcome' 'NOT ROUTED — the words changed nothing durable' - fi -fi - -# The glance that says which model is answering. v3 carries the model word on -# the status deck's chip (drawn from app.modelWord) and in the foot's seam. -# Reported, not graded, as before — the graded half of this journey is the -# durable routing above. -record 'foot row' "$(pane | tail -2 | tr '\n' ' ' | sed 's/ *$//')" -if [ -n "$boost_bare" ]; then - record 'model word on screen' "$(pane | grep -iF -- "$boost_bare" | head -1 | sed 's/^ *//')" -else - record 'model word on screen' '' -fi - -dump_turn "$since" -finish diff --git a/test/ux/journeys/17-attachments.sh b/test/ux/journeys/17-attachments.sh deleted file mode 100755 index 089156bd78..0000000000 --- a/test/ux/journeys/17-attachments.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/usr/bin/env bash -# J17 · Attach things (cheap variant) -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J17 Attach things — a mentioned file is copied at mention (CAS) and survives the source changing' - -# The composer captures path-shaped tokens with known media extensions and -# turns them into chips, so the cheap variant is a real (tiny) PNG. -img="$UX_RUN/dot.png" -python3 - "$img" <<'PY' -import base64, sys -# 1x1 PNG, the smallest honest image there is. -sys.stdout.buffer # noqa -open(sys.argv[1], "wb").write(base64.b64decode( - "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mP8z8BQDwAEhQGAhKmMIQAAAABJRU5ErkJggg==")) -PY -record 'file mentioned' "$img ($(wc -c < "$img" | tr -d ' ') bytes)" - -before_cas="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" -record 'CAS objects before' "$before_cas" - -expect_nodes 1 -since="$(mark)" -tmux send-keys -t "$UX_SESSION" -l -- "tell me in one word what colour this image is: $img" -sleep 2 -snap chip -press Enter -note "said: tell me in one word what colour this image is: $img" - -sleep 25 -snap answered - -# "Not the empty list" is not the same as "an attachment". A surface that never -# captured the mention writes the JSON literal `null` into the column, which is -# also != '[]' — and the check passed on nothing. The column has to hold a -# populated list for this to mean what it says. v1's value is unchanged by the -# tightening; it writes ["cas://…"]. -assert_journal "select count(*) from messages where seq > $since and role='user' - and attachments is not null and attachments not in ('[]','null','')" \ - 'the mention is journaled as an attachment on the user message' 60 -record 'attachments recorded' "$(journal "select attachments from messages where seq > $since and role='user' order by seq limit 1")" - -after_cas="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" -record 'CAS objects after' "$after_cas" -if [ "$after_cas" -gt "$before_cas" ]; then - _check yes 'the file was copied into the CAS at mention time' \ - 'a new content-addressed object' "$((after_cas - before_cas)) new object(s)" -else - _check no 'the file was copied into the CAS at mention time' \ - 'a new content-addressed object' "still $after_cas objects" -fi - -# Survives the source changing: the copy is content, not a pointer. -rm -f "$img" -still="$(find "$UX_STATE/cas" -type f 2>/dev/null | wc -l | tr -d ' ')" -[ "$still" = "$after_cas" ] \ - && _check yes 'the copy outlives its source' 'CAS unchanged after deleting the original' "$still objects" \ - || _check no 'the copy outlives its source' 'CAS unchanged after deleting the original' "$still objects" - -assert_journal "select count(*) from messages where seq > $since and role in ('agent','system')" \ - 'the mention got a reply, not silence' - -dump_turn "$since" -finish diff --git a/test/ux/journeys/18-take-out.sh b/test/ux/journeys/18-take-out.sh deleted file mode 100755 index 0ffc652432..0000000000 --- a/test/ux/journeys/18-take-out.sh +++ /dev/null @@ -1,52 +0,0 @@ -#!/usr/bin/env bash -# J18 · Take things out (hint-only variant) -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J18 Take things out — y/Y copy, /open, clickable paths: deliverables exit the terminal without friction' - -expect_nodes 0 - -# The clipboard itself is the machine's, not codeaf's — pbcopy on a headless CI -# box is a different journey. What is codeaf's, and what this asserts, is that -# the way out is discoverable and that the paths are real. -note 'variant: the clipboard is environment-dependent, so this asserts the affordance renders and the paths are real files' - -# The door to the help sheet: v3 answers a bare `?` from the composer -# (commands.go binds it as helpAskKey), the same door the old surfaces had. -press '?' -sleep 3 -snap help - -assert_screen 'help' 'the help overlay opened on ?' 20 -# Discoverability: the copy verb has to be readable somewhere the user can -# find it. The registry's row reads "copy answer / copy the focused answer to -# the clipboard y" (registry/catalog.go). -assert_screen 'copy .*answer' 'the y/Y copy hint is spelled out in help' 20 -record 'copy hint' "$(pane | grep -Ei 'copy .*answer' | head -1 | sed 's/^ *//')" -assert_screen '/open' 'the /open route out is named' 20 - -press Escape -sleep 2 -snap help-closed - -# The other half of "without friction": the paths the thread printed are real. -paths="$(journal "select body from messages where role in ('agent','system')" | grep -Eo "$UX_STATE/workspace/[^ )\"']*" | sort -u | head -5)" -record 'paths named in the thread' "$(printf '%s' "$paths" | tr '\n' ' ')" -missing=0 -while IFS= read -r p; do - [ -z "$p" ] && continue - [ -e "$p" ] || { missing=$((missing + 1)); note "MISSING: $p"; } -done <<< "$paths" - -if [ -n "$paths" ]; then - [ "$missing" -eq 0 ] \ - && _check yes 'every workspace path the thread printed is a real file' 'all paths exist' 'all exist' \ - || _check no 'every workspace path the thread printed is a real file' 'all paths exist' "$missing missing" -else - # Not a failure. An answer-first deliverable is allowed to hand back the - # content and no path at all — /open and the workspace link are the way out - # in that case, and grading their absence would punish the better behaviour. - record 'OBSERVED' 'this run delivered answer-first with no absolute path in the thread; the way out is /open and the workspace link, not a printed path' -fi - -dump_turn "$(mark)" -finish diff --git a/test/ux/journeys/19-self-pitch.sh b/test/ux/journeys/19-self-pitch.sh deleted file mode 100755 index 953b820789..0000000000 --- a/test/ux/journeys/19-self-pitch.sh +++ /dev/null @@ -1,39 +0,0 @@ -#!/usr/bin/env bash -# J19 · Ask what it can do -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'J19 Ask what it can do — the product carries its own pitch: journeys, in its own voice, from live capabilities' - -expect_nodes 0 -since="$(mark)" -say "what can you do?" -assert_journal "select count(*) from messages where seq > $since and role='agent'" 'the head replied' 150 -sleep 8 -snap answered - -reply="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" -record 'reply' "$(printf '%s' "$reply" | head -c 900)" - -length="$(printf '%s' "$reply" | wc -c | tr -d ' ')" -record 'reply length (bytes)' "$length" -[ "${length:-0}" -gt 150 ] \ - && _check yes 'the pitch has substance' 'more than a sentence' "$length bytes" \ - || _check no 'the pitch has substance' 'more than a sentence' "$length bytes" - -# It must describe the journeys, not itself in the abstract. -hits=0 -for word in task work file build remember learn charter remind cost watch run graph question; do - printf '%s' "$reply" | grep -q "$word" && hits=$((hits + 1)) -done -record 'capability words named' "$hits of 13" -[ "$hits" -ge 4 ] \ - && _check yes 'the answer names actual capabilities, not vibes' 'at least 4 capability words' "$hits" \ - || _check no 'the answer names actual capabilities, not vibes' 'at least 4 capability words' "$hits" - -if printf '%s' "$reply" | grep -Eq "i'?m an ai|language model|as an ai|i can'?t do|i have no"; then - _check no 'it pitches the product, not a disclaimer' 'no assistant boilerplate' "$(printf '%s' "$reply" | grep -Eo "i'?m an ai|language model|as an ai" | head -1)" -else - _check yes 'it pitches the product, not a disclaimer' 'no assistant boilerplate' 'none' -fi - -dump_turn "$since" -finish diff --git a/test/ux/judge.py b/test/ux/judge.py deleted file mode 100644 index 607eaf1a50..0000000000 --- a/test/ux/judge.py +++ /dev/null @@ -1,104 +0,0 @@ -#!/usr/bin/env python3 -"""judge.py — one model call that grades one deliverable. - -A journey can prove that a haiku was journaled, spliced, and shown. It cannot -prove the haiku was any good, and "it ran" is not the product's promise. So the -deliverables get read by a cheap model against a three-line rubric, and the -score rides in the report beside the pass. - - judge.py - -prints: | -""" - -import json -import os -import sys -import urllib.error -import urllib.request - -RUBRIC = """You are grading one deliverable from an AI agent. Be a hard marker. - -Grade on exactly three things: -1. Does the deliverable satisfy the LITERAL ask — every constraint the user - actually wrote (counts, subjects, named requirements), not the gist? -2. Is it answer-first: does the final message contain the CONTENT itself, - rather than only a pointer to a file the user would have to go and open? -3. Is it competently done for what it is? - -Reply with JSON only: {"score": <1-5 integer>, "why": ""} -5 = satisfies every literal constraint and leads with the content. -3 = broadly right but misses a stated constraint or buries the content. -1 = does not do what was asked, or hands back only a path.""" - - -def main() -> int: - state, ask_path, deliverable_path = sys.argv[1], sys.argv[2], sys.argv[3] - with open(os.path.join(state, "config.json")) as handle: - key = json.load(handle)["api_key"] - # The "~" prefix is OpenRouter's floating-alias marker and part of the slug, - # not decoration: stripping it turns a valid model id into a 400. The - # fallbacks exist because the judge failing must never fail the journey. - try: - configured = json.load(open(os.path.join(state, "settings.json"))).get("chat_model") - except (FileNotFoundError, ValueError): - configured = None - candidates = [m for m in (configured, "deepseek/deepseek-v4-flash", - "google/gemini-3.6-flash") if m] - - ask = open(ask_path).read().strip()[:2000] - deliverable = open(deliverable_path).read().strip()[:6000] - if not deliverable: - print("1|nothing was delivered to grade") - return 0 - - text, failure = None, "no model answered" - for model in candidates: - payload = { - "model": model, - "messages": [ - {"role": "system", "content": RUBRIC}, - {"role": "user", "content": "THE ASK:\n%s\n\nWHAT CAME BACK:\n%s" % (ask, deliverable)}, - ], - "temperature": 0, - # Generous, because a reasoning-capable model spends its first - # tokens thinking and returns a null content when the cap lands - # mid-thought — which reads as "the judge said nothing" rather than - # as the truncation it is. - "max_tokens": 900, - } - request = urllib.request.Request( - "https://openrouter.ai/api/v1/chat/completions", - data=json.dumps(payload).encode(), - headers={"Authorization": "Bearer " + key, "Content-Type": "application/json"}, - ) - try: - with urllib.request.urlopen(request, timeout=90) as response: - body = json.load(response) - message = body["choices"][0]["message"] - text = message.get("content") or message.get("reasoning") - if text: - break - failure = "%s returned no content (%s)" % ( - model, body["choices"][0].get("finish_reason")) - except (urllib.error.URLError, KeyError, ValueError, TypeError) as exc: - failure = "%s on %s" % (exc, model) - if text is None: - print("?|the judge call failed: %s" % failure) - return 0 - - start, end = text.find("{"), text.rfind("}") - if start >= 0 and end > start: - try: - verdict = json.loads(text[start:end + 1]) - why = str(verdict.get("why", "")).replace("|", "/").replace("\n", " ") - print("%s|%s" % (verdict.get("score", "?"), why[:200])) - return 0 - except ValueError: - pass - print("?|the judge did not answer in the rubric's shape: %s" % text[:120].replace("\n", " ")) - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/test/ux/lib.sh b/test/ux/lib.sh deleted file mode 100644 index 125807337c..0000000000 --- a/test/ux/lib.sh +++ /dev/null @@ -1,336 +0,0 @@ -#!/usr/bin/env bash -# lib.sh — the vocabulary a journey script is written in. -# -# A journey is a person at a terminal. It says things, waits for the screen to -# say something back, and then asks the journal whether anything durable -# actually happened. Those three verbs — say, wait_screen, journal — are the -# whole library; everything else is bookkeeping so the report can print WHAT -# was expected beside WHAT was there. -# -# Sourced by every journeys/*.sh under an environment run.sh exports: -# UX_BIN UX_HOME UX_STATE UX_DB UX_SESSION UX_EVIDENCE UX_CAP UX_JOURNEY - -set -uo pipefail - -: "${UX_REPLY_TIMEOUT:=120}" # a conversational reply -: "${UX_JOB_TIMEOUT:=300}" # a commissioned job -: "${UX_POLL:=2}" - -# ---------------------------------------------------------------- the frame -# -# alive_re matches "the surface drew a frame of its own". v3 always draws the -# composer prompt "›" — it is on screen before anything has been said, and it -# is the one glyph a dead pane (the CODEAF-EXITED marker, a bare shell) never -# draws. Match on it whenever a journey asks "is this window alive at all". -alive_re() { printf '%s' '›'; } - -UX_FAILURES=0 -UX_CHECKS=0 -UX_FORCED="" -UX_DIR="$UX_EVIDENCE/$UX_JOURNEY" -mkdir -p "$UX_DIR" -: > "$UX_DIR/notes.md" - -# ---------------------------------------------------------------- narration - -note() { printf '%s\n' "$*" >> "$UX_DIR/notes.md"; printf ' %s\n' "$*" >&2; } -head_line() { printf '\n## %s\n\n' "$*" >> "$UX_DIR/notes.md"; printf ' %s\n' "$*" >&2; } - -# journey_is prints the journey's own one-line description, so a reader of the -# transcript never has to go and look up what number 4 was. -journey_is() { - printf '# %s — %s\n\n' "$UX_JOURNEY" "$*" >> "$UX_DIR/notes.md" - printf '\n\033[1m▶ %s — %s\033[0m\n' "$UX_JOURNEY" "$*" >&2 -} - -record() { printf -- '- **%s**: %s\n' "$1" "$2" >> "$UX_DIR/notes.md"; printf -- ' · %s: %s\n' "$1" "$2" >&2; } - -# ---------------------------------------------------------------- the mouth - -# say types a sentence and presses Enter, with enough pacing that the composer -# has drawn the draft before the newline lands. Never start a sentence with -# "/" (the command palette eats Enter) or "?" (it opens the help overlay). - -# clear_composer empties the draft. A digit typed at a question that has since -# been answered stays in the composer and silently prefixes the next sentence -# ("2stand down the stretch reminder"), which would make the suite lie about -# what the user said. -# -# v3 binds ctrl+u as the readline kill — the input guard and the draft ring -# both test it by name — so the words go to the ring and the box is empty: -# a gesture with no second meaning while a reply streams. -clear_composer() { - local sess="${1:-$UX_SESSION}" - tmux send-keys -t "$sess" C-u - sleep 0.3 -} - -say() { - local text="$1" sess="${2:-$UX_SESSION}" - case "$text" in - /*) note "REFUSED to say a line starting with / — the palette would eat it"; return 1 ;; - \?*) note "REFUSED to say a line starting with ? — it opens help"; return 1 ;; - esac - clear_composer "$sess" - tmux send-keys -t "$sess" -l -- "$text" - sleep 0.6 - tmux send-keys -t "$sess" Enter - note "said: $text" -} - -# press sends a named key (Enter, Up, C-c …). -press() { tmux send-keys -t "${2:-$UX_SESSION}" "$1"; note "pressed: $1"; } - -# type_raw sends characters with no Enter. Answering an option question is a -# bare digit that submits itself, so it must NOT be followed by a newline. -type_raw() { tmux send-keys -t "${2:-$UX_SESSION}" -l -- "$1"; note "typed: $1"; } - -# answer_number answers a numbered question. -# -# v3 binds the digits a question draws (questionset.go's questionGroupDigit, -# which maps a digit key to the row it names): a bare digit picks its row and -# submits itself, so it must NOT be followed by Enter. Every journey asserts -# the OUTCOME — the question is recorded answered, by the user's own message — -# never the keystroke. -answer_number() { - local number="$1" sess="${2:-$UX_SESSION}" - type_raw "$number" "$sess" - sleep 3 -} - -# ---------------------------------------------------------------- the eyes - -pane() { tmux capture-pane -t "${1:-$UX_SESSION}" -p 2>/dev/null; } - -snap() { - local name="$1" sess="${2:-$UX_SESSION}" - pane "$sess" > "$UX_DIR/$name.snap.txt" - note "snap: $name.snap.txt" -} - -# wait_screen polls the pane until the regex matches. Match on SHORT fragments: -# every line is wrapped and hard-clamped to the pane width, so a long literal -# can straddle two rows and never appear whole. -wait_screen() { - local re="$1" limit="${2:-$UX_REPLY_TIMEOUT}" sess="${3:-$UX_SESSION}" waited=0 - while [ "$waited" -lt "$limit" ]; do - if pane "$sess" | grep -Eqi -- "$re"; then return 0; fi - sleep "$UX_POLL"; waited=$((waited + UX_POLL)) - done - return 1 -} - -# ---------------------------------------------------------------- the journal - -journal() { - sqlite3 -readonly "$UX_DB" "$1" 2>/dev/null -} - -journal_table() { sqlite3 -readonly -header -column "$UX_DB" "$1" 2>/dev/null; } - -# mark is the journal watermark before a turn. Every assertion afterwards is -# scoped to "since the mark", so journey 5 can never pass on journey 2's rows. -mark() { journal "select coalesce(max(seq),0) from events"; } - -spend() { journal "select coalesce(round(sum(cost),6),0) from usage"; } - -# wait_journal polls a SQL count until it is greater than zero. -wait_journal() { - local sql="$1" limit="${2:-$UX_JOB_TIMEOUT}" waited=0 value - while [ "$waited" -lt "$limit" ]; do - value="$(journal "$sql")" - if [ -n "$value" ] && [ "$value" != "0" ]; then return 0; fi - sleep "$UX_POLL"; waited=$((waited + UX_POLL)) - done - return 1 -} - -# ---------------------------------------------------------------- assertions - -_check() { - local ok="$1" what="$2" expected="$3" actual="$4" - UX_CHECKS=$((UX_CHECKS + 1)) - if [ "$ok" = "yes" ]; then - printf -- '- ✅ %s\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >> "$UX_DIR/notes.md" - printf ' \033[32m✅ %s\033[0m\n' "$what" >&2 - else - UX_FAILURES=$((UX_FAILURES + 1)) - printf -- '- ❌ %s\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >> "$UX_DIR/notes.md" - printf ' \033[31m❌ %s\033[0m\n expected: %s\n found: %s\n' "$what" "$expected" "$actual" >&2 - snap "fail-$UX_CHECKS" - fi -} - -# assert_screen waits for a pattern and reports the last screen when it never -# arrives, because "the screen never said it" is only useful beside the screen. -assert_screen() { - local re="$1" what="$2" limit="${3:-$UX_REPLY_TIMEOUT}" - if wait_screen "$re" "$limit"; then - _check yes "$what" "screen matching /$re/" "$(pane | grep -Eio -- "$re" | head -1)" - else - _check no "$what" "screen matching /$re/ within ${limit}s" "timed out; last screen in fail snap" - fi -} - -refute_screen() { - local re="$1" what="$2" - if pane | grep -Eqi -- "$re"; then - _check no "$what" "no screen line matching /$re/" "$(pane | grep -Eio -- "$re" | head -1)" - else - _check yes "$what" "no screen line matching /$re/" "none" - fi -} - -# assert_journal passes when the query returns something that is neither empty -# nor zero. The query IS the expectation, so it prints verbatim on failure. -assert_journal() { - local sql="$1" what="$2" limit="${3:-0}" value - if [ "$limit" -gt 0 ]; then wait_journal "$sql" "$limit" >/dev/null; fi - value="$(journal "$sql")" - if [ -n "$value" ] && [ "$value" != "0" ]; then - _check yes "$what" "$sql → non-zero" "$value" - else - _check no "$what" "$sql → non-zero" "${value:-}" - fi -} - -assert_journal_is() { - local sql="$1" want="$2" what="$3" value - value="$(journal "$sql")" - if [ "$value" = "$want" ]; then - _check yes "$what" "$sql = $want" "$value" - else - _check no "$what" "$sql = $want" "${value:-}" - fi -} - -assert_cmd() { - local what="$1"; shift - if "$@" >/dev/null 2>&1; then - _check yes "$what" "$* succeeds" "exit 0" - else - _check no "$what" "$* succeeds" "exit $?" - fi -} - -# ---------------------------------------------------------------- gauges -# -# Completion is the cheap half. A journey that passes every assertion can still -# have spliced eleven tasks to write a haiku, spent a dollar doing it, taken -# four minutes, filled the thread with chatter, and produced something bad. -# These four leave a trace the runner turns into the report card. - -# expect_nodes declares the proportionate size of this ask. The runner flags a -# journey that formed materially more work than the ask deserved. -expect_nodes() { printf '%s\n' "$1" > "$UX_DIR/expect-nodes"; } - -# judge_this hands one deliverable to a cheap model against a fixed rubric. -# The suite can prove a haiku was journaled; only a reader can say it is a -# haiku about rivers that mentions a heron. -judge_this() { - printf '%s\n' "$1" > "$UX_DIR/judge-ask" - printf '%s\n' "$2" > "$UX_DIR/judge-deliverable" - note "queued for the quality judge (${#2} bytes of deliverable)" -} - -# deliverable_since is the usual thing to judge: everything the agent said -# after the ask, which is where an answer-first product puts the content. -deliverable_since() { - journal "select group_concat(body, char(10)) from messages where seq > $1 and role in ('agent','system')" -} - -# ---------------------------------------------------------------- verdicts - -# observed marks a journey whose mechanism has a threshold we cannot force -# cheaply. It is not a pass and not a failure: it is evidence, reported. -observed() { UX_FORCED="OBSERVED"; note "OBSERVED: $*"; printf '%s\n' "$*" > "$UX_DIR/verdict-note"; } -skipped() { UX_FORCED="SKIPPED"; note "SKIPPED: $*"; printf '%s\n' "$*" > "$UX_DIR/verdict-note"; exit 0; } - -# ---------------------------------------------------------------- windows - -# ux_launch opens another window onto the same brain, from the same UX_ENV the -# runner built, so a relaunch is a relaunch and not a different program. -ux_launch() { - local sess="$1"; shift - local extra="${*:-}" - tmux kill-session -t "$sess" 2>/dev/null - tmux new-session -d -s "$sess" -x "$UX_WIDTH" -y "$UX_HEIGHT" \ - "env $UX_ENV $extra '$UX_BIN' chat; echo CODEAF-EXITED; sleep 900" - sleep 8 - note "launched window $sess" -} - -ux_kill() { tmux kill-session -t "$1" 2>/dev/null; note "killed window $1"; } - -# ---------------------------------------------------------------- hygiene - -# dismiss_standing_watch answers the "should I keep watching when you're not -# here" offer with NO. Saying yes installs a launchd agent on the real machine, -# and a test suite does not get to leave a daemon behind. -dismiss_standing_watch() { - local open - local waited=0 - while [ "$waited" -lt 20 ]; do - open="$(journal "select count(*) from agent_questions where status in ('pending','asked') and lower(text) like '%keep watching%'")" - [ "${open:-0}" != "0" ] && break - sleep 2; waited=$((waited + 2)) - done - if [ "${open:-0}" != "0" ]; then - note "standing-watch offer is open — declining (a suite must not install a daemon)" - answer_number 2 - sleep 5 - else - note "no standing-watch offer open; nothing to decline" - fi - clear_composer -} - -# settle waits for the resident to go quiet: no journal growth for a few polls. -settle() { - local limit="${1:-30}" last="" now waited=0 stable=0 - while [ "$waited" -lt "$limit" ]; do - now="$(mark)" - if [ "$now" = "$last" ]; then - stable=$((stable + 1)) - [ "$stable" -ge 3 ] && return 0 - else - stable=0 - fi - last="$now"; sleep 2; waited=$((waited + 2)) - done - return 0 -} - -dump_turn() { - local since="$1" - { - echo; echo '### messages since the mark' - journal_table "select seq,role,node_id,command_seq,substr(replace(body,char(10),' / '),1,220) as body from messages where seq > $since order by seq" - echo; echo '### commands since the mark' - journal_table "select seq,kind,status,target,substr(instruction,1,70) as instruction,substr(result,1,70) as result from commands where seq > $since order by seq" - echo; echo '### nodes since the mark' - journal_table "select id,parent_id,status,origin,grp,substr(title,1,50) as title from nodes where created_seq > $since order by created_seq" - echo; echo '### usage since the mark' - journal_table "select model,count(*) as calls,round(sum(cost),5) as cost from usage where seq > $since group by model" - } >> "$UX_DIR/notes.md" -} - -# finish is the last line of every journey. -finish() { - local verdict - if [ -n "$UX_FORCED" ]; then - verdict="$UX_FORCED" - elif [ "$UX_CHECKS" -eq 0 ]; then - verdict="OBSERVED" - elif [ "$UX_FAILURES" -eq 0 ]; then - verdict="PASS" - else - verdict="FAIL" - fi - printf '%s\n' "$verdict" > "$UX_DIR/verdict" - printf '%s/%s checks\n' "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" > "$UX_DIR/checks" - printf '\n**verdict: %s** (%d/%d checks)\n' "$verdict" "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" >> "$UX_DIR/notes.md" - printf ' \033[1m%s\033[0m — %d/%d checks\n' "$verdict" "$((UX_CHECKS - UX_FAILURES))" "$UX_CHECKS" >&2 - [ "$verdict" = "FAIL" ] && exit 1 - exit 0 -} diff --git a/test/ux/quality/20-coding.sh b/test/ux/quality/20-coding.sh deleted file mode 100755 index 819bd6c35f..0000000000 --- a/test/ux/quality/20-coding.sh +++ /dev/null @@ -1,115 +0,0 @@ -#!/usr/bin/env bash -# Q1 · Does it actually solve a coding task? -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q1 Coding quality — a real programming task, graded by running the code the suite is given, not by reading the claim' - -# The whole point: codeaf saying "done, tests pass" is a claim. This journey -# takes the files it produced, runs them here, and grades the claim against -# what the interpreter says. - -expect_nodes 2 -since="$(mark)" -started="$(date +%s)" -touch "$UX_DIR/.stamp" - -say "write a python file called stats.py with a function median(numbers) that returns the median of a list of numbers and raises ValueError on an empty list. write pytest tests for it in test_stats.py. run the tests and tell me the result." - -assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ - 'the coding job finished' 420 -elapsed=$(( $(date +%s) - started )) -sleep 12 -snap delivered - -record 'wall time' "${elapsed}s" -record 'cost' "\$$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since")" -record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since")" -record 'node shape' "$(journal "select group_concat(id || '(' || status || ')' || case when parent_id='root' then '' else ' under ' || parent_id end, ' · ') from nodes where created_seq > $since")" - -# ------------------------------------------------------------ find the files - -work="$(find "$UX_STATE/workspace" -name 'stats.py' -newer "$UX_DIR/.stamp" 2>/dev/null | head -1)" -if [ -z "$work" ]; then - work="$(find "$UX_STATE/workspace" -name 'stats.py' 2>/dev/null | head -1)" -fi -if [ -n "$work" ]; then - dir="$(dirname "$work")" - _check yes 'stats.py exists in the workspace' 'a file the user can pick up' "$work" -else - dir="" - _check no 'stats.py exists in the workspace' 'a file the user can pick up' 'not found anywhere under the workspace' -fi -record 'workspace' "${dir:-}" -[ -n "$dir" ] && record 'files produced' "$(ls "$dir" | tr '\n' ' ')" -[ -n "$dir" ] && cp -R "$dir" "$UX_DIR/workspace" 2>/dev/null - -if [ -n "$dir" ] && [ -f "$dir/test_stats.py" ]; then - _check yes 'it wrote the tests it was asked for' 'test_stats.py' 'present' -else - _check no 'it wrote the tests it was asked for' 'test_stats.py' 'missing' -fi - -# ------------------------------------------------- grade by running the code - -if [ -n "$dir" ]; then - grade="$(cd "$dir" && python3 - <<'PY' 2>&1 -import importlib.util, json, sys -spec = importlib.util.spec_from_file_location("stats", "stats.py") -verdict = {"import": False, "odd": None, "even": None, "empty": None} -try: - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - verdict["import"] = True - median = getattr(module, "median") - verdict["odd"] = (median([3, 1, 2]) == 2) - verdict["even"] = (abs(median([1, 2, 3, 4]) - 2.5) < 1e-9) - try: - median([]) - verdict["empty"] = False - except ValueError: - verdict["empty"] = True - except Exception as exc: - verdict["empty"] = "wrong exception: %s" % type(exc).__name__ -except Exception as exc: - verdict["error"] = "%s: %s" % (type(exc).__name__, exc) -print(json.dumps(verdict)) -PY -)" - record 'independent grade' "$grade" - - for facet in odd even empty; do - if printf '%s' "$grade" | python3 -c "import json,sys; d=json.loads(sys.stdin.read()); sys.exit(0 if d.get('$facet') is True else 1)" 2>/dev/null; then - _check yes "median() is correct: $facet case" "the suite's own call returns the right answer" 'correct' - else - _check no "median() is correct: $facet case" "the suite's own call returns the right answer" "$grade" - fi - done - - # And the tests it wrote — do they pass when someone else runs them? - pytest_out="$(cd "$dir" && python3 -m pytest -q 2>&1 | tail -5)" - record 'pytest, run by the suite' "$(printf '%s' "$pytest_out" | tr '\n' ' ')" - if printf '%s' "$pytest_out" | grep -Eq '[0-9]+ passed'; then - _check yes 'the tests it wrote pass when the suite runs them' 'pytest green' "$(printf '%s' "$pytest_out" | grep -Eo '[0-9]+ passed[^ ]*' | head -1)" - elif printf '%s' "$pytest_out" | grep -Eq 'No module named pytest'; then - record 'note' 'pytest is not installed here; the direct grade above is the authority' - else - _check no 'the tests it wrote pass when the suite runs them' 'pytest green' "$(printf '%s' "$pytest_out" | tr '\n' ' ')" - fi -fi - -# ------------------------------------------------- did it run what it built? - -claim="$(journal "select group_concat(lower(body),' ') from messages where seq > $since and role in ('agent','system')")" -record 'what it claimed' "$(printf '%s' "$claim" | head -c 700)" - -if printf '%s' "$claim" | grep -Eq '[0-9]+ passed|tests? pass|all pass|passed in|=+ .*passed'; then - _check yes 'it ran the tests and reported real output, not a promise' \ - 'test output in the deliverable' "$(printf '%s' "$claim" | grep -Eo '[0-9]+ passed[a-z ,]*' | head -1)" -else - _check no 'it ran the tests and reported real output, not a promise' \ - 'test output in the deliverable' "$(printf '%s' "$claim" | head -c 200)" -fi - -judge_this 'write stats.py with median(numbers) raising ValueError on empty, write pytest tests in test_stats.py, run the tests and tell me the result' "$(deliverable_since "$since")" - -dump_turn "$since" -finish diff --git a/test/ux/quality/21-research.sh b/test/ux/quality/21-research.sh deleted file mode 100755 index 98294c043a..0000000000 --- a/test/ux/quality/21-research.sh +++ /dev/null @@ -1,69 +0,0 @@ -#!/usr/bin/env bash -# Q2 · Does it actually solve a research task? -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q2 Research quality — a finance question with checkable answers, graded against the real filed numbers' - -# Apple's FY2024 (ended 28 Sep 2024) 10-K: total net sales $391.035B, of which -# Services $96.169B and Products $294.866B. Those are the ground truth this -# journey grades against — a number in the right place is worth more than a -# confident paragraph, and a wrong number is worth less than nothing. - -expect_nodes 1 -since="$(mark)" -started="$(date +%s)" - -say "research Apple's total revenue in fiscal year 2024 and how much of it was Services. give me the two numbers and cite where they come from." - -assert_journal "select count(*) from messages where seq > $since and role in ('agent','system')" 'it replied' 240 -wait_journal "select count(*) from messages where seq > $since and role in ('agent','system') and length(body) > 120" 300 -sleep 15 -snap answered - -elapsed=$(( $(date +%s) - started )) -record 'wall time' "${elapsed}s" -record 'cost' "\$$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $since")" -record 'nodes formed' "$(journal "select count(*) from nodes where created_seq > $since")" - -answer="$(journal "select group_concat(body,char(10)) from messages where seq > $since and role in ('agent','system')")" -printf '%s\n' "$answer" > "$UX_DIR/answer.txt" -record 'answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 1200)" - -grade="$(printf '%s' "$answer" | python3 -c " -import re, sys -text = sys.stdin.read() -numbers = [float(n.replace(',', '')) for n in re.findall(r'[0-9][0-9,]*\.?[0-9]*', text)] -def near(target, tolerance): - return any(abs(n - target) <= tolerance for n in numbers) -total = near(391.0, 2.0) or near(391035.0, 2000.0) or near(391035000000.0, 2e9) -services = near(96.0, 1.5) or near(96169.0, 1500.0) or near(96169000000.0, 1.5e9) -print('total=%s services=%s' % (total, services)) -")" -record 'number grade' "$grade" - -printf '%s' "$grade" | grep -q 'total=True' \ - && _check yes 'total FY2024 revenue is right (\$391.0B)' 'a number within \$2B of 391.0' 'found' \ - || _check no 'total FY2024 revenue is right (\$391.0B)' 'a number within \$2B of 391.0' "$(printf '%s' "$answer" | grep -Eo '\$?[0-9][0-9,.]*( ?(billion|B|million))?' | head -8 | tr '\n' ' ')" - -printf '%s' "$grade" | grep -q 'services=True' \ - && _check yes 'Services revenue is right (\$96.2B)' 'a number within \$1.5B of 96.2' 'found' \ - || _check no 'Services revenue is right (\$96.2B)' 'a number within \$1.5B of 96.2' "$(printf '%s' "$answer" | grep -Eo '\$?[0-9][0-9,.]*( ?(billion|B))?' | head -8 | tr '\n' ' ')" - -# A citation is a document a person could go and check. "Form 8-K, filed -# 31 October 2024" is one; so is a URL. The earlier spelling accepted only the -# URL and failed a correctly cited answer. -if printf '%s' "$answer" | grep -Eqi 'http|sec\.gov|10-k|8-k|form 10|annual report|investor\.apple|press release|earnings release|filing'; then - _check yes 'the claim carries evidence, not confidence' 'a named source' "$(printf '%s' "$answer" | grep -Eoi 'https?://[^ )]*|sec\.gov[^ )]*|10-K' | head -3 | tr '\n' ' ')" -else - _check no 'the claim carries evidence, not confidence' 'a named source' 'no URL, filing, or report named' -fi - -if printf '%s' "$answer" | grep -Eqi 'https?://'; then - record 'looked it up' 'yes — a live URL appears in the answer' -else - record 'looked it up' 'no live URL in the answer; the numbers may be from model memory rather than the web tool' -fi - -judge_this "research Apple's total revenue in fiscal 2024 and how much was Services; give the two numbers and cite where they come from" "$(deliverable_since "$since")" - -dump_turn "$since" -finish diff --git a/test/ux/quality/22-quicker.sh b/test/ux/quality/22-quicker.sh deleted file mode 100755 index 14861b99f9..0000000000 --- a/test/ux/quality/22-quicker.sh +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env bash -# Q3 · Does asking for it quicker actually make it quicker? -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q3 Urgency — "quick, do not overthink" against a matched control: does the shape of the work change?' - -expect_nodes 2 - -run_one() { - local label="$1" sentence="$2" - local from t0 t1 - from="$(mark)" - t0="$(date +%s)" - say "$sentence" - wait_journal "select count(*) from messages where seq > $from and role in ('agent','system') and length(body) > 60" 300 - wait_journal "select count(*) from nodes where created_seq > $from and origin='user' and status in ('done','failed')" 60 >/dev/null - t1="$(date +%s)" - sleep 10 - QN="$(journal "select count(*) from nodes where created_seq > $from")" - QC="$(journal "select round(coalesce(sum(cost),0),5) from usage where seq > $from")" - QCALLS="$(journal "select count(*) from usage where seq > $from")" - QT=$((t1 - t0)) - record "$label" "$QN nodes · $QCALLS model calls · \$$QC · ${QT}s" - printf '%s %s %s %s' "$QN" "$QCALLS" "$QC" "$QT" -} - -since="$(mark)" - -control="$(run_one 'control (no urgency words)' 'write a short explanation of what a binary search tree is into a file')" -snap control -urgent="$(run_one 'urgent ("quick, one pass")' 'quick, one pass, do not overthink this: write a short explanation of what a hash table is into a file')" -snap urgent - -read -r cn ccalls ccost ctime <<< "$control" -read -r un ucalls ucost utime <<< "$urgent" - -record 'nodes control vs urgent' "$cn vs $un" -record 'calls control vs urgent' "$ccalls vs $ucalls" -record 'cost control vs urgent' "\$$ccost vs \$$ucost" -record 'seconds control vs urgent' "${ctime}s vs ${utime}s" - -# The honest bar: urgency must not make the work BIGGER. Whether it makes it -# genuinely smaller on one paired sample is reported, not asserted — one pair -# is an anecdote, and the suite refuses to dress an anecdote as a measurement. -if [ "${un:-99}" -le "${cn:-0}" ]; then - _check yes 'urgency words did not inflate the work' "urgent nodes ≤ control nodes" "$un ≤ $cn" -else - _check no 'urgency words did not inflate the work' "urgent nodes ≤ control nodes" "$un > $cn" -fi - -cheaper="$(python3 -c "print('yes' if float('$ucost') <= float('$ccost') else 'no')")" -faster="$([ "${utime:-99999}" -le "${ctime:-0}" ] && echo yes || echo no)" -record 'OBSERVED — cheaper when asked to hurry?' "$cheaper" -record 'OBSERVED — faster when asked to hurry?' "$faster" - -urgency_cmd="$(journal "select group_concat(kind || '/' || status, ', ') from commands where seq > $since")" -record 'commands across both jobs' "${urgency_cmd:-}" - -dump_turn "$since" -finish diff --git a/test/ux/quality/23-learning-applied.sh b/test/ux/quality/23-learning-applied.sh deleted file mode 100755 index 9e00b1b3b4..0000000000 --- a/test/ux/quality/23-learning-applied.sh +++ /dev/null @@ -1,60 +0,0 @@ -#!/usr/bin/env bash -# Q4 · Does what it learned change what it does? -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q4 Learning applied — the lesson taught in J7 ("run what you build before saying it is done") must show up as behaviour, not as a stored string' - -lesson="$(journal "select group_concat(substr(body,1,120), ' | ') from facts where status='active' and (lower(body) like '%run%' or lower(body) like '%verif%')")" -record 'the lesson on the shelf' "${lesson:-}" -[ -n "$lesson" ] \ - && _check yes 'the lesson is still in the notebook when the next job starts' 'an active fact about running/verifying' "$(printf '%s' "$lesson" | head -c 160)" \ - || _check no 'the lesson is still in the notebook when the next job starts' 'an active fact about running/verifying' 'nothing' - -expect_nodes 1 -since="$(mark)" -# Deliberately NOT asking it to run anything. If the lesson landed, it runs the -# script anyway and shows the output. If it only stored a sentence, it will -# hand back a file and call it done. -say "write a python script called fizz.py that prints fizzbuzz for 1 to 15" - -assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ - 'the job finished' 360 -sleep 12 -snap delivered - -claim="$(journal "select group_concat(body,char(10)) from messages where seq > $since and role in ('agent','system')")" -printf '%s\n' "$claim" > "$UX_DIR/claim.txt" -record 'what it said' "$(printf '%s' "$claim" | tr '\n' ' ' | head -c 800)" - -# Evidence of having actually run it: real fizzbuzz output in the deliverable. -if printf '%s' "$claim" | grep -qi 'fizzbuzz' && printf '%s' "$claim" | grep -qi 'buzz' && printf '%s' "$claim" | grep -q '\b11\b'; then - _check yes 'it ran what it built and showed the output — the lesson changed behaviour' \ - 'real program output in the deliverable' 'fizzbuzz output present' - record 'LEARNING' 'APPLIED — the taught lesson visibly changed how the next job was reported' -else - _check no 'it ran what it built and showed the output — the lesson changed behaviour' \ - 'real program output in the deliverable' "$(printf '%s' "$claim" | tr '\n' ' ' | head -c 200)" - record 'LEARNING' 'NOT APPLIED — the lesson is stored but the next job did not honour it' -fi - -# And the file itself has to be right. -script="$(find "$UX_STATE/workspace" -name 'fizz*.py' 2>/dev/null | head -1)" -record 'script' "${script:-}" -if [ -n "$script" ]; then - out="$(python3 "$script" 2>&1 | tr '\n' ' ')" - record 'suite ran it' "$(printf '%s' "$out" | head -c 200)" - # Case is not the contract — the sequence is. 3 and 15 must be fizz-ish, - # 5 must be buzz-ish, 15 must be the combined word. - if printf '%s' "$out" | grep -qi 'fizzbuzz' && printf '%s' "$out" | grep -qi 'fizz' \ - && printf '%s' "$out" | grep -qi 'buzz' && printf '%s' "$out" | grep -q '11'; then - _check yes 'the script is correct when the suite runs it' 'fizzbuzz through 15' "$(printf '%s' "$out" | head -c 120)" - else - _check no 'the script is correct when the suite runs it' 'fizzbuzz through 15' "$(printf '%s' "$out" | head -c 200)" - fi -else - _check no 'the script exists on disk' 'a fizz*.py in the workspace' 'not found' -fi - -judge_this 'write a python script called fizz.py that prints fizzbuzz for 1 to 15' "$(deliverable_since "$since")" - -dump_turn "$since" -finish diff --git a/test/ux/quality/24-shape.sh b/test/ux/quality/24-shape.sh deleted file mode 100755 index 9f72419803..0000000000 --- a/test/ux/quality/24-shape.sh +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env bash -# Q5 · Is the decomposition any good? -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q5 Graph shape — across everything the suite commissioned: how many tasks did it form, and was that the right number?' - -expect_nodes 0 - -# Nothing is said in this journey. It reads the whole run's graph and judges the -# one thing a task graph can get obviously wrong in both directions: a haiku -# that became eleven tasks, or a research report that became none. - -record 'user jobs commissioned' "$(journal "select count(*) from nodes where origin='user' and parent_id='root'")" -record 'total nodes' "$(journal "select count(*) from nodes")" -record 'failed nodes' "$(journal "select count(*) from nodes where status='failed'")" -record 'cancelled nodes' "$(journal "select count(*) from nodes where status='cancelled'")" -record 'max depth below root' "$(journal "select coalesce(max(stage),0) from nodes")" - -{ - echo - echo '### every job the suite commissioned, and what it became' - echo - journal_table " - select - n.id as job, - n.status, - (select count(*) from nodes c where c.parent_id = n.id) as children, - (select count(*) from usage u where u.node_id = n.id) as calls, - round((select coalesce(sum(cost),0) from usage u where u.node_id = n.id), 5) as cost, - substr(n.title,1,44) as title - from nodes n - where n.origin='user' and n.parent_id='root' - order by n.created_order" -} >> "$UX_DIR/notes.md" - -# The other half of proportionality, and the more interesting one: how often -# did a job branch AT ALL? A graph product whose every job is one leaf is not -# over-decomposing, but it is also not decomposing, and a reader of this report -# should be told which of those two things is happening. -branched="$(journal " - select count(*) from nodes n - where n.origin='user' and n.parent_id='root' - and (select count(*) from nodes c where c.parent_id = n.id) > 0")" -total_jobs="$(journal "select count(*) from nodes where origin='user' and parent_id='root'")" -record 'jobs that decomposed into children at all' "$branched of $total_jobs" - -bloated="$(journal " - select count(*) from nodes n - where n.origin='user' and n.parent_id='root' - and (select count(*) from nodes c where c.parent_id = n.id) > 6")" -record 'jobs that fanned out past 6 children' "$bloated" -[ "${bloated:-0}" = "0" ] \ - && _check yes 'no toy job exploded into a bureaucracy' 'no user job with more than 6 children' "$bloated" \ - || _check no 'no toy job exploded into a bureaucracy' 'no user job with more than 6 children' "$bloated jobs did" - -# Held work is not forgotten work: J9 refuses a plan on purpose and the node -# that stays pending behind that refusal is the refusal being honoured. -orphans="$(journal "select count(*) from nodes where origin='user' and status in ('pending','claimed') and held=0 and created_seq < (select max(seq) - 200 from events)")" -record 'jobs still stuck pending long after they were asked for' "$orphans" -record 'jobs deliberately held (a refused plan is not a forgotten one)' \ - "$(journal "select count(*) from nodes where origin='user' and held=1")" -[ "${orphans:-0}" = "0" ] \ - && _check yes 'nothing was commissioned and then quietly forgotten' 'no stale pending user job' "$orphans" \ - || _check no 'nothing was commissioned and then quietly forgotten' 'no stale pending user job' "$orphans stuck" - -failed="$(journal "select count(*) from nodes where origin='user' and status='failed'")" -[ "${failed:-0}" = "0" ] \ - && _check yes 'no commissioned job failed outright' 'zero failed user nodes' "$failed" \ - || _check no 'no commissioned job failed outright' 'zero failed user nodes' "$(journal "select group_concat(id || ': ' || substr(error,1,60), ' | ') from nodes where origin='user' and status='failed'")" - -# Every user turn must have ended in a reply. Silence wearing a message id is -# the cardinal sin in docs/JOURNEY.md, so it gets its own count. -users="$(journal "select count(*) from messages where role='user'")" -agents="$(journal "select count(*) from messages where role in ('agent','system')")" -record 'user turns / agent+system replies' "$users / $agents" -[ "${agents:-0}" -ge "${users:-1}" ] \ - && _check yes 'every user turn was answered — no silence wearing a message id' 'replies ≥ user turns' "$agents ≥ $users" \ - || _check no 'every user turn was answered — no silence wearing a message id' 'replies ≥ user turns' "$agents < $users" - -# The thread is the only mouth, so a journaled agent message that says nothing -# is silence wearing a message id — the exact sin the law names. v3 draws a -# turn in flight as a titled block ("▸ worked …"), so there is no bare stream -# cursor to count; the law is measured where the message lives, in the journal: -# an agent message whose body is empty IS silence wearing a message id. -snap final-thread -blanks="$(journal "select count(*) from messages where role in ('agent','system') and trim(body) = ''")" -record 'journaled agent messages with an empty body' "${blanks:-0}" -[ "${blanks:-0}" = "0" ] \ - && _check yes 'agent messages carry words, not an empty message id' 'no agent message with an empty body' "${blanks:-0}" \ - || _check no 'agent messages carry words, not an empty message id' 'no agent message with an empty body' "$blanks empty bodies — journaled replies that said nothing" - -{ - echo - echo '### where the money went' - echo - journal_table "select model, count(*) as calls, sum(prompt_tokens) as prompt_tok, sum(completion_tokens) as out_tok, round(sum(cost),5) as cost from usage group by model order by cost desc" - echo - echo '### the notebook at the end of the run' - echo - journal_table "select kind, channel, status, substr(body,1,90) as body from facts order by seq" - echo - echo '### charters at the end of the run' - echo - journal_table "select id, status, autonomy, substr(invariant,1,60) as invariant from charters" -} >> "$UX_DIR/notes.md" - -finish diff --git a/test/ux/quality/25-writing.sh b/test/ux/quality/25-writing.sh deleted file mode 100755 index c5ec8c4637..0000000000 --- a/test/ux/quality/25-writing.sh +++ /dev/null @@ -1,56 +0,0 @@ -#!/usr/bin/env bash -# Q6 · A writing task with no code and no numbers in it -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q6 Writing quality — a professional email, where the only measure is whether a person would send it' - -# Coding and research are the two tasks that grade themselves. Most work does -# not: it is a paragraph somebody has to be willing to put their name on. This -# is the journey that catches a product which is good at tests and bad at -# English. - -expect_nodes 1 -since="$(mark)" -started="$(date +%s)" - -say "draft a short professional email declining an invitation to speak at a conference next month because of a scheduling conflict. show me the email." - -assert_journal "select count(*) from messages where seq > $since and role in ('agent','system') and length(body) > 150" \ - 'a draft came back with something in it' 300 -sleep 10 -snap drafted - -elapsed=$(( $(date +%s) - started )) -record 'wall time' "${elapsed}s" - -draft="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" -printf '%s\n' "$draft" > "$UX_DIR/email.txt" -record 'the draft' "$(printf '%s' "$draft" | tr '\n' ' ' | head -c 800)" - -# The literal shape of an email, checked without a model: the parts a person -# would notice missing at a glance. -for part in 'subject' 'dear|hello|hi ' 'regards|sincerely|best'; do - if printf '%s' "$draft" | grep -Eqi "$part"; then - _check yes "the draft has the '$part' part of an email" "an email-shaped section" 'present' - else - _check no "the draft has the '$part' part of an email" "an email-shaped section" 'missing' - fi -done - -if printf '%s' "$draft" | grep -Eqi 'conflict|unable to|regret|decline|cannot attend|won.t be able'; then - _check yes 'it actually declines, and gives the reason it was told to give' \ - 'a decline citing the scheduling conflict' "$(printf '%s' "$draft" | grep -Eoi 'scheduling conflict|conflict|regret[a-z]*' | head -1)" -else - _check no 'it actually declines, and gives the reason it was told to give' \ - 'a decline citing the scheduling conflict' "$(printf '%s' "$draft" | head -c 200)" -fi - -if printf '%s' "$draft" | grep -Eq '\[|\{\{|XXX|TODO'; then - record 'placeholders' "the draft contains fill-in slots: $(printf '%s' "$draft" | grep -Eo '\[[^]]{1,30}\]' | head -3 | tr '\n' ' ')" -else - record 'placeholders' 'none — the draft is ready to send as written' -fi - -judge_this 'draft a short professional email declining an invitation to speak at a conference next month because of a scheduling conflict; show me the email' "$draft" - -dump_turn "$since" -finish diff --git a/test/ux/quality/26-data.sh b/test/ux/quality/26-data.sh deleted file mode 100755 index a96ea3f553..0000000000 --- a/test/ux/quality/26-data.sh +++ /dev/null @@ -1,61 +0,0 @@ -#!/usr/bin/env bash -# Q7 · A data task with exactly one right answer -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q7 Data quality — five numbers, a ranking, and a median: arithmetic has a right answer and this checks it' - -# 12, 7, 41, 3, 28 sorted is 3, 7, 12, 28, 41 — the median is 12. There is -# nothing to argue about, which is the point: a product that is fluent and -# wrong fails here and nowhere else. - -expect_nodes 1 -since="$(mark)" -started="$(date +%s)" -touch "$UX_DIR/.stamp" - -say "here are five numbers: 12, 7, 41, 3, 28 — make a file ranking them from largest to smallest and tell me the median" - -assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ - 'the job finished' 300 -sleep 10 -snap answered -record 'wall time' "$(( $(date +%s) - started ))s" - -answer="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" -printf '%s\n' "$answer" > "$UX_DIR/answer.txt" -record 'the answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 600)" - -# The median is 12 and nothing else. -if printf '%s' "$answer" | grep -Eqi 'median[^0-9]{0,40}12|12[^0-9]{0,20}(is the )?median'; then - _check yes 'the median is right (12)' 'the number 12 named as the median' 'found' -else - _check no 'the median is right (12)' 'the number 12 named as the median' \ - "$(printf '%s' "$answer" | grep -Eoi 'median[^.]{0,40}' | head -1)" -fi - -# The ranking, largest first, in the file it was asked to write. -ranked="$(find "$UX_STATE/workspace" -type f -newer "$UX_DIR/.stamp" 2>/dev/null | head -5)" -record 'files written' "$(printf '%s' "$ranked" | tr '\n' ' ')" -found_order=0 -while IFS= read -r file; do - [ -z "$file" ] && continue - order="$(grep -Eo '\b(41|28|12|7|3)\b' "$file" 2>/dev/null | tr '\n' ' ')" - record "order in $(basename "$file")" "$order" - case "$order" in "41 28 12 7 3 "*) found_order=1 ;; esac -done <<< "$ranked" - -if [ "$found_order" = "1" ]; then - _check yes 'the file ranks them largest to smallest' '41 28 12 7 3 in that order' 'correct' -else - # The thread may carry the ranking even when the file is shaped differently. - if printf '%s' "$answer" | grep -Eq '41.{0,12}28.{0,12}12.{0,12}7.{0,12}3'; then - _check yes 'the ranking is right, largest first' '41 28 12 7 3 in that order' 'correct, in the thread' - else - _check no 'the ranking is right, largest first' '41 28 12 7 3 in that order' \ - "$(printf '%s' "$answer" | grep -Eo '41[^a-z]{0,40}' | head -1)" - fi -fi - -judge_this 'here are five numbers: 12, 7, 41, 3, 28 — make a file ranking them from largest to smallest and tell me the median' "$answer" - -dump_turn "$since" -finish diff --git a/test/ux/quality/27-errand.sh b/test/ux/quality/27-errand.sh deleted file mode 100755 index df28de59d0..0000000000 --- a/test/ux/quality/27-errand.sh +++ /dev/null @@ -1,54 +0,0 @@ -#!/usr/bin/env bash -# Q8 · An everyday errand, shaped the way a person actually asks -source "$(dirname "${BASH_SOURCE[0]}")/../lib.sh" -journey_is 'Q8 Errand quality — an ordinary, slightly vague household ask: does it just do it, or does it interview you first?' - -# Nobody says "produce a structured artifact enumerating". They say "make me a -# packing list". The failure this catches is a product that answers a small -# errand with a clarifying question, a plan, and a preamble. - -expect_nodes 1 -since="$(mark)" -started="$(date +%s)" -touch "$UX_DIR/.stamp" - -say "make me a packing list for a 3-day work trip and save it as a file" - -assert_journal "select count(*) from nodes where created_seq > $since and origin='user' and status='done'" \ - 'the errand was actually run' 300 -sleep 10 -snap done -elapsed=$(( $(date +%s) - started )) -record 'wall time' "${elapsed}s" - -answer="$(journal "select group_concat(body, char(10)) from messages where seq > $since and role in ('agent','system')")" -printf '%s\n' "$answer" > "$UX_DIR/answer.txt" -record 'the answer' "$(printf '%s' "$answer" | tr '\n' ' ' | head -c 700)" - -# It must have just done it: no question asked back for something this small. -asked="$(journal "select count(*) from agent_questions where seq > $since")" -record 'questions asked back' "$asked" -[ "${asked:-0}" = "0" ] \ - && _check yes 'a small errand is done, not interviewed' 'no clarifying question for a 3-day packing list' '0 questions' \ - || _check no 'a small errand is done, not interviewed' 'no clarifying question for a 3-day packing list' "$asked asked" - -# The list itself has to be a list of things a person packs. -hits=0 -for item in charger toothbrush shirt sock laptop 'passport|id' 'trousers|pants' shoes; do - printf '%s' "$answer" | grep -qiE "$item" && hits=$((hits + 1)) -done -record 'ordinary packing items named' "$hits of 8" -[ "$hits" -ge 4 ] \ - && _check yes 'the list is a real packing list' 'at least 4 of the obvious items' "$hits" \ - || _check no 'the list is a real packing list' 'at least 4 of the obvious items' "$hits" - -file="$(find "$UX_STATE/workspace" -type f -newer "$UX_DIR/.stamp" 2>/dev/null | head -1)" -record 'file written' "${file:-}" -[ -n "$file" ] && [ -s "$file" ] \ - && _check yes 'it saved the file it was asked to save' 'a non-empty file in the workspace' "$(wc -c < "$file" | tr -d ' ') bytes" \ - || _check no 'it saved the file it was asked to save' 'a non-empty file in the workspace' "${file:-nothing written}" - -judge_this 'make me a packing list for a 3-day work trip and save it as a file' "$answer" - -dump_turn "$since" -finish diff --git a/test/ux/run.sh b/test/ux/run.sh deleted file mode 100755 index d9c534f5ac..0000000000 --- a/test/ux/run.sh +++ /dev/null @@ -1,435 +0,0 @@ -#!/usr/bin/env bash -# run.sh — drive the real codeaf, as a person, and write down which journeys -# are true today. -# -# The doctrine is docs/JOURNEY.md: a journey passes only when the screen showed -# it, the journal durably recorded it, and nothing else happened. So this -# builds the real binary, gives it a disposable brain that shares nothing with -# the user's, opens it in a real terminal, and types. -# -# test/ux/run.sh everything -# test/ux/run.sh --only 01,02,05 these journeys -# test/ux/run.sh --suite quality the quality suite only -# test/ux/run.sh --keep leave the tmux session and home behind -# test/ux/run.sh --work-model SLUG repin the model that does the work -# test/ux/run.sh --tag NAME write evidence-NAME/ and report-NAME.md -# test/ux/tier-matrix.sh the same journeys on two work models -# -# Money: every model call lands in the disposable store's usage table, and the -# cumulative total is checked between journeys against UX_CAP (default $5). -# Over the cap the run aborts — a test suite must not be able to spend a -# surprise. - -set -uo pipefail - -UX_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -REPO="$(cd "$UX_ROOT/../.." && pwd)" - -: "${UX_CAP:=5.00}" -: "${UX_WIDTH:=200}" -: "${UX_HEIGHT:=50}" -: "${UX_REAL_HOME:=$HOME}" - -SUITES=(journeys quality) -ONLY="" -KEEP=0 -WORK_MODEL="" -TAG="" -REPORT_OVERRIDE="" - -while [ $# -gt 0 ]; do - case "$1" in - --only) ONLY="$2"; shift 2 ;; - --suite) SUITES=("$2"); shift 2 ;; - --keep) KEEP=1; shift ;; - --cap) UX_CAP="$2"; shift 2 ;; - # --work-model repins the model that does the WORK for this whole run. The - # talk model is left alone on purpose: the tier question is whether a - # smarter executor earns its tokens, not whether a smarter receptionist does. - --work-model) WORK_MODEL="$2"; shift 2 ;; - --tag) TAG="$2"; shift 2 ;; - --report) REPORT_OVERRIDE="$2"; shift 2 ;; - -h|--help) sed -n '2,21p' "$0"; exit 0 ;; - *) echo "unknown flag $1" >&2; exit 2 ;; - esac -done - -for tool in tmux sqlite3 python3; do - command -v "$tool" >/dev/null || { echo "run.sh needs $tool on PATH" >&2; exit 2; } -done - -# ------------------------------------------------------------------ the home -# -# Disposable, fresh per run, and named nothing like the user's. HOME is moved -# as well as CODEAF_HOME: the state root override covers everything codeaf -# writes on purpose, and moving HOME too means that anything which does not yet -# honour it — a launchd plist, a stray dotfile — lands here rather than in the -# user's actual home. Whatever shows up under $UX_HOME/.codeaf is a leak, and -# the report says so. - -RUN_ID="$(date +%Y%m%d-%H%M%S)" -UX_RUN="${UX_RUN:-$(mktemp -d "${TMPDIR:-/tmp}/codeaf-ux-$RUN_ID-XXXX")}" -UX_HOME="$UX_RUN/home" -UX_STATE="$UX_HOME/state" -UX_DB="$UX_STATE/graph.db" -# Evidence is filed under the tag when the caller named one, so a tier run or a -# hand-run subset never overwrites the plain run's evidence. .gitignore already -# covers evidence-*/ and report-*.md. -UX_EVIDENCE="$UX_ROOT/evidence${TAG:+-$TAG}" -UX_REPORT="${REPORT_OVERRIDE:-$UX_ROOT/report${TAG:+-$TAG}.md}" -UX_SESSION="codeaf-ux-$$" -UX_SESSION_B="codeaf-ux-$$-b" -UX_BIN="${UX_BIN:-$REPO/bin/codeaf}" - -mkdir -p "$UX_STATE" "$UX_HOME/Library/LaunchAgents" -rm -rf "$UX_EVIDENCE"; mkdir -p "$UX_EVIDENCE" -echo 'journey,verdict,jobs,nodes,cost,tokens_in,tokens_out,seconds,quality,flags' > "$UX_EVIDENCE/gauges.csv" - -cleanup() { - local code=$? - tmux kill-session -t "$UX_SESSION" 2>/dev/null - tmux kill-session -t "$UX_SESSION_B" 2>/dev/null - if [ "$KEEP" = "1" ]; then - echo "kept: $UX_RUN" - else - rm -rf "$UX_RUN" - fi - exit $code -} -trap cleanup EXIT INT TERM - -# ------------------------------------------------------- provider + settings -# -# The one and only read of the user's real store: the API key, and the model -# pins as a baseline. Nothing is ever written back there. - -# A machine that keeps its key in the environment (OPENROUTER_API_KEY, the same -# variable codeaf itself reads first) is as valid a source as a persisted -# config.json, and refusing it made the suite unrunnable on boxes that have -# never run `codeaf` interactively. Neither source is ever written back. -if [ ! -f "$UX_REAL_HOME/.codeaf/config.json" ] \ - && [ -z "${OPENROUTER_API_KEY:-}${OPENAI_API_KEY:-}" ]; then - echo "no $UX_REAL_HOME/.codeaf/config.json and no OPENROUTER_API_KEY — the suite needs a provider key" >&2 - exit 2 -fi - -python3 - "$UX_REAL_HOME/.codeaf" "$UX_STATE" "$WORK_MODEL" <<'PY' -import json, os, sys -real, disposable = sys.argv[1], sys.argv[2] -work_model = sys.argv[3] if len(sys.argv) > 3 else "" -try: - config = json.load(open(os.path.join(real, "config.json"))) -except FileNotFoundError: - config = {} -key = config.get("api_key", "") or os.environ.get("OPENROUTER_API_KEY", "") or os.environ.get("OPENAI_API_KEY", "") -if not key: - raise SystemExit("no api_key in the real config.json and none in the environment") -json.dump({"api_key": key}, open(os.path.join(disposable, "config.json"), "w")) -os.chmod(os.path.join(disposable, "config.json"), 0o600) - -# Settings are the user's pins as a baseline — they already name the cheapest -# capable models — with the rails a disposable run wants on top. -try: - settings = json.load(open(os.path.join(real, "settings.json"))) -except FileNotFoundError: - settings = {} -settings.setdefault("chat_model", "~deepseek/deepseek-v4-flash-latest") -settings.setdefault("task_model", "~deepseek/deepseek-v4-flash-latest") -if work_model: - settings["task_model"] = work_model - # The plan model follows the work model for a tier run, or the comparison - # measures two things at once and attributes the difference to one of them. - settings["plan_model"] = work_model -json.dump(settings, open(os.path.join(disposable, "settings.json"), "w"), indent=2) -print("talk=%s work=%s plan=%s boost=%s" % ( - settings.get("chat_model"), settings.get("task_model"), - settings.get("plan_model"), settings.get("boost_model"))) -PY -[ $? -eq 0 ] || exit 2 - -# ------------------------------------------------------------------- binary - -# UX_SKIP_BUILD=1 with a pre-set UX_BIN runs a binary somebody else built — -# a build from a known commit while the working tree is mid-edit, say — so a -# tree that moves underneath a long run cannot change the program mid-suite. -if [ "${UX_SKIP_BUILD:-0}" = "1" ]; then - [ -x "$UX_BIN" ] || { echo "UX_SKIP_BUILD=1 but $UX_BIN is not executable" >&2; exit 2; } - echo "using $UX_BIN (build skipped)" -else - echo "building $UX_BIN" - ( cd "$REPO" && go build -o "$UX_BIN" ./cmd/codeaf ) || exit 2 -fi - -# UX_ENV is the one description of the disposable brain. Both this runner and -# the journeys that relaunch it (leave-and-return, second window) build their -# tmux command from exactly this string, so a window opened by journey 12 is -# the same brain the suite has been talking to all along. -UX_ENV="HOME=$UX_HOME CODEAF_HOME=$UX_STATE CODEAF_PROFILE_DIR=$UX_STATE" -UX_ENV="$UX_ENV CODEAF_DAILY_BUDGET=$UX_CAP CODEAF_BRIEF_AFTER=0 CODEAF_PRACTICE_BUDGET=0" -UX_ENV="$UX_ENV TERM=xterm-256color" - -export UX_BIN UX_HOME UX_STATE UX_DB UX_EVIDENCE UX_CAP UX_ROOT REPO UX_RUN UX_ENV -export UX_SESSION UX_SESSION_B UX_WIDTH UX_HEIGHT - -launch() { - local sess="$1"; shift - tmux kill-session -t "$sess" 2>/dev/null - tmux new-session -d -s "$sess" -x "$UX_WIDTH" -y "$UX_HEIGHT" \ - "env $UX_ENV $* '$UX_BIN' chat; echo CODEAF-EXITED; sleep 900" - sleep 6 -} - -# --------------------------------------------------------------- the money - -total_spend() { sqlite3 -readonly "$UX_DB" "select coalesce(round(sum(cost),6),0) from usage" 2>/dev/null || echo 0; } - -over_cap() { - local spent="$1" - python3 -c "import sys; sys.exit(0 if float(sys.argv[1]) > float(sys.argv[2]) else 1)" "$spent" "$UX_CAP" -} - -# --------------------------------------------------------------- the journeys - -declare -a ORDER=() -for suite in "${SUITES[@]}"; do - for script in "$UX_ROOT/$suite"/*.sh; do - [ -e "$script" ] || continue - ORDER+=("$script") - done -done - -wanted() { - [ -z "$ONLY" ] && return 0 - local name; name="$(basename "$1" .sh)" - case ",$ONLY," in *",${name%%-*},"*) return 0 ;; *",$name,"*) return 0 ;; esac - return 1 -} - -echo -echo "codeaf UX suite — $(date)" -echo " binary $UX_BIN" -echo " home $UX_HOME (real home untouched: $UX_REAL_HOME)" -echo " session $UX_SESSION at ${UX_WIDTH}x${UX_HEIGHT}" -echo " cap \$$UX_CAP" -echo - -launch "$UX_SESSION" -# The first frame: v3 draws the composer prompt "›" before anything has been -# said, and the wordmark rides the frame's own chrome. Accepting either is not -# a weakened check — the point of it is "the program drew something of its -# own", and CODEAF-EXITED on the pane means it did not. -if ! tmux capture-pane -t "$UX_SESSION" -p | grep -Eq 'codeaf|›'; then - echo "codeaf did not draw a first frame; aborting" >&2 - tmux capture-pane -t "$UX_SESSION" -p - exit 1 -fi -if tmux capture-pane -t "$UX_SESSION" -p | grep -q 'CODEAF-EXITED'; then - echo "codeaf exited instead of drawing a frame; aborting" >&2 - tmux capture-pane -t "$UX_SESSION" -p - exit 1 -fi - -declare -a RESULTS=() -declare -a GAUGES=() -ABORTED="" -JUDGE_CALLS=0 -START_TS="$(date +%s)" - -sql() { sqlite3 -readonly "$UX_DB" "$1" 2>/dev/null; } -watermark() { sql "select coalesce(max(seq),0) from events"; } - -for script in "${ORDER[@]}"; do - name="$(basename "$script" .sh)" - wanted "$script" || continue - - before="$(total_spend)" - mark_before="$(watermark)" - started="$(date +%s)" - UX_JOURNEY="$name" UX_SESSION="$UX_SESSION" bash "$script" - code=$? - elapsed=$(( $(date +%s) - started )) - after="$(total_spend)" - cost="$(python3 -c "print('%.4f' % (float('$after') - float('$before')))")" - - verdict="$(cat "$UX_EVIDENCE/$name/verdict" 2>/dev/null || echo FAIL)" - checks="$(cat "$UX_EVIDENCE/$name/checks" 2>/dev/null || echo '0/0 checks')" - [ "$code" -ne 0 ] && [ "$verdict" = "PASS" ] && verdict=FAIL - RESULTS+=("$name|$verdict|$checks|${elapsed}s|\$$cost") - - # ------------------------------------------------------------- the gauges - # - # Proportionality: how much machinery did the ask actually buy? A haiku is - # one node. A haiku that became five is a passing journey with a quality - # problem, and the report has to be able to say so. - nodes="$(sql "select count(*) from nodes where created_seq > $mark_before")" - jobs="$(sql "select count(*) from nodes where created_seq > $mark_before and origin='user' and parent_id='root'")" - tok_in="$(sql "select coalesce(sum(prompt_tokens),0) from usage where seq > $mark_before")" - tok_out="$(sql "select coalesce(sum(completion_tokens),0) from usage where seq > $mark_before")" - calls="$(sql "select count(*) from usage where seq > $mark_before")" - users="$(sql "select count(*) from messages where seq > $mark_before and role='user'")" - agents="$(sql "select count(*) from messages where seq > $mark_before and role='agent'")" - systems="$(sql "select count(*) from messages where seq > $mark_before and role='system'")" - expect="$(cat "$UX_EVIDENCE/$name/expect-nodes" 2>/dev/null || echo 1)" - - # Noise: agent turns beyond one reply per thing the user said. Receipts and - # deliverables are not noise — they are the product working. - noise="$(python3 -c "print(max(0, ${agents:-0} - ${users:-0}))")" - - flags="" - if [ "${nodes:-0}" -gt "${expect:-1}" ] && [ "${nodes:-0}" -gt 2 ]; then - flags="$flags OVER-DECOMPOSED(${nodes}v${expect})" - fi - [ "${noise:-0}" -ge 2 ] && flags="$flags CHATTY(+$noise)" - python3 -c "import sys; sys.exit(0 if float('$cost') > 0.05 else 1)" && flags="$flags EXPENSIVE" - [ "${elapsed:-0}" -gt 180 ] && flags="$flags SLOW" - - # Tokens are the honest unit. Dollars hide behind whichever model happened to - # answer; 50k of input tokens to write a haiku is disproportionate whoever - # charged for it, and it is the number that moves when a smarter work model - # re-reads the same context on every turn. - [ "${tok_in:-0}" -gt 50000 ] && flags="$flags TOKEN-HEAVY(${tok_in}in)" - - # ------------------------------------------------------------- the judge - quality="—"; why="" - if [ -s "$UX_EVIDENCE/$name/judge-deliverable" ]; then - judged="$(python3 "$UX_ROOT/judge.py" "$UX_STATE" \ - "$UX_EVIDENCE/$name/judge-ask" "$UX_EVIDENCE/$name/judge-deliverable" 2>/dev/null)" - JUDGE_CALLS=$((JUDGE_CALLS + 1)) - quality="${judged%%|*}" - why="${judged#*|}" - { - echo - echo "### quality judge" - echo - echo "- **score**: $quality/5" - echo "- **why**: $why" - } >> "$UX_EVIDENCE/$name/notes.md" - printf ' · quality %s/5 — %s\n' "$quality" "$why" >&2 - case "$quality" in - 1|2) flags="$flags POOR-RESULT" ;; - 3) flags="$flags MIDDLING" ;; - esac - fi - - concurrency="$(cat "$UX_EVIDENCE/$name/concurrency" 2>/dev/null || echo '')" - GAUGES+=("$name|$verdict|${jobs:-0}/${nodes:-0}|\$$cost|${tok_in:-0}|${tok_out:-0}|${elapsed}|$quality|${flags:-—}|$why|$concurrency|${calls:-0}") - printf '%s,%s,%s,%s,%s,%s,%s,%s,%s,"%s"\n' \ - "$name" "$verdict" "${jobs:-0}" "${nodes:-0}" "$cost" "${tok_in:-0}" "${tok_out:-0}" \ - "$elapsed" "$quality" "${flags# }" >> "$UX_EVIDENCE/gauges.csv" - - if over_cap "$after"; then - ABORTED="spend \$$after exceeded the \$$UX_CAP cap after $name" - echo "ABORT: $ABORTED" >&2 - break - fi -done - -TOTAL="$(total_spend)" -WALL=$(( $(date +%s) - START_TS )) - -# --------------------------------------------------------------- leak check - -LEAKS="$(find "$UX_HOME" -maxdepth 1 -name '.codeaf' 2>/dev/null)" -PLIST="$(ls "$UX_HOME/Library/LaunchAgents" 2>/dev/null | tr '\n' ' ')" - -# ------------------------------------------------------------------ report - -{ - echo "# codeaf UX suite — which journeys are true today" - echo - echo "Run $(date -u '+%Y-%m-%d %H:%M UTC') · binary \`$(cd "$REPO" && git rev-parse --short HEAD)\` · wall ${WALL}s" - echo - echo "Real binary, real terminal (tmux ${UX_WIDTH}x${UX_HEIGHT}), real provider (OpenRouter)," - echo "disposable brain at \`\$CODEAF_HOME\`. Models: $(python3 -c " -import json;s=json.load(open('$UX_STATE/settings.json'));print('talk %s · work %s · plan %s · boost %s' % (s.get('chat_model'),s.get('task_model'),s.get('plan_model'),s.get('boost_model')))")" - echo - echo "**Total spend: \$$TOTAL** (cap \$$UX_CAP) · plus $JUDGE_CALLS quality-judge calls made outside codeaf and not in its journal" - [ -n "$ABORTED" ] && echo && echo "> **RUN ABORTED** — $ABORTED" - echo - echo '## The report card' - echo - echo 'Completion is the cheap half. `jobs/nodes` is how much machinery the ask bought' - echo '(a haiku should be one node); `quality` is a cheap model reading the deliverable' - echo 'against a fixed rubric — literal ask satisfied, answer-first, competent, 1–5.' - echo - echo '| journey | pass | jobs/nodes | $ | tokens in/out | calls | seconds | quality/5 | flags |' - echo '|---|---|---|---|---|---|---|---|---|' - for row in "${GAUGES[@]}"; do - IFS='|' read -r n v nodes money tin tout secs q f _why _conc calls <<< "$row" - echo "| $n | $v | $nodes | $money | $tin / $tout | $calls | $secs | $q | $f |" - done - echo - for row in "${GAUGES[@]}"; do - IFS='|' read -r n v nodes money tin tout secs q f why conc calls <<< "$row" - [ -n "$why" ] && echo "- **$n** judged $q/5 — $why" - done - echo - for row in "${GAUGES[@]}"; do - IFS='|' read -r n v nodes money tin tout secs q f why conc calls <<< "$row" - if [ -n "$conc" ]; then - echo "**Parallelism** — \`$n\` ran ${nodes%%/*} jobs from one sentence in ${secs}s of the person's time." - echo "Concurrency factor **${conc}×** (summed job seconds ÷ wall seconds). 1.0 means a queue;" - echo 'above it the graph is doing what terminal tabs would otherwise be for.' - echo - fi - done - echo '## Journeys' - echo - echo '| journey | verdict | checks | time | cost | evidence |' - echo '|---|---|---|---|---|---|' - for row in "${RESULTS[@]}"; do - IFS='|' read -r n v c t m <<< "$row" - echo "| $n | **$v** | $c | $t | $m | [notes](evidence/$n/notes.md) |" - done - echo - echo '### Not run' - echo - echo '| journey | why |' - echo '|---|---|' - echo '| Practice loop | idle-time loop: `CODEAF_PRACTICE_IDLE` defaults to 20m of quiet before self-practice, so an honest test costs 20 minutes of nothing happening. Out of scope for a suite that must stay under a few minutes per journey. |' - echo "| J9 consent (full) | needs a plan estimated over the real \$3 rail. The cheap variant runs instead: \`CODEAF_PLAN_CONSENT\` lowered so the same gate fires on a job costing cents. |" - echo - echo '### Housekeeping' - echo - if [ -n "$LEAKS" ]; then - echo "- ⚠️ **state leak**: something wrote to \`\$HOME/.codeaf\` despite \`CODEAF_HOME\` — $LEAKS" - else - echo '- ✅ no state leak: nothing landed in `$HOME/.codeaf`; the whole run lived under `$CODEAF_HOME`.' - fi - if [ -n "$PLIST" ]; then - echo "- ⚠️ the run installed a launch agent in the disposable home: $PLIST" - else - echo '- ✅ no launch agent installed (the standing-watch offer is declined by the suite).' - fi - echo "- the user's real store at \`$UX_REAL_HOME/.codeaf\` was read once (api_key, settings.json) and never written." - echo - for row in "${RESULTS[@]}"; do - IFS='|' read -r n v c t m <<< "$row" - echo "---" - echo - echo "## $n — $v" - echo - sed 's/^# .*$//' "$UX_EVIDENCE/$n/notes.md" 2>/dev/null - echo - done -} > "$UX_REPORT" - -echo -echo "================================================================" -printf '%-26s %-9s %-14s %6s %8s\n' journey verdict checks time cost -for row in "${RESULTS[@]}"; do - IFS='|' read -r n v c t m <<< "$row" - printf '%-26s %-9s %-14s %6s %8s\n' "$n" "$v" "$c" "$t" "$m" -done -echo "----------------------------------------------------------------" -echo "total spend \$$TOTAL / cap \$$UX_CAP · report: $UX_REPORT" -echo - -FAILED=0 -for row in "${RESULTS[@]}"; do - IFS='|' read -r n v c t m <<< "$row" - [ "$v" = "FAIL" ] && FAILED=$((FAILED + 1)) -done -[ -n "$ABORTED" ] && FAILED=$((FAILED + 1)) -[ "$FAILED" -gt 0 ] && exit 1 -exit 0 diff --git a/test/ux/tier-matrix.sh b/test/ux/tier-matrix.sh deleted file mode 100755 index b011e77da8..0000000000 --- a/test/ux/tier-matrix.sh +++ /dev/null @@ -1,134 +0,0 @@ -#!/usr/bin/env bash -# tier-matrix.sh — the same four journeys, twice, on two work models. -# -# The product's claim is that it is cheap AND good on a small work model. Two -# things can falsify that. It might be cheap and bad, which the quality column -# catches. Or it might be no better when handed a smarter model — paying tier-2 -# prices for tier-1 results — and worse, burning materially more tokens on the -# same small ask because a bigger model re-reads more context per turn. That -# second failure is invisible in dollars alone, which is why tokens are in the -# table beside them. -# -# test/ux/tier-matrix.sh flash vs z-ai/glm-5.2 -# test/ux/tier-matrix.sh other/model flash vs whatever you name -# -# Each tier gets a fresh disposable home. The talk model stays flash in both: -# the question is about the executor, not the receptionist. The table is -# appended to test/ux/report.md when one exists. - -set -uo pipefail - -UX_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -SUBSET="${UX_TIER_SUBSET:-02,04,20,21}" -UPPER="${1:-z-ai/glm-5.2}" -BASE_LABEL="flash (settings default)" - -echo "tier matrix: $SUBSET on the default work model, then on $UPPER" - -"$UX_ROOT/run.sh" --only "$SUBSET" --tag tier-base --report "$UX_ROOT/report-tier-base.md" -base_code=$? -"$UX_ROOT/run.sh" --only "$SUBSET" --tag tier-upper --work-model "$UPPER" \ - --report "$UX_ROOT/report-tier-upper.md" -upper_code=$? - -python3 - "$UX_ROOT" "$BASE_LABEL" "$UPPER" <<'PY' -import csv, os, sys - -root, base_label, upper_label = sys.argv[1], sys.argv[2], sys.argv[3] - - -def read(tag): - path = os.path.join(root, "evidence-%s" % tag, "gauges.csv") - rows = {} - try: - with open(path) as handle: - for row in csv.DictReader(handle): - rows[row["journey"]] = row - except FileNotFoundError: - pass - return rows - - -base, upper = read("tier-base"), read("tier-upper") - - -def number(row, key, default=0.0): - try: - return float(row.get(key) or default) - except ValueError: - return default - - -lines = [] -lines.append("## The model-tier matrix") -lines.append("") -lines.append("The same journeys, the same words, two work models. The talk model is flash in") -lines.append("both rows — the question is whether a smarter *executor* earns its tokens.") -lines.append("") -lines.append("| journey | tier | $ | tokens in/out | seconds | quality/5 | verdict |") -lines.append("|---|---|---|---|---|---|---|") -flags = [] -for journey in sorted(set(base) | set(upper)): - for label, rows in ((base_label, base), (upper_label, upper)): - row = rows.get(journey) - if not row: - lines.append("| %s | %s | — | — | — | — | not run |" % (journey, label)) - continue - lines.append("| %s | %s | $%s | %s / %s | %s | %s | %s |" % ( - journey, label, row["cost"], row["tokens_in"], row["tokens_out"], - row["seconds"], row["quality"], row["verdict"])) - - low, high = base.get(journey), upper.get(journey) - if not (low and high): - continue - low_q, high_q = low.get("quality", "—"), high.get("quality", "—") - if low_q.isdigit() and high_q.isdigit() and int(high_q) <= int(low_q): - flags.append( - "- **TIER-WASTE — %s**: the bigger model scored %s/5 against flash's %s/5 " - "while costing $%s instead of $%s. Paying more bought nothing." - % (journey, high_q, low_q, high["cost"], low["cost"])) - low_in, high_in = number(low, "tokens_in"), number(high, "tokens_in") - if low_in > 0 and high_in > low_in * 1.5: - flags.append( - "- **TIER-WASTE — %s**: the same small ask burned %d input tokens on the bigger " - "model against %d on flash (%.1f×). Whatever the answer was worth, that context " - "was re-read for it." % (journey, high_in, low_in, high_in / low_in)) - elif high_in > 0 and low_in > high_in * 1.5: - flags.append( - "- %s: the bigger model used FEWER input tokens (%d vs %d) — it needed fewer " - "turns to get there." % (journey, high_in, low_in)) - -lines.append("") -if flags: - lines.append("### What the matrix says") - lines.append("") - lines.extend(flags) -else: - lines.append("No tier-waste flags: the bigger model either scored higher or used") - lines.append("comparable tokens on every journey in the subset.") -lines.append("") - -base_total = sum(number(r, "cost") for r in base.values()) -upper_total = sum(number(r, "cost") for r in upper.values()) -base_tok = sum(number(r, "tokens_in") for r in base.values()) -upper_tok = sum(number(r, "tokens_in") for r in upper.values()) -lines.append("**Subset totals** — %s: $%.4f over %d input tokens · %s: $%.4f over %d input tokens" - % (base_label, base_total, base_tok, upper_label, upper_total, upper_tok)) -if base_total > 0: - lines.append("") - lines.append("The upper tier cost **%.1f×** the flash tier for this subset." - % (upper_total / base_total)) -lines.append("") - -section = "\n".join(lines) -report = os.path.join(root, "report.md") -if os.path.exists(report): - with open(report, "a") as handle: - handle.write("\n---\n\n" + section) - print("appended the tier matrix to report.md") -with open(os.path.join(root, "report-tiers.md"), "w") as handle: - handle.write(section) -print(section) -PY - -exit $(( base_code || upper_code )) From 98b8dd5bcf0a9c4999ff35ac43737b12c9179e5b Mon Sep 17 00:00:00 2001 From: Abir Abbas Date: Wed, 16 Sep 2026 17:18:26 -0400 Subject: [PATCH 7/8] docs/changes: entry for #1068 Co-Authored-By: Claude Fable 5.1 --- .../unreleased/1068-internal-records-leave-the-tree.md | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 docs/changes/unreleased/1068-internal-records-leave-the-tree.md diff --git a/docs/changes/unreleased/1068-internal-records-leave-the-tree.md b/docs/changes/unreleased/1068-internal-records-leave-the-tree.md new file mode 100644 index 0000000000..e861a94892 --- /dev/null +++ b/docs/changes/unreleased/1068-internal-records-leave-the-tree.md @@ -0,0 +1,10 @@ +--- +kind: removed +title: the internal experiment records leave the tree, and the design docs stop naming the bench machine +pr: 1068 +surface: [docs] +invalidates: + - "`docs/design/pin-doe/`, `docs/design/workspace-foundation/` and the day-by-day wave records under `docs/design/conversation-runtime/` (status, plans, audits, local change notes, the prototype) were in the tree. They are gone; `docs/design/conversation-runtime/PARETO.md` stays because `bench/conversation/README.md` cites it as the measurement contract. `docs/design/plan-gate-doe/REPORT.md` and `docs/design/turn-wall-share-doe/` stay too: live comments in `cmd/codeaf`, `internal/config`, `internal/session` and `internal/splitgate` point at them as the measured evidence behind a default." + - "Bench READMEs and the design docs under `docs/design/` spelled the bench machine by name (`ssh spark`, `~/.config/fleet/secrets.env`, `/private/tmp/af-conversation-ops/…`). Those pages now say `ssh benchhost`, a sourced `secrets.env` and a scratch bench-ops directory. `CLAUDE.md`, the `Makefile`, `docs/rules/ci.md`, `BENCHMARKS.md` and code comments were not part of this pass and still say Spark." + - "`test/ux/` was removed by #991 because it drove the v1 and v2 surfaces. A revival of it was proposed here and withdrawn: its journal assertions read `graph.db` tables the v3 chat never writes, nothing ran it, and it had never been run. `internal/e2e` remains the one suite that drives the binary, as `docs/JOURNEY.md` says." +--- From 72a5c6b79fec5179c13270f21ae1e35bb16cbd27 Mon Sep 17 00:00:00 2001 From: Abir Abbas Date: Wed, 16 Sep 2026 17:30:52 -0400 Subject: [PATCH 8/8] docs/changes: the #1068 entry says why the journey suite is out, correctly The first draft said the v3 chat never writes the graph.db tables the suite reads. Running the suite showed both message rows landing, so that sentence was wrong; what is true is that nothing runs the suite, it had never been run, and its first journey fails on its first screen check (#1072). Co-Authored-By: Claude Fable 5.1 --- docs/changes/unreleased/1068-internal-records-leave-the-tree.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/changes/unreleased/1068-internal-records-leave-the-tree.md b/docs/changes/unreleased/1068-internal-records-leave-the-tree.md index e861a94892..a35db8e729 100644 --- a/docs/changes/unreleased/1068-internal-records-leave-the-tree.md +++ b/docs/changes/unreleased/1068-internal-records-leave-the-tree.md @@ -6,5 +6,5 @@ surface: [docs] invalidates: - "`docs/design/pin-doe/`, `docs/design/workspace-foundation/` and the day-by-day wave records under `docs/design/conversation-runtime/` (status, plans, audits, local change notes, the prototype) were in the tree. They are gone; `docs/design/conversation-runtime/PARETO.md` stays because `bench/conversation/README.md` cites it as the measurement contract. `docs/design/plan-gate-doe/REPORT.md` and `docs/design/turn-wall-share-doe/` stay too: live comments in `cmd/codeaf`, `internal/config`, `internal/session` and `internal/splitgate` point at them as the measured evidence behind a default." - "Bench READMEs and the design docs under `docs/design/` spelled the bench machine by name (`ssh spark`, `~/.config/fleet/secrets.env`, `/private/tmp/af-conversation-ops/…`). Those pages now say `ssh benchhost`, a sourced `secrets.env` and a scratch bench-ops directory. `CLAUDE.md`, the `Makefile`, `docs/rules/ci.md`, `BENCHMARKS.md` and code comments were not part of this pass and still say Spark." - - "`test/ux/` was removed by #991 because it drove the v1 and v2 surfaces. A revival of it was proposed here and withdrawn: its journal assertions read `graph.db` tables the v3 chat never writes, nothing ran it, and it had never been run. `internal/e2e` remains the one suite that drives the binary, as `docs/JOURNEY.md` says." + - "`test/ux/` was removed by #991 because it drove the v1 and v2 surfaces. A revival of it was proposed here and withdrawn: nothing ran it, it had never been run, and when it was run its first journey failed on its first screen check (#1072). `internal/e2e` remains the one suite that drives the binary, as `docs/JOURNEY.md` says." ---