Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions PERF.md
Original file line number Diff line number Diff line change
Expand Up @@ -1634,10 +1634,10 @@ the pointer crosses.

**The deadline is `callDeadline`, ten seconds, for every call**, and it is one
number rather than one per kind for a reason worth writing down, because the
obvious change was made and taken back out (#846). A person's keystroke on a
obvious change was made and then taken back out. A person's keystroke on a
question is a call too — internal/tui3's `answerQuestion` asks its door straight
from Update rather than from a command — so a longer window for an act would be
a longer time the terminal can sit without drawing. Measured on the Spark:
a longer time the terminal can sit without drawing. Measured on the bench host:
eighteen copies of the questions e2e, six at a time, and with a thirty-second
act window two of them took 53 seconds where every other copy took 22, both
losing the receipt because nothing repainted. Twelve copies of the parent commit
Expand Down
4 changes: 2 additions & 2 deletions bench/census/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@ make census TOP=40 DAYS=7 MIN=10 CHAINS=20 # widen what it prints

It is pure Go with no dependencies, and it never writes to the log it reads.

## Nightly, on the Spark
## Nightly, on the bench host

The measurement is the point: five waves of recovery work each move a number in
that table, and without a run that happens without anybody asking, every one of
them is an argument about anecdotes. So the Spark syncs the laptop's log and
them is an argument about anecdotes. So the bench host syncs the work machine's log and
runs the same target once a night, keeping one file per day:

```sh
Expand Down
6 changes: 3 additions & 3 deletions bench/conversation/lib/adapters.sh
Original file line number Diff line number Diff line change
Expand Up @@ -507,7 +507,7 @@ arm_tui_argv() {
# 2026-09-10 — after the seven-panel home landed — a fresh screen drew the
# state word at the right edge of the status row with nothing in front of
# it, so the needle matched nothing and every interactive codeaf cell
# recorded `unsupported` at the ready wait. Measured on the Spark that
# recorded `unsupported` at the ready wait. Measured on the bench host that
# day: `followup-while-working` and `work-result-recalled` both ended
# `noready` after 91s against a pane whose last line read `idle`.
#
Expand All @@ -520,7 +520,7 @@ arm_tui_argv() {
ARM_READY_RE='(· idle|idle[[:space:]]*$)'
ARM_BUSY_RE='(· (working|interrupted)|(working|interrupted)[[:space:]]*$|[0-9]+ running)'
ARM_ASK_RE='waiting · your call'
ARM_DOOR_NOTE='markers from internal/tui3/render.go (stateWord, waitingWord), matched against a live pane in /private/tmp/af-conversation-ops/host-live-01'
ARM_DOOR_NOTE='markers from internal/tui3/render.go (stateWord, waitingWord), matched against a live pane in a scratch bench-ops dir (host-live-01)'
# Hosted, like any other conversation: the interactive door is the one
# place this suite can measure the product's actual default, and passing
# --no-host here would quietly measure something else. The cell's own
Expand All @@ -542,7 +542,7 @@ arm_tui_argv() {
# whichever provider is in use, so a rig that wired pi to the guard and
# then waited for "(openrouter)" waits forever: a live pane on a guarded
# run reads "(guard) deepseek/deepseek-v4-flash-0731 • low"
# (/private/tmp/af-conversation-ops/live-interactive-peers-01/followup-while-working-pi/screen.txt).
# (a scratch bench-ops dir: live-interactive-peers-01/followup-while-working-pi/screen.txt).
ARM_READY_RE="\\($(arm_provider_name)\\)"
ARM_BUSY_RE='Working\.\.\.'
ARM_ASK_RE='(\[y\]|approve|Allow\?)'
Expand Down
6 changes: 3 additions & 3 deletions bench/conversation/notes/pilot-02-03.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
Three live runs the owner made against codeaf `4d9d91fb0`, pinned to
`deepseek/deepseek-v4-flash-0731` throughout:

/private/tmp/af-conversation-ops/live-pilot-02 print door, 4 scenarios × 3 arms
/private/tmp/af-conversation-ops/live-pilot-03 the same, repeated
/private/tmp/af-conversation-ops/live-interactive-peers-01 interactive door, pi and omp
scratch bench-ops dir: live-pilot-02 print door, 4 scenarios × 3 arms
scratch bench-ops dir: live-pilot-03 the same, repeated
scratch bench-ops dir: live-interactive-peers-01 interactive door, pi and omp

This file adjudicates the non-pass cells in those runs. **Nothing in the raw
evidence has been touched**: no receipt is rewritten, no verdict is edited in
Expand Down
5 changes: 2 additions & 3 deletions bench/prompt-diet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,8 @@ comparing two harnesses rather than two prompts.

## Running it

Everything runs on the Spark. `~/.local/bin/go`, the key in
`~/.config/fleet/secrets.env`, and `tmux` are all there; the laptop has the
first two and the owner's standing order says the suites do not run on it.
Everything runs on a bench host. `~/.local/bin/go`, the key in a sourced
`secrets.env`, and `tmux` are all there.

```sh
bench/prompt-diet/run.sh 6aa6a946e dev --layers a # free, seconds
Expand Down
14 changes: 7 additions & 7 deletions bench/prompt-diet/run.sh
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@
# side — and a rig that moved with the subject would be measuring two harnesses
# rather than two prompts.
#
# IT RUNS ON THE SPARK. Layer B alone is seventeen minutes of a real binary in a
# real terminal against a real model, and the owner's standing order is that no
# full suite runs on the laptop. Nothing here refuses to run elsewhere — a
# IT RUNS ON THE BENCH HOST. Layer B alone is seventeen minutes of a real binary in a
# real terminal against a real model, and the full suite is not meant to run on a
# work machine. Nothing here refuses to run elsewhere — a
# refusal keyed off a hostname would be a lie on the next machine — but every
# recipe in docs/design/prompt-diet/BENCH.md is spelled for `ssh spark`.
# recipe in docs/design/prompt-diet/BENCH.md is spelled for `ssh benchhost`.
#
# IT SPENDS REAL MONEY. Layers B and C call a real model. Layer A does not, and
# is worth running on its own while editing.
Expand Down Expand Up @@ -72,7 +72,7 @@ LAYERS="a,c,d"
# THE PIN IS THE ONE THE BATTERIES WERE CALIBRATED ON, and it is not the one
# this wave started with.
#
# MEASURED on the Spark, 2026-09-10. Pinned to `deepseek/deepseek-v4.1-flash`,
# MEASURED on a bench host, 2026-09-10. Pinned to `deepseek/deepseek-v4.1-flash`,
# cells died mid-turn with the provider's own sentence: "0 endpoints out of 1
# requested are available matching your guardrail restrictions and data policy
# … Paid model training violation (account settings): 1 endpoint excluded". It
Expand Down Expand Up @@ -169,14 +169,14 @@ if git -C "$OUT" rev-parse --show-toplevel >/dev/null 2>&1; then
exit 1
fi

# GO IS NOT ON THE SPARK'S NON-INTERACTIVE PATH. Finding it here rather than
# GO IS NOT ON THE BENCH HOST'S NON-INTERACTIVE PATH. Finding it here rather than
# asking every caller to export it is what keeps the ssh recipes in BENCH.md to
# one line, and a missing toolchain says so once instead of failing four layers
# deep with "go: command not found".
GO="${GO:-$(command -v go || true)}"
[ -n "$GO" ] && [ -x "$GO" ] || GO="$HOME/.local/bin/go"
if [ ! -x "$GO" ]; then
warn "no go toolchain: set GO=<path> (the Spark's lives at ~/.local/bin/go)"
warn "no go toolchain: set GO=<path> (the bench host's lives at ~/.local/bin/go)"
exit 1
fi

Expand Down
10 changes: 10 additions & 0 deletions docs/changes/unreleased/1068-internal-records-leave-the-tree.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
kind: removed
title: the internal experiment records leave the tree, and the design docs stop naming the bench machine
pr: 1068
surface: [docs]
invalidates:
- "`docs/design/pin-doe/`, `docs/design/workspace-foundation/` and the day-by-day wave records under `docs/design/conversation-runtime/` (status, plans, audits, local change notes, the prototype) were in the tree. They are gone; `docs/design/conversation-runtime/PARETO.md` stays because `bench/conversation/README.md` cites it as the measurement contract. `docs/design/plan-gate-doe/REPORT.md` and `docs/design/turn-wall-share-doe/` stay too: live comments in `cmd/codeaf`, `internal/config`, `internal/session` and `internal/splitgate` point at them as the measured evidence behind a default."
- "Bench READMEs and the design docs under `docs/design/` spelled the bench machine by name (`ssh spark`, `~/.config/fleet/secrets.env`, `/private/tmp/af-conversation-ops/…`). Those pages now say `ssh benchhost`, a sourced `secrets.env` and a scratch bench-ops directory. `CLAUDE.md`, the `Makefile`, `docs/rules/ci.md`, `BENCHMARKS.md` and code comments were not part of this pass and still say Spark."
- "`test/ux/` was removed by #991 because it drove the v1 and v2 surfaces. A revival of it was proposed here and withdrawn: nothing ran it, it had never been run, and when it was run its first journey failed on its first screen check (#1072). `internal/e2e` remains the one suite that drives the binary, as `docs/JOURNEY.md` says."
---
Loading
Loading