diff --git a/packages/extension/skills/detached-run/SKILL.md b/packages/extension/skills/detached-run/SKILL.md new file mode 100644 index 000000000..c69612d02 --- /dev/null +++ b/packages/extension/skills/detached-run/SKILL.md @@ -0,0 +1,126 @@ +--- +name: detached-run +description: Launch a long solve, scan, or fit in the background so it survives the tool call, is bounded in wall-clock and memory, and can be stopped cleanly (PID file, whole process group, children included). Use for runs the run gate does not cover (Python lanes, reduced-space or custom-driver arms, scans), before trusting a "free memory" number, and whenever a background run must be killed. +agents: [experimenter, engineer] +surface: public +revision: 1 +--- + +# detached-run — bounded background runs you can actually stop + +A background run needs four things the raw `nohup cmd &` does not give you: it must **survive the tool call**, it must be **bounded** (wall-clock and memory), it must **leave a handle** (a PID file), and it must be **stoppable without orphaning children**. This card is the launcher and the stop recipe, plus the traps that three experimenter casts in one campaign (fluxonium-cz, 2026-11) re-derived by hand five times. + +> **Status: draft, partially verified.** The launcher and stop scripts below were run on a Linux box with a stand-in child (details under *What was verified*). Behaviour of a real Julia/Python child under `timeout` is observed in the campaign but its mechanism is **not** established — see *Unverified*. + +## Gate first, this card second + +Production solves go through the run gate (`amico run`, see the `solve` card if your session has it): the gate validates the spec, checks the tier, and writes the run record that verification and catalog ingest require. This card is for the work the gate does not cover — and **a result from a run launched this way is UNTRUSTED** until something independent has re-derived it (a lab-frame or full-model re-rollout, a pinned verifier). Never bank or report its numbers as verified on the solver's own say-so. + +Reasons a run legitimately lands here (all observed in the fluxonium-cz campaign, 2026-11; re-check against your `amico` version before relying on them): + +- The lane is **Python** (Floquet scans, eigensolver sweeps, Modal fan-outs): `amico run` was Julia-only. +- The lane is a **reduced-space / custom-driver arm** with no gate-supported route. +- `amico run` **stripped custom environment variables** (campaign `B1_*` variables vanished; `JULIA_NUM_THREADS` passed), required `--julia ` (bare `julia` not found), and **killing the `amico run` parent left the Julia child running**. + +If the gate *can* run it, use the gate. + +## The launcher + +Save as `launch.sh`; it writes `runs/.{log,pid,start}` under the current directory. + +```bash +#!/bin/bash +# launch.sh TAG WALL_S MIN_AVAIL_GIB -- CMD [ARGS...] +set -euo pipefail +tag=$1; wall=$2; need=$3; shift 4 # drops the literal "--" +mkdir -p runs +avail=$(awk '/MemAvailable/ {printf "%d", $2/1048576}' /proc/meminfo) # GiB; NOT the `free` column +[ "$avail" -ge "$need" ] || { echo "refuse: MemAvailable ${avail} GiB < ${need} GiB" >&2; exit 2; } +[ ! -e runs/$tag.pid ] || { echo "refuse: runs/$tag.pid exists (stop or remove it first)" >&2; exit 2; } +# setsid: new session, survives the caller. bash writes its own PID and then `exec`s, so the recorded PID is the +# final `timeout` process and (as session leader) its PGID == PID. `-k 60`: SIGKILL if SIGTERM is ignored. +setsid nohup bash -c 'echo $$ > "$0"; exec timeout -k 60 "$1" nice -n 10 "${@:2}"' \ + runs/$tag.pid "$wall" "$@" > runs/$tag.log 2>&1 < /dev/null & +for _ in 1 2 3 4 5 6 7 8 9 10; do [ -s runs/$tag.pid ] && break; sleep 0.1; done +pid=$(cat runs/$tag.pid) +awk '{print $22}' /proc/$pid/stat > runs/$tag.start # start time: guards against PID reuse at stop time +echo "launched $tag pid=$pid avail=${avail}GiB log=runs/$tag.log load=$(cut -d' ' -f1 /proc/loadavg)" +``` + +Typical use: `./launch.sh h2c_far1 5400 24 -- env OMP_NUM_THREADS=1 JULIA_NUM_THREADS=1 julia --project=. arm.jl` — wall-clock 90 min, refuse below 24 GiB available. + +Design choices, each of which was a real failure: + +- **`setsid` + `nohup` + `< /dev/null`** — a plain `nohup … &` is reaped when a sandboxed tool call returns (see `debugging`, *Background process death*); a new session survives it. +- **The PID is recorded from inside the new session** (`echo $$` then `exec`), not from `$!`. `setsid` forks instead of exec-ing when its caller is a process-group leader (an interactive shell with job control), and then `$!` names a process that has already exited. +- **`timeout -k 60`** — without `-k`, `timeout` sends SIGTERM once and then waits forever on a child that ignores it. +- **`nice -n 10`** — the campaign ran several lanes on one shared box; the launcher yields to interactive work. +- **Refuse, don't warn, on the memory gate and on an existing PID file** — two launches racing on one tag is how a run gets two writers on one log. + +Set per-process thread counts yourself when several runs share a box (`OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 JULIA_NUM_THREADS=`); the launcher does not guess. + +## The stop recipe + +Save as `stop.sh`. It kills **the process group named by a PID file** and nothing else. + +```bash +#!/bin/bash +# stop.sh TAG [GRACE_S] +tag=$1; grace=${2:-20}; pid=$(cat runs/$tag.pid) +[ "$(awk '{print $22}' /proc/$pid/stat 2>/dev/null)" = "$(cat runs/$tag.start)" ] || { echo "$tag: not running (or PID reused)"; exit 0; } +kill -TERM -- -$pid +for _ in $(seq "$grace"); do kill -0 -- -$pid 2>/dev/null || { echo "$tag: stopped"; exit 0; }; sleep 1; done +kill -KILL -- -$pid; echo "$tag: killed" +``` + +Rules that go with it: + +- **Kill by recorded PID or PGID only. Never `pkill -f`, `killall `, or a `pgrep`-built kill list.** A shared box runs other people's and other lanes' `julia`/`python`/`timeout` processes; a name match will take them with yours. Record what you launched, kill what you recorded. +- **Kill the group (`kill -- -PID`), not the single PID.** SIGKILL to the `timeout` PID alone leaves its child running, reparented to init (verified below). A group signal also reaches grandchildren (MPI-style workers, `xargs` children, solver subprocesses). +- **Check the start-time stamp before signalling a PID from an old file** — PIDs are reused. The script above does. +- After a stop, **confirm nothing survived** by listing the group (`ps -o pid,pgid,cmd -g ` returns nothing when the group is gone) rather than by name. + +## Memory: read MemAvailable, not the `free` column + +`free -g` prints `free` and `available` side by side. The `free` column excludes reclaimable page cache; on a box that has been running scans it can be a fraction of what is really usable. Briefs that say "`free -g` ≥ 20 GB" are ambiguous, and in the campaign they were misread both ways: + +| cast | `free` column | `available` (= MemAvailable) | +|---|---|---| +| H2c | 19 GB (launched 1 GB under the literal bar) | 26 GB | +| D2a | 4–12 GB at every launch | 23–34 GB | +| B8 | 17–20 GB | 26–33 GB | + +Gate on `MemAvailable` from `/proc/meminfo` (the launcher does). Two companions to that gate: budget **peak RSS × the number of concurrent runs**, not one run (one campaign solver measured 34–40 GB resident, so only one could run at a time); and re-check the gate before each launch in a batch, because the first launch's footprint is not yet visible when the second checks. + +## Observability and honest timings + +- **Stdout redirected to a file is block-buffered** in Julia and in Python: no progress shows until a flush or exit. Flush explicitly in the script (`flush(stdout)`, `print(..., flush=True)`) or the log looks dead while the run is healthy. +- **Timings under load are load-dependent.** One campaign's ledger quoted "≈5 s per point" for an eigensolver step; the production path later measured 36–62 s per point at box load 23–57 on 24 cores. When you record a timing, record the load average beside it (the launcher prints it at launch) and call the number a load-conditioned measurement, not a constant. +- If your session has the `live-solve` skill, register the run on the live widget at launch — runs launched outside the gate are exactly what it mirrors from `runs/.log`. + +## What was verified (and how) + +On a Linux box (procps/coreutils `timeout`), with a stand-in child that ignores SIGTERM (`bash -c 'trap "" TERM; exec sleep 60'`) in place of a real solver: + +1. The launcher records a PID equal to the `timeout` process's PGID and SID; the child shares the group. +2. `kill -9 ` of the recorded `timeout` process alone **left the child running** under init (reproduces the campaign's "killing the wrapper leaves the Julia child alive" report). Tested on the launcher above and on the campaign's original `setsid nohup timeout … nice …` pattern. +3. `timeout` **without `-k`** left both `timeout` and a SIGTERM-ignoring child alive well past its 2 s limit (original pattern, checked at 4 s). +4. With the launcher's `-k 60` (wall-clock 2 s, SIGTERM-ignoring child): both processes still alive at 10 s, both gone at 65 s. `stop.sh` with a 3 s grace reported "killed" and left no survivor; a group SIGKILL cleared both immediately. +5. The memory gate refused an impossible request (exit 2); a second launch on an existing PID file refused (exit 2); `stop.sh` on a finished run reported "not running". +6. The launched run survived the exit of the shell that started it. + +## Unverified + +- **Why the real Julia child outlived `timeout`'s SIGTERM** in the campaign (H2c: killed by hand ~200 s after the limit; B11: the child outlived the SIGTERM by about 60 s; a later cast recommended `timeout -k 60`). The stand-in reproduces the *effect* of a child that ignores SIGTERM; whether Julia was blocked inside a long native call or something else is not established. `-k` plus the group kill is the robust answer either way. +- The launcher assumes Linux `/proc` and GNU `timeout`. On macOS there is no `/proc/meminfo` (use `vm_stat`) and no `setsid` by default; adapt before use. +- `amico run` behaviours in *Gate first* are the campaign's observations at the time, not a current spec. + +## Provenance + +fluxonium-cz research campaign (2026-11-05 … 11-07): friction notes F-fcz-07, 08, 09, 15 and proposal S-fcz-02, drawn from the B1, B6, B7, B8, H2c, D2a, G1 and B11 observation notes (launchers hand-copied five-plus times: setsid / nohup / nice / timeout / PID file / MemAvailable gate). The recipe's behavioural claims are the ones listed under *What was verified*; everything in *Unverified* is labelled as such. + +## Related + +- `debugging` — *Background process death at harness boundaries* (why `setsid`). +- `prereg-seal` — seal the predictions *before* the scan this card launches. +- `live-solve` — put the run on the home widget (if available in your session).