diff --git a/complete/2026/08/test-performance-board.md b/complete/2026/08/test-performance-board.md new file mode 100644 index 00000000..15e13899 --- /dev/null +++ b/complete/2026/08/test-performance-board.md @@ -0,0 +1,186 @@ +# The test-performance board — the organism always knows what makes testing slow + +PyAutoHeart#164 → `d8c125e` and PyAutoBrain#261 → `1b79bb3`, closing +PyAutoHeart#163, both merged 2026-08-24 on branch +`claude/test-performance-dashboard-y3fdy7`. Design record: +[`docs/pyautoheart/test_performance_board_assessment.md`](../../../docs/pyautoheart/test_performance_board_assessment.md) +— the deep-research pass that preceded the code, with receipts into the board +family, the kill-timer arc, the NO_RUN policy history and the Actions API. + +The want: development speed tracks test speed, and knowing what is slow took a +manual archaeology session over CI job logs each time. The Heart board now +carries a standing **⏱ performance surface** — run times of the PR smoke gates, +unit-test gates, Workspace Smoke and Nightly Release, with history to flag +drops, kill-timer/hang events, and the NO_RUN census — every actionable row +carrying its own ready-to-paste `/bug` prompt. The Brain board renders the +headline verbatim. Advisory only: the readiness verdict and `badge.json` shape +are untouched. + +## What shipped + +- **`heart/checks/ci_timing.{sh,py}`** (cloud-safe, API-only, the `ci_status` + shape): per polled repo, one `actions/runs?per_page=50` fetch — deliberately + NO branch filter and NO `exclude_pull_requests`, because the + contributor-facing number is what someone waits on before a merge. Durations + from `run_started_at`→`updated_at`, success-only medians per tracked gate + (the group's `required_workflows` + `performance.extra_workflows`), queue + delay, conclusion mix. Cancelled runs disambiguated: superseded-by-a-newer + same-branch PR run is benign; cancelled on main or successorless is a + suspect hang event; `timed_out` always an event. +- **Self-carrying history**: the aggregate pass re-reads the previously + published `board.json`, appends today's p50 per gate, dedupes by date, caps + at 30 — no commits, idempotent per date; a publish gap costs a sparkline, + never a render (the Brain board's `updated_history` trick, generalised). +- **`heart/checks/no_run_census.{sh,py}`**: contents-API census of every + workspace's `config/build/no_run.yaml` — SLOW / NEEDS_FIX / permanent tiers + with marker dates and reasons; SLOW rows whose reason carries no real + seconds figure are flagged **unmeasured** (the 2026-08-23 audit's "a SLOW + marker is not evidence of slowness" made this a first-class distinction); + a missing file records `present: false` honestly. +- **`heart/dashboard.py`**: "CI wall-clock" and "NO_RUN census" sections + (events → FAIL, drift → WARN, both advisory) and the additive `performance` + block in `board.json` — the `blockers` contract extended to timings. +- **Drift thresholds** (`config/repos.yaml thresholds.ci_timing`): warn only + when ratio ≥1.5 AND ≥120s absolute — the profiling conductor's + both-gates doctrine, imported wholesale so the alarm never cries wolf. +- **PyAutoBrain `board/_board.py`**: one Heart `board.json` read now serves + blockers and performance; the `⏱ Test performance` section renders flagged + rows' prompts verbatim; hang events count as attention, never blocking. +- Both checks run in `heart-health.yml`'s daily cloud step — deliberately not + the <30s tick (internals rule 3). +- Tests: Heart 573 (+73), Brain 447 (+5); stdlib+PyYAML, hermetic, fake names. + +## Key traps / findings + +- **The tenant firewall fired exactly as predicted.** Every board that shipped + hit it, and so did this one: a real library repo name in the new + `tests/test_ci_timing.py` (an "instance fact in a new file") reddened the + first CI run; `LibA` serves. Budget for it in any organ-code PR that adds + files. +- **A shallow `--depth 1` clone push looks unpushed.** The single-branch fetch + refspec means `git push -u` succeeds but the local branch shows no upstream + and no remote-tracking ref — `git ls-remote` proved the remote had the + commit all along. Widen `remote.origin.fetch` before trusting `git status` + on such clones. +- **Absent ≠ empty on the machine surface.** The `performance` block is + omitted entirely when neither slice was observed — "not measured" must stay + distinguishable from "measured, nothing there" (the hygiene + `unscanned ≠ clean` invariant, applied to a board block). The Brain consumer + renders no section and no degraded row for an absent block. +- **The measured-SLOW heuristic needs a cap guard.** "flakes at the 1800s cap" + contains a seconds figure but is not a measurement; the regex refuses a + seconds figure immediately followed by "cap". +- **`data-copy`/`data-cmd` attributes vs html self-containment tests**: prompt + payloads legitimately embed URLs, so self-containment assertions must strip + those attributes first — third board in a row to note it; it is in the + assessment's caution list. + +## Follow-ups (tracked, not started here) + +- Phase 0 (prerequisite for per-script trends): + `draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md` + — still unissued. +- Phase 2: per-script `smoke_timings.json` from the delegated PyAutoHands + runner (one change, ten repos inherit) — answers item 4 of + `draft/research/ci/smoke_timing_and_profiling.md` with "yes". +- First `heart-health.yml` run seeds the history; sparklines are meaningful + from ~day 3. If the run flags Pages enablement ("Resource not accessible by + integration"), create the site once with + `gh api -X POST repos//PyAutoHeart/pages -f build_type=workflow`. +- Cloud session: no task worktree existed; the dev-box worktree survey has + nothing to release. + +## Original prompt + +# Test-performance section on the Heart board — run times, hangs, NO_RUN, with one-tap fix prompts + +Type: feature +Target: pyautoheart +Repos: +- @PyAutoHeart +- @PyAutoHands +- @PyAutoBrain +Difficulty: large +Autonomy: supervised +Priority: high +Status: formalised +Filed: 2026-08-24 +Issued: 2026-08-24 + +Development speed tracks test speed, and today knowing what is slow takes a +manual archaeology session over CI job logs. Build the standing surface: a +**⏱ Performance section on the Heart board** that always shows the run times +of the testing/integration infrastructure — PR smoke gates (`*_workspace_test`, +normal workspaces, HowTo), unit-test gates, weekly `workspace-smoke`, the +nightly release driver, import time — with enough history to flag drops, +kill-timer/hang events surfaced, the NO_RUN census with reasons, and a one-tap +Claude prompt on every row so "speed this up" is a paste. + +Full design rationale, data-plane analysis, chip payloads, and the trap list: +[`docs/pyautoheart/test_performance_board_assessment.md`](../docs/pyautoheart/test_performance_board_assessment.md). +Read it before starting — it cites the mechanisms (kill timer, no_run tiers, +board contracts, Actions API fields) with receipts. + +## Shape (from the assessment) + +- **Ownership**: Heart measures and publishes ("measurement lives in Heart"); + rows land in the Heart's `board.json` (schema v2, additive `performance` + block), each carrying its **own** prompt string; the Brain board renders a + headline row consuming it verbatim — the exact `heart_blockers` contract. + Advisory only: the GREEN/STALE/YELLOW/RED verdict is untouched. +- **Plane A (this task's core)**: scrape completed workflow runs of the + tracked gates via the Actions API with the default `GITHUB_TOKEN` + (production precedent: `PyAutoMind/.github/workflows/morning_health.yml`). + Duration = `updated_at − run_started_at` (never `created_at`), queue delay, + conclusion mix, per-job fanout only for runs worth the detail. Tracked + `repo:workflow` list is declared config in `PyAutoHeart/config/repos.yaml`, + never code (tenant firewall). +- **History**: per-gate daily aggregates via the self-carrying published + artifact roll-forward (the Brain board's `updated_history`/`sparkline` + pattern, 30-day cap); durable per-script baselines stay Heart's tracked + legs (phase 0 below). +- **Kill-timer/hang rows**: per-script `TIMEOUT (s)` events; job-level + `cancelled` disambiguated (≈`timeout-minutes` or no successor → kill/hang, + red; superseded PR run → benign; `cancelled` on `main` → always red); + missing `=== Smoke test summary ===` line → aborted run, coverage + discarded. Coverage counts render beside every duration. +- **NO_RUN census**: parse each workspace's `config/build/no_run.yaml`; + SLOW/NEEDS_FIX rows with marker age and reason, chips that say re-measure + first ("a marker is a claim with a timestamp, not a fact"); flag entries + whose reason carries no measurement and matchers matching zero files; + untagged permanent skips are a collapsed count. + +## Phasing + +0. **Prerequisite, separate PR** — issue + `draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md` + (filed 2026-08-04, never issued): rename-aware slugs, real 7-run windows, + run-id provenance. Plane C reads what it fixes. +1. **This task** — the Heart board section + `board.json` `performance` block + (plane A + NO_RUN census + kill-timer rows + history/sparklines), and the + Brain board headline row (can ride + `draft/feature/pyautobrain/brain_board_follow_ups.md`). +2. **Follow-up, PyAutoHands** — per-script standing dataset: one change in + the delegated runner (`autohands/run_python.py` report machinery) emits + `smoke_timings.json` per gate run for all ten repos; the board ingests it + and gains per-script rows with STALL/SLOW verdicts (reuse `retime.py`'s + vocabulary; bimodality is first-class). This answers item 4 of + `draft/research/ci/smoke_timing_and_profiling.md` with "yes". + +Regression flagging reuses the profiling conductor's drift doctrine verbatim: +≥2.0× the pin AND ≥1.0s absolute, sticky pins, explicit re-pin. Comparability +key includes runner image × Python leg × event type. + +## Acceptance + +- The Heart board page shows, per tracked gate: latest duration, p50/max over + the window, trend sparkline, conclusion mix, coverage count — refreshed on + the Heart's existing render cadence with no commit noise. +- A hang/kill event (per-script TIMEOUT, unexplained `cancelled`, aborted + summary) appears within one render, red, with a `/bug … — ` chip. +- Every SLOW/NEEDS_FIX `no_run.yaml` entry is listed with marker age, reason, + and a re-measure-first fix chip; unmeasured SLOW markers are visibly + flagged. +- The Brain board carries the headline row consuming the published block + verbatim. +- Nothing gates: readiness verdict and `badge.json` message shape unchanged. diff --git a/complete/index.md b/complete/index.md index d07e2c98..e5032cee 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1094 records across 7 buckets. +1095 records across 7 buckets. ## Highlights @@ -208,6 +208,7 @@ _(curate hard-won records here — survives regeneration.)_ - [tenth-sample-hardcoded-index](2026/08/tenth-sample-hardcoded-index.md) — results/database/aggregator tutorials hardcoded stored-sample index 9 ("the tenth - [test-mode-fit-exception-finalization](2026/08/test-mode-fit-exception-finalization.md) — `PYAUTO_TEST_MODE=1` no longer finalizes a model point whose reconstruction raises `FitException`; it substitu… - [test-mode-samples-info-hook-contract](2026/08/test-mode-samples-info-hook-contract.md) — `NonLinearSearch._test_mode_samples_info()`'s docstring told subclasses +- [test-performance-board](2026/08/test-performance-board.md) - [transformed-message-semantics-doc](2026/08/transformed-message-semantics-doc.md) - [undo-community-file-declutter](2026/08/undo-community-file-declutter.md) — Undid the 2026-08-19 community-file declutter (#248 Mind, #32 Memory): - [uniform-prior-bounds-numpy-path](2026/08/uniform-prior-bounds-numpy-path.md) — auto-closed by the merge's `Closes` line diff --git a/docs/pyautoheart/test_performance_board_assessment.md b/docs/pyautoheart/test_performance_board_assessment.md new file mode 100644 index 00000000..951dba38 --- /dev/null +++ b/docs/pyautoheart/test_performance_board_assessment.md @@ -0,0 +1,281 @@ +# The test-performance board — scoping assessment + +Date: 2026-08-24. The deep-research write-up for "the speed of AI development +is heavily dependent on the speed of tests": a dashboard that always shows the +run times of the testing/integration infrastructure (PR smoke gates on the +`*_workspace_test` and normal workspaces, HowTo, unit tests, import time), with +history to flag performance drops, kill-timer/hang events surfaced, NO_RUN +scripts listed with their reasons, and one-tap Claude prompts on every row so +"speed this one up" is a paste, not an archaeology session. Actionable prompt: +`draft/feature/pyautoheart/test_performance_board.md`. + +## The problem, with receipts + +- Manual timing archaeology is the current tool. The jax_grad budget work + records that "the diagnosis had to be rebuilt from CI job logs by hand" + (`complete/2026/08/jax-grad-smoke-timeout-budget.md`), and the 2026-08-23 + slow-vs-stall audit hand-scraped `[PASS] — s` lines per run. +- The cost of not watching is measured: four `autogalaxy_workspace_test` runs + burned ~24h of runner time at the 6-hour Actions ceiling before the kill + timer existed (`draft/bug/ci/jax_vmap_jit_compile_stall.md`); the + autolens_workspace_test PR gate is ~11m20s wall-clock of which 553s is + scripts, at ~17 runs/week (`draft/test/workspaces/slowest_smoke_gate_scripts.md`, + `draft/test/pyautoheart/smoke_relevance_gate.md`). +- The existing markers are untrustworthy: "**a SLOW marker is not evidence of + slowness**" — the first SLOW-marked entry ever measured was wrong by ~50×, + and every 2026-07-14 marker records no timing at all + (`complete/2026/08/jax-compile-stall-slow-vs-stall-audit.md`). +- The tracked signal is broken: Heart's `script_timing` baselines are orphaned + by path-derived slugs (no history accumulated for the moved jax_grad scripts + since 2026-07-24) and every stored history is one value repeated seven times + (`draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md` + — filed 2026-08-04, **never issued**). + +## What is recommended + +**A "⏱ Performance" section on the Heart board, published in the Heart's +`board.json` (schema v2, additive) with every row carrying its own one-tap +prompt — and a headline row on the Brain board consuming it verbatim, exactly +the way `heart_blockers` already works.** Not a seventh board. + +Why Heart, not Brain: + +- The doctrine is explicit and repeated: "**measurement lives in Heart; + hygiene acts**" and "new standing signals (import cost, CLI noise) become + Heart *legs*, not a new repo" + (`PyAutoBrain/agents/conductors/hygiene/AGENTS.md:143-149`). Heart already + owns every timing signal in the organism: `script_timing`, `test_run`, + `import_time`, `unit_test_timing`, `workspace_testmode_timing`. +- The consumption seam already exists and is tested: the Heart board publishes + structured blockers (`{text, severity, repo, repo_url, run_url, prompt}`) + and the Brain board renders them verbatim, never re-deriving the prompt + (`PyAutoBrain/board/_board.py:228-249`, `tests/test_board.py:43-53`). + A `performance` block rides the same contract; schema v2 is additive by + design (`complete/2026/08/actionable-health-board.md`). +- The cadence already lines up: Heart renders at 05:00 UTC, the Brain board + reads it at 05:30 (`PyAutoBrain/.github/workflows/brain_board.yml:20-25`). +- Timing rows are **advisory, never gating** — the precedent is the + `import_time` leg ("advisory dashboard section, NOT in the readiness gating + set", `complete/2026/07/import-time-heart-leg.md`). The four-tier + GREEN/STALE/YELLOW/RED verdict is untouched. + +Why not a Brain-board section (considered): the Brain board holds the exact +code to copy — `gh_json` (`_board.py:136-150`), the overnight scrape +(`:167-211`), the self-carrying history (`:456-466`) and `sparkline()` — but +putting the *measurement* there splits ownership against the doctrine, and the +Brain "owns no state, no health checks". The Brain board's role is the morning +headline: "N gates slowed / 1 hang flagged", chip → the Heart page. That row +can ride `draft/feature/pyautobrain/brain_board_follow_ups.md`. + +## How the timing information gets calculated and populated + +Three data planes, in cost order. "Just scrape recently completed PRs" — the +suggestion that seeded this scoping — is plane A, it works today, and it is +already proven in production. + +### Plane A — workflow wall-clock, scraped from the Actions API (ships first) + +The default `GITHUB_TOKEN` reads other public PyAutoLabs repos' workflow runs: +`PyAutoMind/.github/workflows/morning_health.yml:68-101` already does exactly +this daily against PyAutoHeart/PyAutoBrain/PyAutoHands/PyAutoFit with +`permissions: contents: read` and no PAT. Per run the API serves everything +the board needs: + +- **duration** = `updated_at − run_started_at` (also served directly as + `run_duration_ms` by `GET /actions/runs/{id}/timing`; ignore the `billable` + fields — they are 0 on public repos). **Trap, already recorded by the Hands + board: use `run_started_at`, never `created_at`** — re-attempted runs + otherwise report multi-day durations (`complete/2026/08/release-board.md`). +- **queue delay** = `run_started_at − created_at`. +- **conclusion** (`success | failure | cancelled | …`) — see the kill-timer + section for disambiguating `cancelled`. +- **PR association**: `event: pull_request` + `head_branch` — so "what a + contributor waits on a PR" is directly measurable, per gate, per week. +- **per-job and per-step timings** via `GET /actions/runs/{id}/jobs` — matrix + leg identity is in the job `name` (`"pytest (3.12)"`), and the *Run smoke + tests* step is separable from checkout/install overhead. Fan out to `/jobs` + only for runs worth the detail (the slowest of the day, every `cancelled` + one) — that keeps the request budget at ~1 call per tracked workflow per + render, trivially inside the 1000 req/hr limit. + +The tracked-workflow list is declared config, not code (the tenant-firewall +rule): a `performance:` block in `PyAutoHeart/config/repos.yaml` naming each +`repo:workflow` pair — every workspace's `Smoke Tests` caller, the HowTo +gates, the organ self-test gates (`tests.yml`), Heart's weekly +`workspace-smoke.yml`, `release-integrate.yml`, and the Brain's +`nightly-release.yml`. One channel note: since #122 the validation channels +are split so runs attribute to the **caller** workflow +(`complete/2026/07/split-validation-channels.md`) — query the callers. + +What plane A yields per gate: last-N durations, p50/max, conclusion mix, +queue delay, and the two contributor-facing numbers that matter — median PR +gate latency and its trend. + +### Plane B — per-script timings, promoted from log lines to a standing dataset + +The runner already prints `[PASS] — s` per entry and a +`=== Smoke test summary ===` terminal line; today that data evaporates into +job logs (retained ~90 days) and is only recovered by hand-scraping. The +smoke-runner delegation (`complete/2026/08/smoke-runner-delegation.md`) turned +all ten workspace runners into thin shims over `autohands/run_python.py` — +**so per-script timing recording is now one PyAutoHands change, not ten +repo sweeps**: extend the existing report machinery (`--report-dir` is +already load-bearing — "without it the gate runs to completion and always +exits 0") to emit a `smoke_timings.json` (entry, status, seconds, cap in +force, exit code), uploaded as a run artifact and/or written to +`$GITHUB_STEP_SUMMARY`. The Heart render fetches the latest artifact per gate. + +This is precisely the open question item 4 of +`draft/research/ci/smoke_timing_and_profiling.md` already poses ("should the +runner record per-script timings routinely, so this is a standing dataset +rather than a periodic archaeology exercise?") — this assessment's answer is +**yes, via the delegated runner**. The `retime.yml` classifier and its +verdict vocabulary (STALL/SLOW/NEITHER/AMBIGUOUS/ERROR, `retime_results.json`) +stay as the on-demand deep probe, reached through `smoke-tests.yml`'s +`runner`/`runner-args`/`script-timeout` inputs. + +**STALL ≠ SLOW is a first-class dimension, not a footnote.** "A slow script +has a tight timing distribution. A stalling one is bimodal" — a healthy +compile of `rectangular_mge.py` is 3.1s; a stalled one exceeds 300s, same +commit, same runner image (`jax-compile-stall-slow-vs-stall-audit.md`). A +single wall-clock number per entry cannot express this; the board should carry +the retime verdict where one exists, and flag bimodality from the standing +dataset where it doesn't. + +### Plane C — baselines and regression flagging (the "history" that must be durable) + +Two history mechanisms exist in the board family, with different guarantees: + +1. **Self-carrying published artifact** — the Brain board fetches its own + previous `board.json` at render, appends today, caps at 30 entries + (`PyAutoBrain/board/_board.py:456-466`). Free, no commits, idempotent per + date — but lossy (a publish gap loses everything) and 30 days max. Right + for the plane-A per-gate daily aggregates and their sparklines. +2. **Heart's tracked rolling baselines** (`~/.pyauto-heart/`, the + `script_timing` mechanism) — durable, but currently broken as filed. Right + for per-script baselines once fixed. + +The fix is a prerequisite, and its prompt is already written: +`draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md` +(rename-aware slugs or a loud no-baseline signal; real 7-run accumulation; +record the source run id per duration). **Issue it first.** + +For flagging drops, do not invent thresholds — reuse the profiling +conductor's considered doctrine: a regression counts only if it is newer than +its pin, **at least 2.0× the pinned value AND at least 1.0s above it in +absolute terms**, with sticky pins that never move without an explicit +`--repin` ("host load alone has produced 7× errors in this corpus and an +alarm that cries wolf gets ignored" — +`PyAutoBrain/agents/conductors/profiling/AGENTS.md`). The comparability-key +lesson from the compile-warm dashboard transfers too: never pool different +hosts under one label — for CI the key is runner image × Python leg × event +type (`complete/2026/08/compile-warm-baseline-dashboard.md`). + +Unit tests and import time ride the same planes: the organ/library `tests.yml` +gates are plane-A rows (they are seconds-to-a-minute today — the board's job +is to notice when that stops being true), and the existing `import_time` leg +(advisory, off-tick, subprocess-measured) surfaces its red/yellow counts as a +row with a `/hygiene` chip — the standing leg promotion that +`hygiene/AGENTS.md:117-119` already names as the deferred follow-up. + +## Kill-timer and hang events on the board + +The kill timer (PyAutoHands `build_util.timeout_for` + `kill_group`; 300s +smoke default, 900s `jax_grad/`, 1800s release; exit 124; `TIMEOUT` status +with the truncated output tail and the cap in force attached) and the +in-process watchdog (faulthandler dump at 80% of `BUILD_SCRIPT_TIMEOUT`, +heartbeat lines) already leave ingestible traces. The board renders, per gate: + +- **Per-script TIMEOUT rows** (plane B): entry, cap, count over the window, + and the retime verdict if one exists. Chip: + `/bug kill timer: