diff --git a/dashboard.html b/dashboard.html index f80e3301..c2f3edd7 100644 --- a/dashboard.html +++ b/dashboard.html @@ -51,7 +51,7 @@

๐Ÿ“‹ PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's ๐Ÿ“‹ and its /start_dev command is on your clipboard โ€” paste it into a Claude Code chat to route Claude straight to that task. Recent is the same work by date โ€” what has been happening rather than what to do next.

-

In flight 1 ยท Parked 3 ยท Planned 6 ยท Backlog 150 ยท markdown version

+

In flight 1 ยท Parked 3 ยท Planned 6 ยท Backlog 152 ยท markdown version

Start here

Highest priority (filed as high) โ€” showing 12 of 17

TRIAGE: needs manual review before routing โ€” medium ยท safe ยท high

@@ -90,7 +90,7 @@

Planned

latent-nan-guard-honest-run โ€” planned 2026-07-22

Backlog markdown version

-

150 filed prompts, not started โ€” sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

+

152 filed prompts, not started โ€” sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

bug โ€” 34

Fix release JAX runtime compatibility and likelihood parity โ€” health_fixes ยท too-large ยท supervised ยท high

@@ -229,9 +229,11 @@

Backlog

Deduplicate repos_sync.py's check/write pairs โ€” pyautomind ยท medium ยท safe ยท low

-test โ€” 3 +test โ€” 5

Re-baseline the MGE imaging JIT profiling regression value โ€” autolens_workspace_developer ยท too-large ยท supervised ยท high

Restore absolute NumPy likelihood regression baselines in the _workspace_test โ€” workspaces ยท too-large ยท supervised ยท high

+

Relevance-gate the reusable smoke workflow so a PR only runs โ€” pyautoheart ยท medium ยท supervised ยท normal

+

Speed up the three slowest autolens_workspace_test smoke-gate scripts โ€” workspaces ยท medium ยท supervised ยท normal

The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced โ€” workspaces ยท too-large ยท supervised ยท normal

diff --git a/dashboard.md b/dashboard.md index c123c5d6..4fc03c1e 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 150 | +| [Backlog](#backlog) (`draft/`) | 152 | ## Start here @@ -235,7 +235,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**150** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below. +**152** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below.
bug โ€” 34 @@ -1220,7 +1220,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
-test โ€” 3 +test โ€” 5
๐Ÿ“‹ Re-baseline the MGE imaging JIT profiling regression value โ€” autolens_workspace_developer ยท too-large ยท supervised ยท high @@ -1238,6 +1238,22 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
+
๐Ÿ“‹ Relevance-gate the reusable smoke workflow so a PR only runs โ€” pyautoheart ยท medium ยท supervised ยท normal + +``` +/start_dev draft/test/pyautoheart/smoke_relevance_gate.md +``` + +
+ +
๐Ÿ“‹ Speed up the three slowest autolens_workspace_test smoke-gate scripts โ€” workspaces ยท medium ยท supervised ยท normal + +``` +/start_dev draft/test/workspaces/slowest_smoke_gate_scripts.md +``` + +
+
๐Ÿ“‹ The new workspace smoke-test GitHub Actions (added via feature/smoke-test-ci) surfaced โ€” workspaces ยท too-large ยท supervised ยท normal ``` diff --git a/draft/test/pyautoheart/smoke_relevance_gate.md b/draft/test/pyautoheart/smoke_relevance_gate.md new file mode 100644 index 00000000..781443c1 --- /dev/null +++ b/draft/test/pyautoheart/smoke_relevance_gate.md @@ -0,0 +1,130 @@ +# Relevance-gate the reusable smoke workflow so a PR only runs what its diff can affect + +Type: test +Target: PyAutoHeart +Repos: +- PyAutoHeart +- autolens_workspace_test +- autogalaxy_workspace_test +- autolens_workspace +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: formalised + +`PyAutoHeart/.github/workflows/smoke-tests.yml` โ€” the reusable workflow every +workspace's `smoke_tests.yml` is a thin caller of โ€” has exactly one skip +condition: a docs-only gate that skips the matrix when every changed file +matches `*.md`, `docs/`, `LICENSE` or `runtime.txt`. Anything else runs the +full curated smoke list, whatever it touched. + +This prompt cuts how *often* the gate runs. Making the slow entries cheaper is +the sibling prompt `draft/test/workspaces/slowest_smoke_gate_scripts.md`; they +compound but are separate changes in separate repos. + +## Why + +- `autolens_workspace_test` smoke costs ~11m20s and fires on every PR event + plus every push to `main`. Run numbers put it at roughly 17 runs/week + (#555 on 2026-07-28 โ†’ #616 on 2026-08-22; approximate, since superseded PR + runs are cancelled). +- `autolens_workspace` costs ~8m40s over 37 entries, 5.8 runs/day in the week + to 2026-08-23. +- The failure mode is already recorded in the repo. `smoke_tests.txt`, on the + 2026-08-22 disable of `multi_dataset/jax_likelihood/mge.py`: *"Hit 4/4 jobs + (3.12 and 3.13, twice each) on autolens_workspace_test#261, whose diff + touches no script this gate runs."* +- The two extremes already exist and are 80x apart: a docs-only PR finishes in + **8 seconds** (autolens_workspace_test run #605, autogalaxy_workspace #545); + a one-character change to any non-markdown file costs the full run. + +## Task + +Two tiers, shippable independently. Tier 1 alone is most of the win. + +1. **Skip when no script can be affected.** Extend the existing `changes` job's + classification: if the diff touches nothing under `scripts/`, `config/`, + `smoke_tests.txt`, `smoke_notebooks.txt` or `.github/`, skip the matrix the + same way the docs-only path does. Same fail-closed shape, same + `docs_only`-style output, one more reason to skip. +2. **Narrow the entry list to the packages the diff touches, plus a fixed + core.** The smoke entries cluster cleanly by top-level package, so a + directory-level mapping is enough โ€” no dependency analysis needed: + + | package | autolens_ws_test | autogalaxy_ws_test | autolens_workspace | + |---|---:|---:|---:| + | `imaging/` | 232.8s (42%) | 175.8s (32%) | 63.6s (18%) | + | `multi_dataset/` | โ€” | 177.1s (32%) | 13.5s (4%) | + | `misc/` | 154.6s (28%) | 37.6s (7%) | โ€” | + | `interferometer/` | 92.1s (17%) | 150.6s (27%) | 54.6s (16%) | + | `point_source/` | 47.7s (9%) | โ€” | 15.7s (4%) | + | `multi_galaxy/` | 25.8s (5%) | 11.8s (2%) | 166.4s (47%) | + | `group/` | โ€” | โ€” | 22.9s (7%) | + | `guides/` | โ€” | โ€” | 14.3s (4%) | + | `cluster/` | โ€” | 4.2s (1%) | โ€” | + + A PR touching only `scripts/misc/` would run 155s instead of 553s; one + touching only `multi_galaxy/` in `_test` would run 26s. Tier 2 needs the + selected set passed down to `run_smoke.py` as an input, so it also touches + each workspace's vendored runner โ€” scope it deliberately or defer it. + +## autogalaxy_workspace_test is the same size and has no other lever + +Measured after this prompt was first filed (run 32533004337, 2026-08-21, py3.12, +**557.1s across 37 entries** โ€” within 4s of autolens_workspace_test's 553.0s, +and again reconciling exactly to the step wall-clock). + +Unlike its autolens sibling it has **no hot spot at all**: slowest entry 40.0s, +median 13.5s, and the top three are only 21% of the run. There is nothing to +speed up, so the sibling prompt does not apply and this gate is the *only* +lever for that repo. + +What it does have is a combinatorial matrix: 25 of its 37 entries are +`{imaging, interferometer, multi_dataset}/jax_{likelihood,grad}/{lp, mge, +mge_group, rectangular, rectangular_mge, delaunay, delaunay_mge}`, i.e. the +same handful of meshes crossed with three dataset types. Tier 2's +package-level narrowing maps onto that cleanly. A representative-subset +policy for the PR gate (full cross-product weekly) is worth considering +alongside it, but is a separate decision and belongs to that repo, not here. + +Checked and NOT a factor: its six `jax_grad/` entries were the obvious +suspect, since Heart budgets that class at 1800s for running at full +resolution. Measured they are 86.4s total (15.5%), 10.8โ€“21.8s each. The +expensive jax_grad scripts are the autolens ones, and those are not in any +smoke list. + +## Hard constraints + +- **Do not implement this with `on.pull_request.paths`.** A job skipped by a + path filter never reports a conclusion, so a required status check sits + pending forever and blocks the merge. The in-workflow `changes` job emits a + real `skipped`, which satisfies required-check semantics โ€” the workflow's own + comment already says so. Keep the skip inside the workflow. +- **Fail closed**, matching the docs-only gate exactly: no base SHA, unfetchable + base, empty diff, or a single unclassifiable path โ‡’ run everything. Only an + explicit every-file-matches verdict may skip. Keep the two-dot diff against + the base *tip* so upstream drift shows up as extra files. +- **Do not touch the push-to-`main` run.** `PyAutoHeart/config/repos.yaml` lists + `workspaces_test: ["Smoke Tests"]` and `workspaces: ["Smoke Tests", + "Navigator Check"]` under `required_workflows`; `ci_status` reads their + conclusion on the `main` HEAD commit and `readiness` gates RED on failure. + `cancelled` is in Heart's `FAILURE_CONCLUSIONS`, which is why the callers' + concurrency block only cancels non-`main` refs. Narrowing PR-side runs is + free โ€” Heart never reads them โ€” but a skipped or cancelled `main` run breaks + the readiness gate. If tier 1 would skip on a `main` push, gate it to + `pull_request` events only. +- The change is Heart-owned and lands once for every caller (both `_workspace` + and `_workspace_test` families, plus the HowTo repos). Verify against at + least one caller of each shape before merging. + +## Acceptance + +- A PR touching only `scripts/misc/` in `autolens_workspace_test` runs + materially less than the full 553s (tier 2), or a PR touching only + `README.md` + a config sidecar still skips (tier 1). +- A PR with an unresolvable base still runs the full matrix. +- The `Smoke Tests` check reports `skipped`, not pending, on every skip path. +- `main`-push runs are unchanged and Heart's `ci_status` still reads a real + conclusion. + + diff --git a/draft/test/workspaces/slowest_smoke_gate_scripts.md b/draft/test/workspaces/slowest_smoke_gate_scripts.md new file mode 100644 index 00000000..7bf6e55a --- /dev/null +++ b/draft/test/workspaces/slowest_smoke_gate_scripts.md @@ -0,0 +1,144 @@ +# Speed up the three slowest autolens_workspace_test smoke-gate scripts + +Type: test +Target: workspaces +Repos: +- autolens_workspace_test +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: formalised + +The per-PR smoke gate in `autolens_workspace_test` costs ~11m20s wall-clock, of +which ~9m13s is script execution. Three entries are 42% of that. This prompt is +the *make them cheaper* half; cutting how often the gate runs at all is the +sibling prompt `draft/test/pyautoheart/smoke_relevance_gate.md`. Do not merge +the two โ€” one edits scripts, the other edits a workflow, and they land in +different repos. + +## Measured + +CI run 32605025472 (2026-08-22, `main`), py3.12 leg (the critical path; py3.13 +is ~6% faster). 23 entries, 23/23 pass, **553.0s** total โ€” which reconciles to +the step wall-clock exactly, so the runner adds no measurable overhead and the +scripts *are* the cost. + +| py3.12 | share | script | +|-------:|------:|--------| +| 120.7s | 21.8% | `imaging/subhalo_recovery.py` | +| 63.4s | 11.5% | `misc/database/scrape/general.py` | +| 47.7s | 8.6% | `point_source/jax_likelihood/point.py` | +| 33.9s | 6.1% | `imaging/jax_likelihood/rectangular.py` | +| 31.5s | 5.7% | `misc/jax_assertions/delaunay_nn.py` | +| 30.2s | 5.5% | `imaging/jax_likelihood/mge.py` | + +The remaining 17 entries are 4.5โ€“29.2s each and are not in scope. Reproduce +with `gh api` on the job log and grep the runner's `[PASS] โ€” s` +lines; the runner prints one per entry. + +Note the tail is short: after these three the curve flattens, so this prompt +can win ~4 minutes and no more. Do not chase entries below ~30s. + +## Task + +Per script, either make it materially faster or demote it โ€” both are +acceptable outcomes, and the choice is per script, not global. + +1. **`imaging/subhalo_recovery.py` (120.7s).** Already the subject of one + speed-up pass: `complete/2026/08/potential-correction-validation.md` leg 1 + recorded it at 232s/224s against the 300s cap, and it now runs at 120.7s, so + half the work is done and the *why* is documented there โ€” read it before + re-deriving. It asserts end-to-end `dkappa` recovery of a simulated 1e10 + Msun subhalo for both the one-shot and iterative engines. Ask whether the + PR gate needs both engines or whether one belongs on the weekly channel. +2. **`misc/database/scrape/general.py` (63.4s).** Un-parked on 2026-07-21 + (`chore(no_run): un-park database/scrape/general`, autolens_workspace_test#192). + A database-scrape regression is the least likely of the three to be broken + by a typical lens-modelling PR, so it is the strongest demotion candidate โ€” + check what it uniquely covers before deciding. +3. **`point_source/jax_likelihood/point.py` (47.7s).** Check whether the cost + is `PointSolver` iterations or JAX compile time; if compile-dominated, the + lever is problem size, not sample count. + +## The cap does not apply to these scripts, and cannot be made to + +Read this before proposing "just turn PYAUTO_SMALL_DATASETS on". + +`config/build/profile_smoke.yaml` has set `PYAUTO_SMALL_DATASETS: "1"` in its +defaults since the file was created (2026-04-08, then spelled +`PYAUTO_WORKSPACE_SMALL_DATASETS`; renamed 2026-04-30). It has never been +missing. What has grown is the set of entries exempted from it: + +| date | entries exempted from the cap | +|---|---| +| 2026-04-30 | 10 profile overrides | +| 2026-05-28 | 23 | +| 2026-07-23 | 27 โ€” then migrated into in-file `ENV:` declarations (PyAutoHands#187), leaving 1 override | +| today | 90 `ENV: โ€ฆ full_datasets` declarations repo-wide | + +Of the 23 live smoke entries, **16 declare `ENV: [jax] full_datasets`** and a +17th (`misc/database/scrape/general.py`) is exempted by the one surviving +profile override. That is **516.5s of the 553.0s โ€” 93% of the gate โ€” running +uncapped.** Only six entries actually run capped (3 aggregator + 2 latent at +9-pixel masks, `multi_galaxy/model_fit.py` at 80), totalling 36.5s. + +Measured mask sizes from the same CI run, against autolens_workspace's +80โ€“208 pixels: + +| entry | mask | +|---|---:| +| `imaging/subhalo_recovery.py` | 5858 px | +| `misc/database/scrape/general.py` | 2828 px | +| `multi_galaxy/jax_likelihood/lp.py` | 2828 px | +| `imaging/jax_likelihood/potential_correction.py` | 1466 px | +| `imaging/jax_likelihood/{rectangular,mge}.py` | 952 px | +| `imaging/jax_likelihood/{lp,smbh}.py` | 716 px | + +The exemptions are deliberate and load-bearing: the `jax_likelihood` scripts +assert hardcoded full-resolution likelihood literals at rtol 1e-4, so a capped +dataset changes the likelihood and the assertion fails. That failure has been +hit and recorded โ€” see `complete/2026/08/sph-transform-name-check.md` +("ran with capped datasets against full-resolution reference assertions and +failed") and `complete/2026/08/mge-sigma-min-workspace-sweep.md`. + +So the real question this prompt has to answer is not "why is the cap off" +but **"should the per-PR gate be running full-resolution parity assertions at +all?"** For each of the three scripts, the options are: re-derive the literals +at a capped size (buys the cap but re-pins every literal, and those pins have +already had to be regenerated once โ€” autolens_workspace_test#257, "repin vmap +literals halved by the PositionsLH penalty fix", which blocked the nightly +release at Stage 3 on 08-18/08-19), shrink the problem some other way, or +demote to the weekly channel that is meant for full-fidelity work. Choose per +script and say which. + +## Constraints + +- **A faster script that no longer tests anything is a regression, not a win.** + This repo has three recorded instances of exactly that failure mode: the + vacuous JAX assertions (`complete/2026/07/vacuous-jax-assertions.md`), the + NUFFT parity legs that compared nufftax against itself and reported + `max |ฮ”| = 0.0000e+00`, and `latent/latent_nan_robustness` passing vacuously + under the smoke profile (see `planned.md`). For every reduction, state what + the assertion still discriminates against and show it failing when the thing + it guards is broken. +- Coverage given up is not coverage lost. Heart's weekly `workspace-smoke.yml` + (Mondays 03:00 UTC) already runs *every* script in this repo against library + `main` under the same `profile_smoke.yaml`, and `release-integrate` re-runs + the matrix at `PYAUTO_TEST_MODE=0`, full resolution. The curated PR list is a + strict subset of both. Demotion = removing the entry from `smoke_tests.txt` + with a comment saying which channel still covers it. +- Do not raise `BUILD_SCRIPT_TIMEOUT` for these. The 300s smoke cap is a + runaway detector; three entries sitting under it is the problem, not the cap. +- Keep the entries that stay inside the cap with real margin โ€” 120.7s against + 300s already flaked into timeout historically under sweep-load contention + (`planned.md` records 252s uncontended vs a 300s timeout under load). + +## Acceptance + +- Smoke step wall-clock for the py3.12 leg drops below ~6 minutes. +- 23/23 still pass, and every touched script has a stated, demonstrated + discriminating assertion. +- Any demoted entry is commented in `smoke_tests.txt` naming the channel that + still runs it. + +