diff --git a/.github/workflows/registry_reconcile.yml b/.github/workflows/registry_reconcile.yml new file mode 100644 index 00000000..128c438f --- /dev/null +++ b/.github/workflows/registry_reconcile.yml @@ -0,0 +1,34 @@ +name: Registry Reconcile + +# The ONLINE lifecycle leg (`lifecycle.py issues`): registry entries +# cross-checked against live GitHub — a tracking issue closed, or a +# `status: pr-open` PR merged, while the entry still reads pending. That +# second case is the crashed-ship signature (2026-08-17, +# version-stamp-sync-guards: six PRs merged, the session died before the +# bookkeeping, and no offline check could see it). A red run means finished +# work is still listed as pending: run the wrap-up (record + retire + issue +# close), don't silence the job. +# +# Instance automation, NOT shipped into the fresh-slate template (spawn spec +# rule 9c): it runs on a schedule and reads sibling-repo issue/PR state +# across the org, neither of which a freshly-spawned repo can satisfy. +# lifecycle_drift.yml stays the hermetic shipped guard; this file is the +# networked companion that must live outside it. + +on: + schedule: + - cron: "23 6 * * *" # daily 06:23 UTC — offset to dodge top-of-hour load + workflow_dispatch: + +permissions: + contents: read + +jobs: + reconcile: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: registry state vs GitHub (closed issues / merged pr-open PRs still listed pending) + env: + GH_TOKEN: ${{ github.token }} + run: python3 scripts/lifecycle.py issues diff --git a/active.md b/active.md index fd9a4176..9d796e91 100644 --- a/active.md +++ b/active.md @@ -51,16 +51,3 @@ - current verdict: Heart's last committed dashboard (2026-08-11T05:51Z, i.e. BEFORE the green run) reads STALE score 65, listing `no release validation for current source` among its evidence gaps. That is the STALE tier behaving correctly — an evidence gap, not a fault, and this ingest is its remedy. - do-not: do NOT re-dispatch `Release Integrate` to "refresh" this. The run is green and its artifact is live; a re-dispatch costs ~70 minutes of CI and proves nothing new. Only re-dispatch if the artifact has expired or main has moved. - repos-none-claimed: this entry claims NO repos — deliberately on one line, NOT as 2-space ` - Repo` bullets, because `worktree_check_conflict` treats any such bullet as a live claim. -## version-stamp-sync-guards -- issue: https://github.com/PyAutoLabs/PyAutoHands/issues/235 -- prompt: active/version_stamp_sync_and_release_sed_guards.md -- session: claude --resume d73342fb-c33f-4028-8741-30cbe0c856a3 -- status: pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700) -- worktree: ~/Code/PyAutoLabs-wt/version-stamp-sync-guards -- repos: - - PyAutoNerves: feature/version-stamp-sync-guards - - PyAutoArray: feature/version-stamp-sync-guards - - PyAutoFit: feature/version-stamp-sync-guards - - PyAutoGalaxy: feature/version-stamp-sync-guards - - PyAutoLens: feature/version-stamp-sync-guards - - PyAutoHands: feature/version-stamp-sync-guards diff --git a/active/autolens_workspace_developer/imaging_delaunay_jax_profiling.md b/active/autolens_workspace_developer/imaging_delaunay_jax_profiling.md deleted file mode 100644 index 8003d3ba..00000000 --- a/active/autolens_workspace_developer/imaging_delaunay_jax_profiling.md +++ /dev/null @@ -1,89 +0,0 @@ -# JAX JIT Profiling: Imaging Delaunay (update existing script) - -## Context - -`autolens_workspace_developer/jax_profiling/imaging/delaunay.py` -already exists (≈1000 lines) and profiles a Delaunay-pixelization -source under `jax.jit` + eager + vmap. It was written earlier in -the JAX migration, before three pytree-readiness pieces landed on -`main`, so its model construction / JAX-leaf plumbing does not -necessarily reflect the current "full pytree" approach used by -`mge.py` and `pixelization.py`. - -Companion scripts that *do* use the current pytree approach: - -- `imaging/mge.py` -- `imaging/pixelization.py` (RectangularAdaptDensity) - -These two should be treated as the reference structure. - -## Pytree infrastructure (already shipped — align `delaunay.py` with it) - -Three library pieces have landed on `main` since the current -`delaunay.py` was written: - -- **PyAutoFit#1222** — `TuplePrior` registered as a JAX pytree. On - a typical Isothermal+Shear+Delaunay model this lifts live JAX-leaf - count from 3 to O(100+), so - `jax.jit(AnalysisImaging.log_likelihood)` flows through the full - model rather than freezing most of it as constants. -- **PyAutoArray#279** — Jacobi preconditioning of the NNLS - curvature matrix. -- **PyAutoArray#282** — `nnls_target_kappa=1.0e-2` config default. - -The aim of this task is to bring `imaging/delaunay.py` in line with -`mge.py` / `pixelization.py` so all three imaging profiling scripts -share a common structure and pytree-readiness baseline. - -## Task - -Update `autolens_workspace_developer/jax_profiling/imaging/delaunay.py`: - -1. **Audit the current model construction** — confirm all priors - flow as pytree leaves (not frozen Python constants) by counting - JAX leaves on the model pytree and printing the count. It should - be comparable to `mge.py` / `pixelization.py` on an equivalent - lens setup, not a small handful. -2. **JIT path parity with `mge.py`**: `jax.jit` wraps the - `Fitness.call` / `AnalysisImaging.log_likelihood` equivalent, - measures first-call (compile) time + N steady-state repeats, - reports mean / median / stdev. -3. **Eager baseline**: matching `FitImaging` figure-of-merit / - log-likelihood, asserted numerically equal to the JIT path - within `rtol`. -4. **vmap path**: batch `batch_size` parameter vectors through - `fitness._vmap`, report per-likelihood cost. -5. **Results artefact**: JSON + PNG into - `jax_profiling/imaging/results/` with the same schema as - `mge.py` / `pixelization.py`. - -## Expected output - -`delaunay.py` runs end-to-end via: - -```bash -cd jax_profiling/imaging -python delaunay.py -``` - -Producing: - -- JIT vs eager timing comparison -- Numerical-agreement assertion PASS -- vmap batch throughput measurement -- JAX leaf count printed at model-construction time, matching the - order of magnitude seen in `mge.py` / `pixelization.py` - -## Likely blockers to raise if encountered - -- If `delaunay.py` currently relies on model construction that - freezes most priors as constants, rewriting it to use the pytree - approach may expose a JAX-tracing issue in the Delaunay mapper - path (e.g. `delaunay_2d_interpolation` gradient behaviour). If - so, file a separate issue and park this one until the library - issue is addressed — do not work around it with a "partial pytree" - hack. -- Delaunay construction itself (`scipy.spatial.Delaunay`) is not - JAX-traceable. The existing script already handles this by - building the triangulation outside the JIT boundary — preserve - that boundary when refactoring. diff --git a/active/autolens_workspace_developer/visualization_profiling_cluster.md b/active/autolens_workspace_developer/visualization_profiling_cluster.md deleted file mode 100644 index 1ff8f209..00000000 --- a/active/autolens_workspace_developer/visualization_profiling_cluster.md +++ /dev/null @@ -1,115 +0,0 @@ -# Visualization Profiling: Cluster Simulator - -## Background - -The cluster-scale simulator at `autolens_workspace/scripts/cluster/simulator.py` -used to take well over five minutes end-to-end. The natural assumption was -that the `PointSolver` was the culprit — solving the lens equation is iterative -and slow in numpy. Issue #89 JAX-jitted the solver and dropped its cost to -about 22 seconds (most of which is the one-off JIT compile), but the -simulator still takes around three minutes on a warm cache. - -When I instrumented each phase, the new picture was clear: - -| Phase | Time | -|--------------------------------------|------| -| Imports + pytree registration | ~3s | -| Solver compile + first solve | ~19s | -| Solver second solve (cached) | ~2s | -| `SimulatorImaging.via_tracer_from` | ~92s | -| `aplt.subplot_imaging_dataset` | ~1s | -| `aplt.subplot_tracer` | ~51s | -| `aplt.subplot_galaxies_images` | ~2s | - -Almost all of the remaining runtime sits in two places — multi-plane image -rendering and the multi-panel `subplot_tracer` plotter. Neither is JAX-jitted, -and both scale with grid pixel count, the depth of over-sampling near galaxy -centres, and the number of internal panels rendered. This task is about -understanding those two costs well enough to make an informed call on what -(if anything) to do about them. - -## Why this matters - -Cluster-scale examples are not just one simulator. Group-scale and the -upcoming scaling-relation cluster examples will share the same multi-plane -ray-tracing path through `via_tracer_from` and the same plotter calls at the -end of every script. If we can JAX-accelerate either piece, every -cluster-style example will get the same speedup for free. If we can't, -that is itself a useful answer — it tells us the right place to push back -on the imaging grid resolution, or to swap `subplot_tracer` for a leaner -plotter inside cluster examples. - -## What's already in place - -A profiling script lives at: - -`autolens_workspace_developer/visualization_profiling/imaging/cluster.py` - -It rebuilds the cluster geometry from the simulator (2 main lens galaxies + -NFW host halo + 2 multi-plane sources at z=1.0 and z=2.0), then independently -times: - -1. Image rendering across four `imaging_grid` variants — full resolution - (1000x1000 with sub_size=[32,8,2]), half resolution, half resolution with - lighter over-sampling, and full resolution with no over-sampling. The aim - is to separate the cost of pixel count from the cost of sub-sampling. -2. `subplot_imaging_dataset` once on a fast pre-rendered dataset. -3. `subplot_tracer` and `subplot_galaxies_images` across four viz grid - resolutions (50x50 through 500x500) all spanning the same 100" field. - -It writes nothing to disk and prints a per-section table at the end. - -## Task - -Run the script as-is, look at the table, and answer: - -- **For image rendering**: of the difference between the 1000x1000-with-heavy- - over-sampling baseline and the 500x500-with-lighter-over-sampling variant, - how much is pixel count and how much is over-sampling? If over-sampling - is the bigger lever, is the cluster simulator's choice of `sub_size=32` - near each centre actually justified — i.e. would `sub_size=8` give a - visibly worse simulated image? -- **For `subplot_tracer`**: how does cost scale with viz grid resolution? It - is roughly linear in pixel count, or is there a fixed per-panel overhead - that dominates at small grids? Counting the panels and timing each in - isolation would clarify this. -- **JAX feasibility**: `via_tracer_from` and the plotter ray-tracing both - ultimately call `Tracer.image_2d_from` (or its multi-plane equivalent). - How much of that path is already pytree-friendly? If `Tracer` is registered - as a pytree (it is, when an `AnalysisPoint` constructs it with - `use_jax=True`) does `jax.jit(simulator.via_tracer_from)` succeed, or does - it bail out somewhere — and if so, where exactly? - -The end goal is a short report inside the script's docstring or a sibling -markdown file in the same folder, recommending one of: - -- **JAX-jit the renderer** — if the bottleneck is amenable and a clean - pytree path exists. Sketch the API surface that would change. -- **Drop default grid sizes / over-sample levels in cluster examples** — if - JAX is not feasible. State which numbers, with reasoning grounded in the - measurements. -- **Live with it** — if the costs are fundamental and the cluster examples - just have to be slow at this scale. - -## Constraints and notes - -- The cluster simulator itself is in good shape after issue #89 — the solver - is JIT-compiled, the script outputs sensible multi-image positions for - both sources, and the 3-minute total is acceptable while we figure out - the rest. So this is **not blocking the cluster work**; it is a follow-up - whose output should feed back into the cluster scripts in a later PR. -- The profiling script is intended to be run, edited, re-run iteratively. - Add new variants, drop old ones, instrument deeper into `via_tracer_from` - if needed. -- Do not modify `autolens_workspace/scripts/cluster/simulator.py` from this - task. Any user-visible changes (e.g. lowering default grid resolution) - should land as a separate cluster-workspace PR once the profiling story - is settled. - -## Companion scripts to copy structure from - -The file is laid out like -`autolens_workspace_developer/jax_profiling/imaging/mge.py` — same -`Timer.section` context manager, same end-of-run summary table. Mirror that -style if you add new instrumentation files alongside it (e.g. -`subplot_tracer_panel_breakdown.py`). diff --git a/active/release/pyautoheart/morning_status_and_daily_release_rehearsal.md b/active/release/pyautoheart/morning_status_and_daily_release_rehearsal.md deleted file mode 100644 index 040e888d..00000000 --- a/active/release/pyautoheart/morning_status_and_daily_release_rehearsal.md +++ /dev/null @@ -1,74 +0,0 @@ -# Morning status message + routine cron re-enable + daily release in rehearsal mode - -Type: release -Target: PyAutoHeart -Repos: -- PyAutoHeart -- PyAutoBuild -- PyAutoMind -- PyAutoFit -Difficulty: medium -Autonomy: safe -Priority: high -Status: formalised - -## Original request (verbatim) - -I recently disabled a number of nightly (or less than daily) auto runs because it was bloating my email. However, PyAutoHeart is now green, CI is passing and I think it would be good to have a nightly task which gives me an email (or ideally a Slack message, but make this follow up if it requires faff with manual inputs) that every morning says 'all is ok' or 'this is a problem'. Also: review all nightly / routine builds and work out if there is a better or more optimal setup. I want to get automated daily release back very soon too, perhaps set up but temporarily publishing to test PyPI (via rehearsal) mode for a bit, then switch over. - -## Fable review of the routine-build landscape (2026-07-08) - -Every routine cron was paused org-wide on 2026-07-06 with a uniform -`--- PAUSED 2026-07-06 ---` comment marker; each workflow keeps its -`workflow_dispatch` trigger. Inventory of paused crons: - -| Workflow | Cadence (paused) | Notes | -|---|---|---| -| `PyAutoBuild/release.yml` | daily 02:00 UTC weekdays | Already has a `rehearsal` input: true = build + publish to TestPyPI only, no PyPI/tag/commits. | -| `PyAutoHeart/heart-health.yml` | daily 06:00 UTC | The authoritative health verdict; writes README badge, tracking issue, Pages board. | -| `PyAutoMind/morning_status.yml` | daily 05:00 UTC | Slack **user-facing update digest** (last-24h commits, Claude-summarised). Slack webhook secret `PYAUTO_UPDATE_WEBHOOK_URL` already exists and works — Slack is NOT faff. | -| `PyAutoHeart/workspace-validation.yml` | Mon 03:00 UTC | Weekly smoke. | -| `PyAutoHeart/url-check.yml` | Mon 04:00 UTC | Weekly. | -| `PyAutoBuild/python_matrix.yml` | Mon 03:00 UTC | Weekly, off-cycle from daily release. | -| `PyAutoFit/nss_install_smoke.yml` | Sun 03:00 UTC | Weekly; also runs on PR. | - -The email bloat came from per-workflow GitHub failure notifications. The fix is -not fewer checks — it is one aggregated morning verdict and muted per-run email. - -## Recommended design (for the start_dev plan to refine) - -1. **One morning Slack message, built on Heart.** Re-enable the - `heart-health.yml` daily cron and append a Slack notifier step (same - webhook-secret pattern as `morning_status.yml`, likely a dedicated - dev-channel webhook rather than the user-facing update channel): one line — - "All is OK" or "Problem: " — plus a link to the Pages board. The - health job should aggregate the latest scheduled-run conclusions of the - other routine workflows (release rehearsal, weekly smokes) so a single - message covers everything; individual GitHub email notifications get muted. -2. **Daily release back in rehearsal mode.** Re-enable the `release.yml` - weekday cron with rehearsal behaviour controlled by a repo variable (e.g. - `vars.RELEASE_REHEARSAL`), since schedule events cannot pass - `workflow_dispatch` inputs. Flipping to real PyPI is then a one-variable - change with no commit — and stays a human action. -3. **Re-enable the weekly crons** (workspace-validation, url-check, - python_matrix, nss_install_smoke), staggered ahead of the Monday morning - message so their results are covered by it. -4. **`morning_status.yml` (user digest)** is a separate audience - (science users, not organism health); re-enabling it is a one-line - uncomment and can ride along, but keep it a distinct message/channel. - -Sequencing on a weekday: release rehearsal 02:00 → heart-health 06:00 -(covering the rehearsal outcome) → one Slack message ~06:15 UK morning. - -Open questions for planning: which Slack channel/webhook for the health -message (new secret vs reuse); whether heart-health polls sibling repos' run -conclusions via `gh api` or the workflows report inward. - -Follow-up (explicitly deferred by the user): only if Slack turns out to need -manual setup faff, fall back to a single daily email — but the existing webhook -suggests it will not. - -The TestPyPI → real-PyPI switch is **human-only**; this task sets up rehearsal -mode and the flip stays with the user. - - diff --git a/active/version_stamp_sync_and_release_sed_guards.md b/active/version_stamp_sync_and_release_sed_guards.md deleted file mode 100644 index 86f2fffa..00000000 --- a/active/version_stamp_sync_and_release_sed_guards.md +++ /dev/null @@ -1,54 +0,0 @@ -# Version-stamp sync to 2026.8.17.1 + release-sed guards - -Type: maintenance -Target: libraries -Repos: -- @PyAutoNerves -- @PyAutoArray -- @PyAutoFit -- @PyAutoGalaxy -- @PyAutoLens -- @PyAutoHands -Difficulty: small -Autonomy: safe -Priority: medium -Status: draft - -Original request (verbatim): "should we even bother with having these versions? -I think we should but if so they should update?" → "ok do it but do a bit more -deep research on the source code first to make sure its the right approach". - -In-session research (2026-08-17, full readers/writers map + handshake -mechanics) concluded the derive-from-`importlib.metadata` migration is -net-negative for this stack: local dev resolves libs via `PYTHONPATH` (no dist -metadata → fallback always taken → per-run `check_version` UserWarning), the -release stamp sed would corrupt a `try/except` block into SyntaxError wheels, -assistant chat-bundle regeneration would publish dev stamps to users, and -per-release bump commits to library mains are the documented cause of the -June/July 2026 accidental-release cascade (PyAutoBuild#118/#120). Human chose -the manual-sync-plus-guards package instead: - -1. **Sync the five frozen `__version__` literals** from `2026.7.23.1` to the - latest release `2026.8.17.1` (same deliberate manual sync that produced - 7.23): `autonerves/__init__.py:119`, `autoarray/__init__.py:110`, - `autofit/__init__.py:163`, `autogalaxy/__init__.py:134`, - `autolens/__init__.py:154`. Add a one-line comment at each stamp site - documenting the freeze design (git tag / stamped wheel = release truth; - this literal = last manual sync, deliberately NOT bumped per release). -2. **Delete the three dead legacy stampers** — `PyAutoLens/release.sh`, - `PyAutoArray/files/release.sh`, `PyAutoFit/files/release.sh` — unreferenced, - and their `grep -v __version__` rewrite would mangle any future refactor of - the stamp lines. -3. **Guard the live release seds** in - `PyAutoHands/.github/workflows/release.yml` (~136 rehearsal build, ~431 live - build): after each `sed`, verify the stamped `__init__.py` still parses - (`python3 -c "import ast; ast.parse(...)"`) and contains the exact - `__version__ = "$VERSION"` line, failing the job loudly otherwise. The sed's - unanchored zero-or-more pattern was empirically shown to be able to ship a - SyntaxError wheel silently. - -Supersedes `nerves_version_stamp_behind_consensus.md` (its drift already -resolved — all five stamps read 2026.7.23.1; the pre_build stamp-sweep question -it raised is moot since pre_build no longer stamps versions). While in -`version_drift.sh` territory: no change needed there — literal date stamps -remain grep-able after this task. diff --git a/active/autolens_workspace_developer/imaging_mge_pytree_migration.md b/complete/2026/08/imaging-mge-pytree-migration.md similarity index 75% rename from active/autolens_workspace_developer/imaging_mge_pytree_migration.md rename to complete/2026/08/imaging-mge-pytree-migration.md index 0323408f..957e0676 100644 --- a/active/autolens_workspace_developer/imaging_mge_pytree_migration.md +++ b/complete/2026/08/imaging-mge-pytree-migration.md @@ -1,3 +1,28 @@ +- issue: none — pre-lifecycle-migration prompt (old `active//` layout); never issued +- completed: retroactively logged by the 2026-08-19 completed-tasks reconcile sweep +- repos: autolens_workspace_developer + +## Summary + +Retroactively logged. The prompt asked for +`jax_profiling/imaging/mge.py` to migrate its JIT'd closure from a flat 1D +parameter vector (`jnp_params`, `model.instance_from_vector` at the JIT +boundary) to pytree-native inputs. The `jax_profiling/` tree was since +restructured and the ask is delivered in the current +`jax_profiling/gradient/imaging/mge.py` (verified on +autolens_workspace_developer main `c5cccb8`, 2026-08-19): the script +registers the model as a JAX pytree via `autofit.jax.register_model`, +converts the sampled `ModelInstance` into a JAX-array-valued pytree, and +each step closure differentiates w.r.t. that pytree — the parameter-identity +and vmap-batching goals of the prompt. + +The prompt file sat unclaimed in `active/autolens_workspace_developer/` +(a pre-migration subdirectory invisible to `lifecycle.py check`/`orphans`, +which only scan `active/*.md`) from before the 2026-07-13 lifecycle +migration until this sweep retired it. + +## Original prompt + # Imaging MGE JAX JIT Profiling — Migrate to Pytree Inputs ## Context diff --git a/active/autolens_workspace_developer/rectangular_spline_adapt_image_jit_fix.md b/complete/2026/08/rectangular-spline-adapt-image-jit-fix.md similarity index 79% rename from active/autolens_workspace_developer/rectangular_spline_adapt_image_jit_fix.md rename to complete/2026/08/rectangular-spline-adapt-image-jit-fix.md index ac34494e..db45d8b9 100644 --- a/active/autolens_workspace_developer/rectangular_spline_adapt_image_jit_fix.md +++ b/complete/2026/08/rectangular-spline-adapt-image-jit-fix.md @@ -1,3 +1,30 @@ +- issue: none — pre-lifecycle-migration prompt (old `active//` layout); never issued +- completed: retroactively logged by the 2026-08-19 completed-tasks reconcile sweep +- repos: autolens_workspace_developer + +## Summary + +Retroactively logged. The prompt reported `RectangularSplineAdaptImage` +crashing on its first JIT call (`AttributeError: 'NoneType' object has no +attribute 'array'` out of the `AdaptImages` plumbing) in +`jax_profiling/imaging/pixelization_spline_vs_linear.py`, blocking the +adapt-image row of the spline-vs-linear benchmark. On current +autolens_workspace_developer main (`c5cccb8`, verified 2026-08-19) the +script lives at `jax_profiling/misc/pixelization_spline_vs_linear.py`, runs +all four rectangular meshes with path-keyed adapt images (so +`Analysis.adapt_images_via_instance_from` rebuilds them inside the trace), +and committed results exist for the previously-crashing variants — +`jax_profiling/results/jit/imaging/spline_vs_linear_fit/` holds +`subplot_fit_RectangularSplineAdaptDensity.png` and a +`spline_vs_linear_hst_v2026.4.13.6.json` benchmark record. + +The prompt file sat unclaimed in `active/autolens_workspace_developer/` +(a pre-migration subdirectory invisible to `lifecycle.py check`/`orphans`, +which only scan `active/*.md`) from before the 2026-07-13 lifecycle +migration until this sweep retired it. + +## Original prompt + # Fix `RectangularSplineAdaptImage` JIT crash in `AdaptImages` plumbing ## Context diff --git a/complete/2026/08/version-stamp-sync-guards.md b/complete/2026/08/version-stamp-sync-guards.md new file mode 100644 index 00000000..4d13c29e --- /dev/null +++ b/complete/2026/08/version-stamp-sync-guards.md @@ -0,0 +1,112 @@ +- issue: https://github.com/PyAutoLabs/PyAutoHands/issues/235 (closed by the 2026-08-19 reconcile sweep — see traps) +- prs: PyAutoNerves#148 (`e9f7c11`), PyAutoArray#447 (`7d1906e`), PyAutoFit#1488 (`fe9f813`), PyAutoGalaxy#573 (`49115ad`), PyAutoLens#698, PyAutoHands#236 — **all six MERGED 2026-08-17T22:03Z**, within 16 seconds of each other +- classification: maintenance (libraries + Hands) — version-stamp sync + release-sed hardening +- branch: `feature/version-stamp-sync-guards` (all six repos; remote branches may still exist — repo_cleanup territory) +- worktree: `~/Code/PyAutoLabs-wt/version-stamp-sync-guards` — release pending; free it in the next laptop session or let repo_cleanup flag it + +## What shipped + +The three-part package PyAutoHands#235 planned, verified merged on every main: + +1. **Stamp sync** — the five frozen `__version__` literals synced + `2026.7.23.1` → `2026.8.17.1` (verified on main: `autofit/__init__.py:167` + reads `2026.8.17.1`), each with the freeze design documented in-line + (git tag / stamped wheel = release truth; literal = last manual sync, + deliberately NOT bumped per release — PyAutoBuild#118/#120). +2. **Dead stampers deleted** — the three unreferenced legacy `release.sh` + scripts removed from PyAutoLens / PyAutoArray / PyAutoFit. +3. **Release-sed guards** — `PyAutoHands/.github/workflows/release.yml` now + refuses an empty `$VERSION` in the live step and, after each stamp sed in + both build jobs, `ast.parse`s the stamped `__init__.py` and asserts the + exact `__version__ = "$VERSION"` line — the empirically-shown + silent-SyntaxError-wheel failure mode is now loud. + +Test evidence at ship (from the issue's 2026-08-17T22:04Z "Shipped" comment): +full library suites green (155/1058/1852/1111/538/314 passed, 0 failures); +smoke euclid 6/6 and HowToLens 50/50 passed. + +## Traps and findings + +- **THE WRAP-UP TAIL WAS DROPPED — this record is 2 days late.** The shipping + session merged all six PRs and posted the "Shipped" comment on #235 at + 22:04Z, promised the re-run smoke result ("autofit/autogalaxy/autolens/ + autolens_test legs are re-running — result will be posted here"), and then + died. The smoke result was never posted, #235 was never closed, no record + was written, and the active.md entry went stale. Found and reconciled by + the 2026-08-19 completed-tasks sweep. The close rests on the six merged + PRs + the green suites; reopen #235 if the lost smoke legs turn out to have + found something real. +- **Wrong-PR pointer in active.md.** The stale entry's status line read + `pr-open (PyAutoLens/pull/700)` — but #700 is the + positions-lh-penalty-accumulation PR (a different task shipped the same + evening from the same session). Consistent with that record's + "registry-edit trap": a blanket `sed` status edit across active.md sections. + Scope registry edits to the task's own `##` section. +- **Detection gap (now closed by this sweep's follow-up):** `lifecycle.py + issues` only flags entries whose tracking ISSUE is closed. In this failure + mode the dying session never closes the issue, so the drift was invisible + to the only reconciliation check that existed — and that check needs `gh`, + so it never runs in CI or cloud sessions anyway. +- Behavioral note from the ship comment (still true until acted on): the + synced stamp is >30 days past the workspace floors (2026.7.9.1), so source + checkouts running workspace scripts see the `check_version` staleness + warning pip users already see. Remedy when it becomes noise: bump + `version.minimum_library_version` in the workspace configs or set + `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1`. + +## Original prompt + +# Version-stamp sync to 2026.8.17.1 + release-sed guards + +Type: maintenance +Target: libraries +Repos: +- @PyAutoNerves +- @PyAutoArray +- @PyAutoFit +- @PyAutoGalaxy +- @PyAutoLens +- @PyAutoHands +Difficulty: small +Autonomy: safe +Priority: medium +Status: draft + +Original request (verbatim): "should we even bother with having these versions? +I think we should but if so they should update?" → "ok do it but do a bit more +deep research on the source code first to make sure its the right approach". + +In-session research (2026-08-17, full readers/writers map + handshake +mechanics) concluded the derive-from-`importlib.metadata` migration is +net-negative for this stack: local dev resolves libs via `PYTHONPATH` (no dist +metadata → fallback always taken → per-run `check_version` UserWarning), the +release stamp sed would corrupt a `try/except` block into SyntaxError wheels, +assistant chat-bundle regeneration would publish dev stamps to users, and +per-release bump commits to library mains are the documented cause of the +June/July 2026 accidental-release cascade (PyAutoBuild#118/#120). Human chose +the manual-sync-plus-guards package instead: + +1. **Sync the five frozen `__version__` literals** from `2026.7.23.1` to the + latest release `2026.8.17.1` (same deliberate manual sync that produced + 7.23): `autonerves/__init__.py:119`, `autoarray/__init__.py:110`, + `autofit/__init__.py:163`, `autogalaxy/__init__.py:134`, + `autolens/__init__.py:154`. Add a one-line comment at each stamp site + documenting the freeze design (git tag / stamped wheel = release truth; + this literal = last manual sync, deliberately NOT bumped per release). +2. **Delete the three dead legacy stampers** — `PyAutoLens/release.sh`, + `PyAutoArray/files/release.sh`, `PyAutoFit/files/release.sh` — unreferenced, + and their `grep -v __version__` rewrite would mangle any future refactor of + the stamp lines. +3. **Guard the live release seds** in + `PyAutoHands/.github/workflows/release.yml` (~136 rehearsal build, ~431 live + build): after each `sed`, verify the stamped `__init__.py` still parses + (`python3 -c "import ast; ast.parse(...)"`) and contains the exact + `__version__ = "$VERSION"` line, failing the job loudly otherwise. The sed's + unanchored zero-or-more pattern was empirically shown to be able to ship a + SyntaxError wheel silently. + +Supersedes `nerves_version_stamp_behind_consensus.md` (its drift already +resolved — all five stamps read 2026.7.23.1; the pre_build stamp-sweep question +it raised is moot since pre_build no longer stamps versions). While in +`version_drift.sh` territory: no change needed there — literal date stamps +remain grep-able after this task. diff --git a/active/kxs_ground_truth.py b/complete/archive/shelved/kxs_ground_truth.py similarity index 100% rename from active/kxs_ground_truth.py rename to complete/archive/shelved/kxs_ground_truth.py diff --git a/active/oversampling_ground_truth.py b/complete/archive/shelved/oversampling_ground_truth.py similarity index 100% rename from active/oversampling_ground_truth.py rename to complete/archive/shelved/oversampling_ground_truth.py diff --git a/complete/index.md b/complete/index.md index ed7baddf..e6b05575 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1028 records across 7 buckets. +1031 records across 7 buckets. ## Highlights @@ -74,6 +74,7 @@ _(curate hard-won records here — survives regeneration.)_ - [hygiene-detail-flag](2026/08/hygiene-detail-flag.md) - [hygiene-gone-ref-counter](2026/08/hygiene-gone-ref-counter.md) — the prompt was never issued — the defects were fixed under other issues, see below - [hygiene-howto-refs-docstrings](2026/08/hygiene-howto-refs-docstrings.md) +- [imaging-mge-pytree-migration](2026/08/imaging-mge-pytree-migration.md) - [inference-method-cards-seed](2026/08/inference-method-cards-seed.md) - [inference-programme-ledger](2026/08/inference-programme-ledger.md) - [interferometer-delaunay-flaky-fitexception](2026/08/interferometer-delaunay-flaky-fitexception.md) @@ -121,6 +122,7 @@ _(curate hard-won records here — survives regeneration.)_ - [pyautofit_plot_rst_dead_plotters](2026/08/pyautofit_plot_rst_dead_plotters.md) - [pyautogalaxy-mge-sigma-test](2026/08/pyautogalaxy-mge-sigma-test.md) - [reconcile-upstream-repo-mode](2026/08/reconcile-upstream-repo-mode.md) +- [rectangular-spline-adapt-image-jit-fix](2026/08/rectangular-spline-adapt-image-jit-fix.md) - [registry-integrity-check](2026/08/registry-integrity-check.md) - [regularization-jax-gradient-gaps](2026/08/regularization-jax-gradient-gaps.md) - [release-drive-2026-08-03](2026/08/release-drive-2026-08-03.md) @@ -150,6 +152,7 @@ _(curate hard-won records here — survives regeneration.)_ - [transformed-message-semantics-doc](2026/08/transformed-message-semantics-doc.md) - [uniform-prior-bounds-numpy-path](2026/08/uniform-prior-bounds-numpy-path.md) — auto-closed by the merge's `Closes` line - [version-skew-yank-awareness](2026/08/version-skew-yank-awareness.md) +- [version-stamp-sync-guards](2026/08/version-stamp-sync-guards.md) — closed by the 2026-08-19 reconcile sweep — see traps - [wiki-provenance-restamp](2026/08/wiki-provenance-restamp.md) - [worktree-claim-parser-forms](2026/08/worktree-claim-parser-forms.md) — CLOSED completed - [yang24-sidm-gravothermal-profile](2026/08/yang24-sidm-gravothermal-profile.md) diff --git a/dashboard.md b/dashboard.md index 22eaa845..3aea889a 100644 --- a/dashboard.md +++ b/dashboard.md @@ -8,7 +8,7 @@ Tasks only — the organism's health lives with the Heart (`/health`), not here. | Where | Count | |-------|------:| -| [In flight](#in-flight) (`active/`) | 5 | +| [In flight](#in-flight) (`active/`) | 1 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | | [Backlog](#backlog) (`draft/`) | 122 | @@ -187,33 +187,6 @@ Live on GitHub: [open issues](https://github.com/search?q=org%3APyAutoLabs+is%3A Issued — each has an open GitHub issue and usually a branch. The full record for each is in [`active.md`](active.md). -- [`@PyAutoFit` Refactor: each density should live in one place, not](active/12_single_source_density_refactor.md) — [issue #1500](https://github.com/PyAutoLabs/PyAutoFit/issues/1500) -
📋 copy for Claude - - ``` - /start_dev active/12_single_source_density_refactor.md - ``` - -
- -- [`@PyAutoFit` Refactor: collapse the `Prior` / `Message` two-layer hierarchy](active/13_collapse_prior_and_message.md) — [issue #1500](https://github.com/PyAutoLabs/PyAutoFit/issues/1500) -
📋 copy for Claude - - ``` - /start_dev active/13_collapse_prior_and_message.md - ``` - -
- -- [PyAutoReduce validation: slacs1430+4105 ACS reduction vs trusted legacy dataset](active/pyautoreduce_slacs1430_acs_comparison.md) -
📋 copy for Claude - - ``` - /start_dev active/pyautoreduce_slacs1430_acs_comparison.md - ``` - -
- - [Reconstructing a stored sample raises through `ignore_assertions=True`](active/to_instance_guard_gap.md) — [issue #1486](https://github.com/PyAutoLabs/PyAutoFit/issues/1486) — library-dev — WORKSPACE HALF SHIPPED; the PyAutoFit hardening (#1486) is what remains
📋 copy for Claude @@ -223,15 +196,6 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
-- [Version-stamp sync to 2026.8.17.1 + release-sed guards](active/version_stamp_sync_and_release_sed_guards.md) — [issue #235](https://github.com/PyAutoLabs/PyAutoHands/issues/235) — pr-open (https://github.com/PyAutoLabs/PyAutoLens/pull/700) -
📋 copy for Claude - - ``` - /start_dev active/version_stamp_sync_and_release_sed_guards.md - ``` - -
- ## Parked Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md). diff --git a/docs/pyautobrain/spawn_spec.md b/docs/pyautobrain/spawn_spec.md index 88defbed..a19fb5c3 100644 --- a/docs/pyautobrain/spawn_spec.md +++ b/docs/pyautobrain/spawn_spec.md @@ -48,7 +48,7 @@ deliberately, never silently shipped into a template. | 9 | `.github/**` | **Per file, by the succeed-on-a-fresh-repo test below.** Not a blanket rule: owner substitution alone does NOT make a workflow work, because `YOURORG` is a literal placeholder — the template's own `spawn_drift` run failed `repository 'https://github.com/YOURORG/PyAutoMind/' not found`. See rules 9a–9c | | 9a | `.github/workflows/lifecycle_drift.yml` | KEEP verbatim — operates only on its own repo (checkout + local scripts) and contains no owner reference at all, so it needs no substitution and succeeds unmodified in a fresh org. Empirically the one green workflow in the template's run history | | 9b | `.github/workflows/spawn_drift.yml` | DROP — was "keep with the `schedule:` stripped", revised in #125. The self-heal added there makes this workflow depend on `secrets.PAT_PYAUTOLABS` AND on published `*-template` repos, neither of which a freshly-spawned org has, so **every** path in it is unrunnable there and the secret reference alone breaks the no-configured-secret condition. "When in doubt DROP" applies: an org that later publishes templates can adopt this workflow deliberately, having read it. The template still ships `scripts/spawn.py` + `tests/`, so the generator and its guards travel; only the org-coupled automation does not | -| 9c | `.github/workflows/{dashboard_refresh,morning_status,morning_health,arxiv_papers}.yml`, `.github/scripts/**` | DROP — instance automation. `dashboard_refresh.yml` checks out `PyAutoLabs/PyAutoBrain` (the dashboard renderer lives with the intake conductor, not in Mind), so it fails on checkout in any org that has no such sibling — and owner substitution only turns that into the literal `YOURORG/PyAutoBrain`. The rest They hardcode sibling repo lists, organ-specific workflow names (`PyAutoHeart`/`PyAutoBrain`/`PyAutoHands`), org secrets (`PYAUTO_PAPERS_WEBHOOK_URL`, `CLAUDE_CODE_OAUTH_TOKEN`) and, in `arxiv_fetch.py`, strong-lensing search vocabulary plus dated incident notes. All 13 failing runs in the published template came from these | +| 9c | `.github/workflows/{dashboard_refresh,registry_reconcile,morning_status,morning_health,arxiv_papers}.yml`, `.github/scripts/**` | DROP — instance automation. `dashboard_refresh.yml` checks out `PyAutoLabs/PyAutoBrain` (the dashboard renderer lives with the intake conductor, not in Mind), so it fails on checkout in any org that has no such sibling — and owner substitution only turns that into the literal `YOURORG/PyAutoBrain`. The rest They hardcode sibling repo lists, organ-specific workflow names (`PyAutoHeart`/`PyAutoBrain`/`PyAutoHands`), org secrets (`PYAUTO_PAPERS_WEBHOOK_URL`, `CLAUDE_CODE_OAUTH_TOKEN`) and, in `arxiv_fetch.py`, strong-lensing search vocabulary plus dated incident notes. All 13 failing runs in the published template came from these | | 9d | any other `.github/**` | **No catch-all rule — UNMATCHED by design.** A fallback here is fail-*open*: a workflow added to Mind later would ride it into the template carrying whatever schedule and secrets it has, which is precisely the defect 9a–9c fix. A new `.github` file must fail the run and get an explicit entry above, like every other new file class | | 10 | `.claude/**`, `.codex/**` | DROP — agent-discovery symlinks are install artifacts recreated by the PyAutoBrain installer, not source content | diff --git a/scripts/lifecycle.py b/scripts/lifecycle.py index 5424f178..9ddc5992 100644 --- a/scripts/lifecycle.py +++ b/scripts/lifecycle.py @@ -37,6 +37,8 @@ fallback, and into the state folder its registry implies * no slug is listed in two registries at once * no active/ prompt is left unclaimed by every registry + * nothing lives under active/ except top-level prompt .md files + (subdirectories and scripts are invisible to every other guard) Wire into /health and CI. orphans @@ -44,10 +46,15 @@ entry claims. `check` grades this too — this lists only them. issues - The ONLINE leg (needs `gh` + network, so deliberately not part of - `check`): every registry entry's tracking issue cross-checked against - GitHub. Catches finished work still listed as pending — the class no - offline check can see. + The ONLINE leg (needs network — `gh` when installed, else plain HTTPS + with an optional GITHUB_TOKEN; deliberately not part of `check`, which + stays hermetic): every registry entry's tracking issue AND every + `status: pr-open` PR cross-checked against GitHub. Catches finished + work still listed as pending — including the crashed-ship case where + the PR merged but the session died before closing the issue, so the + issue leg alone stays green. Run daily by registry_reconcile.yml + (instance automation — deliberately not part of the shipped + lifecycle_drift.yml, which must stay schedule-free per spawn rule 9). This file is intentionally stdlib-only (no PyAuto imports) so it runs in any environment, including a bare template checkout. @@ -246,6 +253,16 @@ def registry_problems(root: Path) -> "list[str]": ISSUE_FIELDS = ("issue", "epic") ISSUE_URL_RE = re.compile(r"https://github\.com/([\w.-]+)/([\w.-]+)/issues/(\d+)") +# The one PR reference that IS tracking state: a `status:` field declaring the +# task is waiting on an open PR. When that PR merges, the ship bookkeeping +# (record + retire + issue close) is owed — and if the shipping session dies +# first, the entry goes stale with its tracking ISSUE still open, which is +# exactly the case the issue leg above cannot see. (version-stamp-sync-guards, +# 2026-08-17: six PRs merged, session died after the "Shipped" comment; found +# two days later only by a manual sweep.) +PR_URL_RE = re.compile(r"https://github\.com/([\w.-]+)/([\w.-]+)/pull/(\d+)") +PR_OPEN_TOKEN = "pr-open" + class GhUnavailable(RuntimeError): """`gh` is not installed. Distinct from "gh ran and said no" so the command @@ -268,23 +285,65 @@ def registry_issue_refs(root: Path) -> "list[tuple[str, str, str]]": return refs -def _gh_issue_states(urls: "list[str]") -> "dict[str, str]": - """{url: state} via the `gh` CLI. Requires gh + network; online leg only.""" +def _http_api_state(owner: str, repo: str, kind: str, num: str) -> str: + """One GitHub REST read via stdlib urllib — the gh-free fallback path. + + `kind` is "issues" or "pulls". Sends `GITHUB_TOKEN`/`GH_TOKEN` when set + (CI runners always have one; anonymous works for public repos, rate-limited). + Returns "merged" for a merged PR, else the API `state`, else "unreadable: …" + — never raises, so one dead URL cannot sink the whole report.""" + import json + import os + import urllib.error + import urllib.request + + req = urllib.request.Request( + f"https://api.github.com/repos/{owner}/{repo}/{kind}/{num}", + headers={ + "Accept": "application/vnd.github+json", + "User-Agent": "pyautomind-lifecycle", + }, + ) + token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") + if token: + req.add_header("Authorization", f"Bearer {token}") + try: + with urllib.request.urlopen(req, timeout=30) as resp: + data = json.load(resp) + except urllib.error.HTTPError as e: + return f"unreadable: HTTP {e.code}" + except Exception as e: # URLError, timeout, bad JSON — report, don't crash + return f"unreadable: {e.__class__.__name__}" + if kind == "pulls" and data.get("merged_at"): + return "merged" + return data.get("state", "unknown") + + +def _states_via_gh_or_http(urls: "list[str]", url_re: "re.Pattern", + kind: str, jq: str) -> "dict[str, str]": + """{url: state} via the `gh` CLI, falling back to plain HTTPS when gh is + not installed (cloud/web sessions, bare CI images). Online leg only.""" import subprocess states: "dict[str, str]" = {} + gh_missing = False for url in urls: - m = ISSUE_URL_RE.match(url) + m = url_re.match(url) if not m: continue owner, repo, num = m.groups() + if gh_missing: + states[url] = _http_api_state(owner, repo, kind, num) + continue try: r = subprocess.run( - ["gh", "api", f"repos/{owner}/{repo}/issues/{num}", "--jq", ".state"], + ["gh", "api", f"repos/{owner}/{repo}/{kind}/{num}", "--jq", jq], capture_output=True, text=True, ) except FileNotFoundError: - raise GhUnavailable + gh_missing = True + states[url] = _http_api_state(owner, repo, kind, num) + continue if r.returncode != 0: tail = (r.stderr.strip().splitlines() or ["error"])[-1] states[url] = f"unreadable: {tail}" @@ -293,6 +352,16 @@ def _gh_issue_states(urls: "list[str]") -> "dict[str, str]": return states +def _gh_issue_states(urls: "list[str]") -> "dict[str, str]": + return _states_via_gh_or_http(urls, ISSUE_URL_RE, "issues", ".state") + + +def _gh_pr_states(urls: "list[str]") -> "dict[str, str]": + return _states_via_gh_or_http( + urls, PR_URL_RE, "pulls", + 'if .merged_at then "merged" else .state end') + + def issue_problems(root: Path, fetch=None) -> "list[str]": """Registry entries whose tracking issue is CLOSED — i.e. finished work still listed as pending. @@ -318,6 +387,53 @@ def issue_problems(root: Path, fetch=None) -> "list[str]": return problems +def registry_pr_refs(root: Path) -> "list[tuple[str, str, str]]": + """(registry, slug, pr_url) for every entry whose `status:` declares an + open PR. Only the `status:` field is read — a PR URL anywhere else in the + entry (prose, `library-pr:` history) is not a claim of in-flight state.""" + refs = [] + for reg in REGISTRY_FILES: + for slug, fields in registry_entries(root / reg): + status = fields.get("status", "") + if PR_OPEN_TOKEN not in status: + continue + for m in PR_URL_RE.finditer(status): + refs.append((reg, slug, m.group(0))) + return refs + + +def pr_problems(root: Path, fetch=None) -> "list[str]": + """Registry entries whose `status: pr-open` PR is no longer open. + + The complement of `issue_problems`: when a shipping session dies between + the merge and the bookkeeping, the tracking issue is never closed — so the + issue leg stays green while the entry rots. The merged PR is the one + signal that survives the crash. `fetch` is injectable for tests.""" + refs = registry_pr_refs(root) + if not refs: + return [] + fetch = fetch or _gh_pr_states + states = fetch([url for _, _, url in refs]) + + problems = [] + for reg, slug, url in refs: + state = states.get(url, "unknown") + if state == "merged": + problems.append( + f"{reg}: {slug}: status says pr-open but the PR is MERGED — " + f"the ship bookkeeping (record + retire + issue close) was " + f"never done: {url}" + ) + elif state == "closed": + problems.append( + f"{reg}: {slug}: status says pr-open but the PR was CLOSED " + f"without merging: {url}" + ) + elif state != "open": + problems.append(f"{reg}: {slug}: could not read PR state ({state}): {url}") + return problems + + def draft_issue_refs(root: Path) -> "list[tuple[str, str]]": """(draft_path, issue_url) for draft prompts citing a GitHub issue. @@ -489,9 +605,10 @@ def draft_gate_notes(root: Path, fetch=None) -> "dict[str, list[str]]": def cmd_issues(args) -> int: - """Cross-check every registry entry's tracking issue against GitHub.""" + """Cross-check every registry entry's tracking issue AND every + `status: pr-open` PR against GitHub.""" try: - problems = issue_problems(ROOT) + problems = issue_problems(ROOT) + pr_problems(ROOT) except GhUnavailable: print( "lifecycle issues: cannot run — the `gh` CLI is not installed.\n" @@ -501,6 +618,18 @@ def cmd_issues(args) -> int: file=sys.stderr, ) return 2 + n_refs = len(registry_issue_refs(ROOT)) + len(registry_pr_refs(ROOT)) + if problems and n_refs and all("could not read" in p for p in problems) \ + and len(problems) == n_refs: + print( + "lifecycle issues: cannot run — GitHub is unreachable from here\n" + " (every state read failed). Run from a session with GitHub\n" + " access, or set GITHUB_TOKEN for the HTTPS fallback.", + file=sys.stderr, + ) + for line in problems[:3]: + print(f" e.g. {line}", file=sys.stderr) + return 2 notes = [] gates = {"shipped": [], "unblocked": [], "partial": [], "unreadable": []} if getattr(args, "drafts", False): @@ -515,7 +644,9 @@ def cmd_issues(args) -> int: for line in problems: print(f" - {line}") else: - print(f"lifecycle issues: OK ({len(registry_issue_refs(ROOT))} tracking issue(s) open)") + print( + f"lifecycle issues: OK ({len(registry_issue_refs(ROOT))} tracking " + f"issue(s) open, {len(registry_pr_refs(ROOT))} pr-open PR(s) open)") # Declared gates first: the prompt author said which reading applies, so # these are actionable rather than a question. Still advisory — retiring a @@ -552,6 +683,30 @@ def cmd_issues(args) -> int: return 1 if problems else 0 +def active_strays(root: Path) -> "list[Path]": + """Files under active/ the lifecycle tooling cannot see. + + `check`, `orphans` and the dashboard all scan `active/*.md` — top level + only. Anything else under active/ (a pre-migration `active//` + subdirectory, a stray script) is therefore invisible to every guard, which + is how five completed leftovers sat there for a month+ until the + 2026-08-19 sweep: three prompts whose completion records had existed since + May/July, and two retired ground-truth scripts. A prompt belongs at + `active/.md`; anything else belongs in another state folder or in + `complete/archive/`.""" + active_dir = root / "active" + if not active_dir.is_dir(): + return [] + strays = [] + for f in sorted(active_dir.rglob("*")): + if f.is_dir(): + continue + if f.parent == active_dir and f.suffix == ".md": + continue + strays.append(f) + return strays + + def orphan_prompts(root: Path) -> "list[Path]": """active/*.md that no registry entry claims — the mirror of registry_problems(). @@ -922,6 +1077,11 @@ def cmd_check(args) -> int: f"active/ prompt no registry entry claims: {f.relative_to(ROOT)}" for f in orphan_prompts(ROOT) ) + problems.extend( + f"active/ stray the lifecycle tooling cannot see (retire, or re-home " + f"to a state folder): {f.relative_to(ROOT)}" + for f in active_strays(ROOT) + ) if problems: print("lifecycle check: DRIFT") @@ -960,7 +1120,9 @@ def main() -> int: c.set_defaults(func=cmd_check) iss = sub.add_parser( - "issues", help="cross-check registry tracking issues against GitHub (needs gh)" + "issues", + help="cross-check registry tracking issues + pr-open PRs against " + "GitHub (needs gh, or network + optional GITHUB_TOKEN)", ) iss.add_argument("--drafts", action="store_true", help="also flag draft/ prompts citing a closed issue (advisory)") diff --git a/scripts/spawn.py b/scripts/spawn.py index b5ce5f43..a5a5ba19 100644 --- a/scripts/spawn.py +++ b/scripts/spawn.py @@ -116,6 +116,10 @@ # under owner substitution the name is the literal YOURORG placeholder), so # every run of it there fails on checkout. (".github/workflows/dashboard_refresh.yml", "DROP"), + # 9c also: the online lifecycle leg — scheduled, and it reads sibling-repo + # issue/PR state across the org. A fresh org has neither the schedule + # tolerance (rule 9's no-unattended-trigger condition) nor the siblings. + (".github/workflows/registry_reconcile.yml", "DROP"), (".github/workflows/morning_status.yml", "DROP"), (".github/workflows/morning_health.yml", "DROP"), (".github/workflows/arxiv_papers.yml", "DROP"), diff --git a/tests/test_lifecycle_check.py b/tests/test_lifecycle_check.py index a8c1ccc2..0cbaf6cc 100644 --- a/tests/test_lifecycle_check.py +++ b/tests/test_lifecycle_check.py @@ -411,18 +411,26 @@ def test_default_fetcher_builds_the_gh_argv_and_parses_state(monkeypatch): ] -def test_default_fetcher_raises_gh_unavailable_when_gh_is_missing(monkeypatch): - """The one error the command turns into "could not run" rather than a - finding — so it must be the exception type, not a state string.""" - import pytest +def test_default_fetcher_falls_back_to_http_when_gh_is_missing(monkeypatch): + """No `gh` (cloud/web sessions, bare CI images) must not mean "could not + run": the fetcher falls back to the stdlib HTTPS path and still returns + real states. The HTTP helper is stubbed so the test stays offline.""" def _missing(argv): raise FileNotFoundError(2, "No such file or directory: 'gh'") _stub_run(monkeypatch, _missing) - with pytest.raises(lifecycle.GhUnavailable): - lifecycle._gh_issue_states([GHOST_ISSUE]) + http_calls = [] + + def _fake_http(owner, repo, kind, num): + http_calls.append((owner, repo, kind, num)) + return "open" + + monkeypatch.setattr(lifecycle, "_http_api_state", _fake_http) + + assert lifecycle._gh_issue_states([GHOST_ISSUE]) == {GHOST_ISSUE: "open"} + assert http_calls == [("FictionalOrg", "FlywheelRepo", "issues", "17")] def test_default_fetcher_reports_a_failed_call_as_unreadable(monkeypatch): @@ -478,6 +486,113 @@ def _by_number(argv): } +# --------------------------------------------------------------------------- # +# the online leg — `status: pr-open` PR state +# +# The crashed-ship signature: the PR merged but the shipping session died +# before the bookkeeping, so the tracking issue is still OPEN and the issue +# leg above stays green. The merged PR is the only signal that survives. +# --------------------------------------------------------------------------- # +GHOST_PR = "https://github.com/FictionalOrg/FlywheelRepo/pull/42" + + +def test_merged_pr_on_a_pr_open_entry_is_drift(tmp_path): + (tmp_path / "active.md").write_text( + _entry("sprocket-calibration").replace( + "- status: planned", f"- status: pr-open ({GHOST_PR})") + ) + problems = lifecycle.pr_problems(tmp_path, fetch=_states({GHOST_PR: "merged"})) + assert len(problems) == 1 + assert "MERGED" in problems[0] + assert "sprocket-calibration" in problems[0] + + +def test_open_pr_on_a_pr_open_entry_is_not_drift(tmp_path): + (tmp_path / "active.md").write_text( + _entry("sprocket-calibration").replace( + "- status: planned", f"- status: pr-open ({GHOST_PR})") + ) + assert lifecycle.pr_problems(tmp_path, fetch=_states({GHOST_PR: "open"})) == [] + + +def test_closed_unmerged_pr_on_a_pr_open_entry_is_drift(tmp_path): + """A PR closed without merging is not shipped — but the entry still claims + an open PR, so its state is wrong either way and needs a human.""" + (tmp_path / "active.md").write_text( + _entry("sprocket-calibration").replace( + "- status: planned", f"- status: pr-open ({GHOST_PR})") + ) + problems = lifecycle.pr_problems(tmp_path, fetch=_states({GHOST_PR: "closed"})) + assert len(problems) == 1 + assert "CLOSED" in problems[0] + + +def test_pr_urls_outside_the_status_field_are_not_tracking_refs(tmp_path): + """A shipped task's `library-pr:`/`workspace-pr:` history, or a PR cited in + prose, is not a claim of in-flight state — only `status: pr-open` is.""" + body = _entry( + "sprocket-calibration", + extra=( + f"- library-pr: {GHOST_PR}\n" + f"- notes: superseded by {GHOST_PR} long ago\n" + ), + ) + (tmp_path / "active.md").write_text(body) + assert lifecycle.registry_pr_refs(tmp_path) == [] + assert lifecycle.pr_problems(tmp_path, fetch=_states({})) == [] + + +def test_pr_state_fetcher_builds_the_pulls_argv_with_merged_jq(monkeypatch): + """PRs need the merged_at disambiguation: GitHub's `state` is "closed" for + both a merged and an abandoned PR, and the two mean opposite things here.""" + calls = _stub_run(monkeypatch, lambda argv: _Completed(stdout="merged\n")) + + states = lifecycle._gh_pr_states([GHOST_PR]) + + assert states == {GHOST_PR: "merged"} + assert calls == [ + [ + "gh", + "api", + "repos/FictionalOrg/FlywheelRepo/pulls/42", + "--jq", + 'if .merged_at then "merged" else .state end', + ] + ] + + +# --------------------------------------------------------------------------- # +# active/ strays — files the lifecycle tooling cannot see +# +# check/orphans/dashboard all scan active/*.md, top level only. Five completed +# leftovers (three with months-old records) hid in active/ subdirectories and +# as stray scripts until the 2026-08-19 sweep; this gate makes that class +# visible hermetically. +# --------------------------------------------------------------------------- # +def test_subdirectory_prompt_and_stray_script_are_strays(tmp_path): + root = _tree( + tmp_path, + active=["tracked_task.md"], + registries={"active.md": _entry("tracked-task", prompt="active/tracked_task.md")}, + ) + sub = root / "active" / "legacy_target" + sub.mkdir() + (sub / "old_prompt.md").write_text("# pre-migration leftover\n") + (root / "active" / "ground_truth.py").write_text("print('retired scratch')\n") + + strays = [str(p.relative_to(root)) for p in lifecycle.active_strays(root)] + assert strays == ["active/ground_truth.py", "active/legacy_target/old_prompt.md"] + + +def test_top_level_prompts_are_not_strays(tmp_path): + root = _tree( + tmp_path, + active=["tracked_task.md"], + registries={"active.md": _entry("tracked-task", prompt="active/tracked_task.md")}, + ) + assert lifecycle.active_strays(root) == [] + + # --------------------------------------------------------------------------- # # the mirror direction — active/ prompts no registry claims # --------------------------------------------------------------------------- # diff --git a/tests/test_spawn_template_contract.py b/tests/test_spawn_template_contract.py index d4038017..e105612b 100644 --- a/tests/test_spawn_template_contract.py +++ b/tests/test_spawn_template_contract.py @@ -93,6 +93,7 @@ def _real(name): ".github/workflows/lifecycle_drift.yml": _real("lifecycle_drift.yml"), ".github/workflows/spawn_drift.yml": _real("spawn_drift.yml"), ".github/workflows/dashboard_refresh.yml": _real("dashboard_refresh.yml"), + ".github/workflows/registry_reconcile.yml": _real("registry_reconcile.yml"), ".github/workflows/morning_status.yml": ( "name: digest\non:\n schedule:\n - cron: \"0 6 * * *\"\n" "jobs:\n d:\n runs-on: ubuntu-latest\n steps:\n" @@ -117,6 +118,9 @@ def _real(name): # rule 9c: checks out PyAutoLabs/PyAutoBrain for the dashboard renderer, # which a freshly-spawned org does not have. ".github/workflows/dashboard_refresh.yml", + # rule 9c: the online lifecycle leg — scheduled, and reads sibling-repo + # issue/PR state, so it can neither auto-run nor succeed on a fresh org. + ".github/workflows/registry_reconcile.yml", ".github/workflows/morning_status.yml", ".github/workflows/morning_health.yml", ".github/workflows/arxiv_papers.yml",