diff --git a/complete/2026/08/bump-autonerves-floor.md b/complete/2026/08/bump-autonerves-floor.md new file mode 100644 index 00000000..2da50440 --- /dev/null +++ b/complete/2026/08/bump-autonerves-floor.md @@ -0,0 +1,141 @@ +- issue: https://github.com/PyAutoLabs/PyAutoArray/issues/482 +- completed: 2026-08-23 +- library-pr: https://github.com/PyAutoLabs/PyAutoArray/pull/483 (merged 0f75c3d) +- workspace-pr: none — no workspace change needed + +The floor `autonerves>=2026.8.22.1` predated the SMALLDAT regime stamp +(PyAutoNerves#153/#154), so a PyPI-installed autoarray resolved an autonerves +whose writer emits no card: `should_simulate` read the absent stamp as "unknown" +and fell back to the shape heuristic, which cannot see capped interferometer +datasets — the case the stamp exists for. Bumped to `2026.8.23.1`. + +**THE BLOCKER CLEARED ITSELF THE SAME DAY.** The prompt was filed 2026-08-22 +blocked on "the PyAutoNerves release". `autonerves 2026.8.23.1` was uploaded +2026-08-23T00:41:41Z, hours before this ran. Worth remembering as a pattern: a +prompt parked on an external release is worth re-checking on sight rather than +trusting its `Status:`. + +**Verified by unpacking both wheels, not by reading commit dates.** The prompt +named commits `39014b6` and `0ecefa0` and said "the release that carries them is +the floor to name" — but commit membership is not the same claim as wheel +contents. Unpacked from PyPI: `2026.8.22.1` has neither +`stamp_small_datasets_regime` nor `SMALL_DATASETS_HEADER_KEY` and still writes +the `[""]` header comment; `2026.8.23.1` has both plus the `""` fix. No release +sits between them, so it is provably the first. + +**THE PROMPT UNDERSTATED THE DUPLICATION ITEM.** It framed converting +`dataset_util.SMALL_DATASETS_HEADER_KEY` to an import as "optional and low +value... decide deliberately". It is not optional in either direction: the +comment justifying the duplication *names the floor* ("floors autonerves at a +release that predates the stamp, so an import would hard-fail"), so bumping the +floor makes that comment **factually false**. The comment had to change whatever +was decided about the literal. + +Decision: **keep the duplication, replace the reason.** A floor constrains +dependency *resolution* only. An editable checkout, `pip install --no-deps`, or a +hand-built virtualenv (the HPC one) can still put a pre-stamp autonerves on the +path. Under the literal that yields "card absent" -> shape fallback, the safe +direction and the same path every pre-stamp dataset on disk already takes. Under +an import it is an `ImportError` at module load. Silent-safe beats hard-fail; the +literal is not debt, it is a deliberate decoupling. + +**TRAPS** +- The other floor reason is still load-bearing. PyAutoLens#687/#702 (JAX in + autonerves' base dependencies) shares this pin; both versions carry + `jax>=0.7.0` in base `Requires-Dist`, so that condition has been met since + `2026.8.22.1` and its "bump once it exists" phrasing was stale. Both reasons + are now stated. Dropping either silently re-opens the other's failure mode. +- Do NOT remove the shape fallback or `_is_capped_at_the_current_cap`. The floor + governs what a fresh install *writes*; every dataset already on disk is + unstamped and depends on those paths. +- A `>=` floor does not constrain the upper end, so this bump reaches no new jax + range — `2026.8.23.1` widens its own cap to `<0.12.0`, but that was already + resolvable under the old floor. The bump raises the minimum only. + +**A TRACKED TEST ARTIFACT CHANGED UNDER US.** Running the suite dirtied +`test_autoarray/structures/arrays/files/array/output_test/array.fits` — the +newly-floored autonerves drops the `/ ['']` comment literal (PyAutoNerves#155), +so the committed copy no longer matched what a compliant install writes: + +``` +HEAD : PIXSCAY = 1.0 / [''] +WORKING : PIXSCAY = 1.0 +``` + +Verified cosmetic (identical cards, values, data, byte size 5760 -> 5760) and +committed, because the file is test *output* — `test_uniform_2d.py:201-208` +rmtrees the directory and rewrites it — so leaving it stale would dirty the tree +on every run under the new floor. Only 1 of 61 tracked `.fits` was affected, +because only it carries `header_dict` comment cards. + +This is a different problem from the one the parent task solved. That one fixed +fixtures that are genuinely *inputs* and became regime-dependent, via an autouse +conftest fixture. This is files that are *outputs* and should not be tracked at +all — and the convention already exists, applied inconsistently: `.gitignore` +names two individual generated files under `dataset/files/array/output_test/` +and misses thirteen others. + +**Follow-up filed** +- `draft/maintenance/libraries/untrack_generated_fits_test_artifacts.md` — the + 13 tracked output files across five `output_test/` directories, their writers, + and the two candidate fixes (`tmp_path` preferred over widening `.gitignore`). + Notes that `structures/arrays/test_uniform_1d.py:12` *reads* from an + `output_test` path and must be checked for a real input before untracking. + +**Method note.** Filed via `/intake`, which classified it low-confidence to +`triage/` at difficulty large (score 7) off a long description, and again at +medium off a short one. Re-homed by hand to `maintenance/libraries` and the body +written from a real inventory (`git ls-files | grep output_test` plus the +writers). The sizing inflation on prose-dense input is the same effect recorded +in the parent task and in `complete/2026/08/jax-grad-smoke-timeout-budget.md`. + +## Original prompt + +# Bump autoarray's autonerves floor once the regime stamp is released + +Type: maintenance +Target: libraries +Repos: +- @PyAutoArray +Difficulty: small +Autonomy: supervised +Priority: medium +Status: formalised + +Filed 2026-08-22 as the step that makes PyAutoNerves#153/#154 actually take +effect. Blocked on the PyAutoNerves release, not on any code. + +`autoarray/pyproject.toml:30` floors `autonerves>=2026.8.22.1`. That version is +currently the **newest release on PyPI** and it **predates the SMALLDAT regime +stamp**. So an autoarray installed from PyPI resolves an autonerves whose writer +emits no card at all: `should_simulate` sees an absent stamp, reads it as +"unknown", and falls back to the shape heuristic — which provably cannot see +capped interferometer datasets, the case the whole change exists for. + +Nothing is broken by that; it is the designed degradation. But until the floor +names a stamped release, the fix is inert for anyone who installs rather than +runs from a checkout. + +## Do this after PyAutoNerves is released + +1. Bump `"autonerves>=2026.8.22.1"` to the first release containing the stamp + (the commits are `39014b6` and `0ecefa0` on nerves main; the release that + carries them is the floor to name). +2. Keep the comment block above the pin current — it currently explains a + *different* historical reason for the floor (PyAutoLens#687/#702, JAX moving + into autonerves' base dependencies). Add the stamp reason rather than + replacing that one; both are now load-bearing. +3. **Do not remove the shape fallback in `should_simulate`.** The floor governs + what a fresh install *writes* going forward; it says nothing about datasets + already on disk, every one of which is unstamped. The fallback protects those + and must stay. Same for `_is_capped_at_the_current_cap` on the capped branch. + +## Note on the duplicated header key + +`autoarray.util.dataset_util.SMALL_DATASETS_HEADER_KEY` duplicates the `"SMALLDAT"` +literal rather than importing it from autonerves, *because* of this floor — an +import would hard-fail against a legitimately-resolved older autonerves. Once the +floor names a stamped release that objection disappears and the import becomes +safe. Converting it is optional and low value: the duplication is documented, and +a stale reader degrades to the fallback, which is the safe direction. Decide +deliberately rather than by reflex. diff --git a/complete/index.md b/complete/index.md index 115030fa..a27f281d 100644 --- a/complete/index.md +++ b/complete/index.md @@ -6,7 +6,7 @@ Token-light navigation over the finished-work records (schema: only then grep a dated bucket. Curators: edit the band between the CURATED markers; everything below GENERATED is rebuilt. -1084 records across 7 buckets. +1085 records across 7 buckets. ## Highlights @@ -36,6 +36,7 @@ _(curate hard-won records here — survives regeneration.)_ - [border-relocator-backend-parity](2026/08/border-relocator-backend-parity.md) — Isolated the native NumPy/JAX likelihood gap to mathematically non-unique PCA axes for a near-isotropic border… - [border-relocator-degenerate-pca](2026/08/border-relocator-degenerate-pca.md) — Stabilized near-isotropic border PCA axes with a deterministic axis-aligned branch at a scale-aware eigenvalue… - [brain-readability-pass](2026/08/brain-readability-pass.md) — auto-closed on merge +- [bump-autonerves-floor](2026/08/bump-autonerves-floor.md) - [circular-ell-comps-image-gradient](2026/08/circular-ell-comps-image-gradient.md) — Replaced the Sersic image path's Cartesian-to-polar ellipticity conversion with an algebraically equivalent Ca… - [cli-noise-autonerves-batch](2026/08/cli-noise-autonerves-batch.md) - [cli-noise-pyautofit-batch](2026/08/cli-noise-pyautofit-batch.md) diff --git a/dashboard.html b/dashboard.html index 183393e5..fa5f2627 100644 --- a/dashboard.html +++ b/dashboard.html @@ -168,7 +168,7 @@

Backlog

Bump autoarray's autonerves floor once the regime stamp is released — libraries · small · supervised · medium

+

Untrack the generated FITS test artifacts in autoarray — libraries · small · supervised · medium

smoke_install.sh's stale jax<0.7 pin — CI is on the right — ci · low · supervised · medium

autolens_workspace_developer rectangular experiments — Gut stash + rename — autolens_workspace_developer · small · supervised · normal

Defer the eager scipy.sparse import in derivative_util (~0.10 s of — libraries · small · safe · normal

diff --git a/dashboard.md b/dashboard.md index b06e13c8..5b605caf 100644 --- a/dashboard.md +++ b/dashboard.md @@ -887,10 +887,10 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
maintenance — 21 -
📋 Bump autoarray's autonerves floor once the regime stamp is released — libraries · small · supervised · medium +
📋 Untrack the generated FITS test artifacts in autoarray — libraries · small · supervised · medium ``` -/start_dev draft/maintenance/libraries/bump_autoarray_autonerves_floor_after_stamp_release.md +/start_dev draft/maintenance/libraries/untrack_generated_fits_test_artifacts.md ```
diff --git a/draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md b/draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md index 89b05f98..36891817 100644 --- a/draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md +++ b/draft/bug/pyautolens/point_source_json_datasets_record_no_regime.md @@ -75,6 +75,44 @@ no FITS". That is wrong: ordinary point-source datasets write a top-level `autolens_workspace_test#260` traps. This needs a naming-convention decision first, and that decision is the real work here. +## Re-check log + +**2026-08-23 — both facts re-verified, STILL BLOCKED. Do not build.** +Checked at the top of a `/start_dev` run; the gate in step 1 below said re-park, +so nothing was built, no issue was opened and no branch was cut. + +1. **PyAutoLens#480 is still open.** Created 2026-04-28, `updated_at` identical + to `created_at`, no assignees, `closed_by_pull_requests: 0`. Untouched in + ~4 months. +2. **`weak/simple` is still regime-invariant.** Its simulator calls + `simulator.via_tracer_from(tracer=tracer, grid=positions, name=dataset_name)` + over an explicit 1500-galaxy annulus with `np.random.default_rng(1)`. It does + not call `via_tracer_random_positions_from`, and the file contains no + `os.environ` read at all, so nothing reads `PYAUTO_SMALL_DATASETS`. + +Two path corrections found while re-checking — the originals below cost a search, +so they are fixed here rather than left to bite again: + +- The `no_run.yaml` exclusion is in **@autolens_workspace**, not + autolens_workspace_test (whose `config/build/no_run.yaml` has no + `multiple_sources` entry at all). Line numbers omitted deliberately: the + original `:41-42` had already drifted. Match on the entries themselves — + + ``` + - point_source/features/multiple_sources/simulator # Blocked by PyAutoLens #480: solver finds 0 positions for intermediate-plane source + - point_source/features/multiple_sources/modeling # Blocked by PyAutoLens #480: same root cause as simulator above + ``` + + Note this is **two** skipped scripts, simulator *and* modeling, not one. +- The script path is `scripts/point_source/features/multiple_sources/`, not + `dataset/point_source/multiple_sources` (that is the *output* directory). + Likewise the weak simulator is `scripts/weak/simulator.py` in + @autolens_workspace. + +Next re-check: when PyAutoLens#480 closes. That is the only trigger — fact 2 is a +standing invariant, not a countdown, and only breaks if someone switches +`weak/simulator.py` to the random-positions helper. + ## Suggested scope 1. Re-check both expiring facts above. If #480 is still open and `weak/simple` diff --git a/draft/maintenance/libraries/bump_autoarray_autonerves_floor_after_stamp_release.md b/draft/maintenance/libraries/bump_autoarray_autonerves_floor_after_stamp_release.md deleted file mode 100644 index efef7dce..00000000 --- a/draft/maintenance/libraries/bump_autoarray_autonerves_floor_after_stamp_release.md +++ /dev/null @@ -1,48 +0,0 @@ -# Bump autoarray's autonerves floor once the regime stamp is released - -Type: maintenance -Target: libraries -Repos: -- @PyAutoArray -Difficulty: small -Autonomy: supervised -Priority: medium -Status: formalised - -Filed 2026-08-22 as the step that makes PyAutoNerves#153/#154 actually take -effect. Blocked on the PyAutoNerves release, not on any code. - -`autoarray/pyproject.toml:30` floors `autonerves>=2026.8.22.1`. That version is -currently the **newest release on PyPI** and it **predates the SMALLDAT regime -stamp**. So an autoarray installed from PyPI resolves an autonerves whose writer -emits no card at all: `should_simulate` sees an absent stamp, reads it as -"unknown", and falls back to the shape heuristic — which provably cannot see -capped interferometer datasets, the case the whole change exists for. - -Nothing is broken by that; it is the designed degradation. But until the floor -names a stamped release, the fix is inert for anyone who installs rather than -runs from a checkout. - -## Do this after PyAutoNerves is released - -1. Bump `"autonerves>=2026.8.22.1"` to the first release containing the stamp - (the commits are `39014b6` and `0ecefa0` on nerves main; the release that - carries them is the floor to name). -2. Keep the comment block above the pin current — it currently explains a - *different* historical reason for the floor (PyAutoLens#687/#702, JAX moving - into autonerves' base dependencies). Add the stamp reason rather than - replacing that one; both are now load-bearing. -3. **Do not remove the shape fallback in `should_simulate`.** The floor governs - what a fresh install *writes* going forward; it says nothing about datasets - already on disk, every one of which is unstamped. The fallback protects those - and must stay. Same for `_is_capped_at_the_current_cap` on the capped branch. - -## Note on the duplicated header key - -`autoarray.util.dataset_util.SMALL_DATASETS_HEADER_KEY` duplicates the `"SMALLDAT"` -literal rather than importing it from autonerves, *because* of this floor — an -import would hard-fail against a legitimately-resolved older autonerves. Once the -floor names a stamped release that objection disappears and the import becomes -safe. Converting it is optional and low value: the duplication is documented, and -a stale reader degrades to the fallback, which is the safe direction. Decide -deliberately rather than by reflex. diff --git a/draft/maintenance/libraries/untrack_generated_fits_test_artifacts.md b/draft/maintenance/libraries/untrack_generated_fits_test_artifacts.md new file mode 100644 index 00000000..18810467 --- /dev/null +++ b/draft/maintenance/libraries/untrack_generated_fits_test_artifacts.md @@ -0,0 +1,93 @@ +# Untrack the generated FITS test artifacts in autoarray + +Type: maintenance +Target: libraries +Repos: +- @PyAutoArray +Difficulty: small +Autonomy: supervised +Priority: medium +Status: formalised + +Filed 2026-08-23 from the PyAutoArray#482/#483 floor bump, which tripped over +this. Not a bug — a git-hygiene defect that converts unrelated upstream changes +into binary diffs. + +## The defect + +`test_autoarray` writes test *output* into **tracked** paths. The tests rmtree +the directory, recreate it, write a FITS, and read it back — the file is never an +input expectation, yet it is committed. So any change to the autonerves FITS +writer rewrites a tracked binary and dirties the tree for every contributor. + +Demonstrated on #483: the PyAutoNerves#155 header-comment fix (`[""]` -> `""`) +silently changed `test_autoarray/structures/arrays/files/array/output_test/array.fits` +the moment the autonerves floor moved to a release carrying it — + +``` +HEAD : PIXSCAY = 1.0 / [''] +WORKING : PIXSCAY = 1.0 +``` + +— identical cards, values, data and byte size, but a modified tracked file that +had to be either committed into an unrelated PR or left dirtying the tree. It +was committed there; that was the least-bad option, not a good one. + +## Why this is worth fixing rather than absorbing + +The convention **already exists and is applied inconsistently**. `.gitignore` +lines 9-10 name two individual generated files: + +``` +test_autoarray/dataset/files/array/output_test/uv_wavelengths.fits +test_autoarray/dataset/files/array/output_test/visibilities.fits +``` + +so someone already hit this and patched the two files in front of them rather +than the pattern. **13 tracked files** across five `output_test/` directories are +still exposed: + +``` +test_autoarray/dataset/files/array/output_test/noise_map.fits +test_autoarray/dataset/files/arrays/output_test/{background_noise_map,background_sky_map, + exposure_time_map,image,noise_map,poisson_noise_map,psf}.fits +test_autoarray/structures/arrays/files/array/output_test/array.fits +test_autoarray/structures/arrays/files/output_test/{array,masked_array}.fits +test_autoarray/structures/arrays/files/output_test/values_test.dat +test_autoarray/structures/arrays/files/values/output_test/values_test.dat +``` + +Writers: `structures/arrays/test_uniform_2d.py:201-211`, +`structures/test_visibilities.py:96-107`, `mask/test_mask_2d.py:417`, +`dataset/interferometer/test_dataset.py:140`, `dataset/imaging/test_dataset.py:18`, +`structures/arrays/test_uniform_1d.py:12`. + +## Suggested scope + +1. Pick one of the two fixes and apply it uniformly — do **not** patch + file-by-file again: + - **`tmp_path`** (preferred): the tests already rmtree/recreate their output + dir, so pytest's `tmp_path` fixture is a near-drop-in and removes the + tracked path entirely. This is the durable fix. + - **`.gitignore` the directories** (`test_autoarray/**/output_test/`) plus + `git rm --cached` the 13 files, if converting the tests is judged too wide. + Cheaper, but leaves the tests writing into the source tree. +2. Confirm each candidate file is genuinely output before untracking it. The + inventory above was read off the writers, but + `structures/arrays/test_uniform_1d.py:12` *reads* from an `output_test` path — + check whether it consumes a file another test produced (ordering dependency) + or a committed one. If any file is a real input, it is not in scope and should + be moved out of `output_test/` instead. +3. Verify the suite passes from a clean checkout **and** twice in a row (a test + that silently depended on a committed artifact will fail on the first run + after untracking, not the second). +4. Leave the sibling repos alone. PyAutoGalaxy/PyAutoLens likely share the + pattern; check, and file separately rather than widening this. + +## Note + +The equivalent problem in the workspaces was solved differently — an autouse +conftest fixture per repo (see `complete/2026/08/small-datasets-regime-stamp.md`, +"Committed FITS fixtures became regime-DEPENDENT"). That solved *regime* +dependence of fixtures that are genuinely inputs. This is the different case: +files that are outputs and should not be tracked at all.