fix: prefer the FITS regime stamp over the shape heuristic in should_simulate

[Jammy2211]

Summary

Read side of PyAutoNerves#153. Merge PyAutoLabs/PyAutoNerves#154 first — see Merge ordering below.

should_simulate now takes the resolution regime from the SMALLDAT header card that PyAutoNerves stamps into every FITS the stack writes, keeping the shape heuristic from #471 as the fallback for datasets already on disk without one.

The three states are not interchangeable, and this predicate ends in shutil.rmtree:

stamp	action
SMALLDAT = T	delete — unless the data contradicts the card (below)
SMALLDAT = F	keep, unconditionally, without consulting shape
absent	unknown → fall back to shape. Never "full"

This closes the interferometer case, which shape provably cannot reach: its data.fits is (n_visibilities, 2) — 108384 × 2 for the committed sdp81 dataset — so a capped run writes identical NAXIS with garbage values and trips no assertion at all.

It also retires a false positive: a dataset legitimately 16×16 at full resolution was indistinguishable from a capped one by shape and was deleted on every run.

The safety guard, and why it exists

Adversarial review found — and reproduced — a data-loss regression in the first cut of this change, and it is the most important thing in this PR.

The stamp and this predicate were not about the same proposition. The stamp records "the env var was set in the writing process". should_simulate acted on it as "this data is capped, therefore stale and disposable." The library already makes those diverge: Kernel2D.from_gaussian passes respect_small_datasets=False (convolver.py:729) so a PSF written under the cap is full resolution; Interferometer.from_fits never caps; and anyone converting real telescope data in a shell exporting PYAUTO_SMALL_DATASETS=1 — the documented harness default — stamps T on genuinely full-resolution data.

Reproduced end to end: a 300×300 image written under the cap, then read in a full run, was deleted — and the pre-stamp shape heuristic had explicitly refused to delete it. That made the first cut a strict weakening of the safety property #471 established, not a residual risk carried over.

Every capped 2D image is rewritten to exactly SMALL_DATASETS_SHAPE_NATIVE, so SMALLDAT=T on an image larger than the cap in both axes is a self-contradiction — and a predicate ending in rmtree resolves a contradiction toward keep. Both axes, never either: interferometer data.fits has NAXIS1 = 2, so an "either axis" test would refuse to delete the one family this whole change exists for.

Verified lossless against the real committed datasets:

on-disk data.fits	before guard	after guard
capped imaging (16,16)	delete	delete
interferometer sdp81 (108384,2)	delete	delete
real imaging 151×151 / 209×209 / 300×300	delete	keep

API Changes

None — internal changes only.

should_simulate's signature is unchanged and its contract is unchanged for every dataset already on disk (no stamp → prior shape behaviour). Behaviour changes only for stamped datasets, and every change is in the safe direction except the intended new detection of capped interferometer data.

Test Plan

• python -m pytest test_autoarray/ — 1101 passed, 64 skipped, 0 failed
• Data-loss scenario reproduced against the unguarded version, then verified closed at HEAD
• Guard proven lossless: (16,16) and (108384,2) still delete; 151×151, 209×209, 300×300 now kept
• Non-boolean card reads as unknown — bool("F") is True, so a hand-edited string card must never be coerced into a delete
• Missing / unreadable / corrupt data.fits reads as unknown → keep
• Shape fallback still exercised (unstamped 16×16 still deletes), so it cannot rot into dead code
• PSF-carrying dataset not deleted — guards feat: add title_prefix support to subplot_imaging_dataset #260's "never glob *.fits" trap
• Green against both the stamped autonerves and a simulated pre-stamp one
• Tree clean with PYAUTO_SMALL_DATASETS=1 exported and unset

Merge ordering

Merge PyAutoLabs/PyAutoNerves#154 first, and release it, before this.

pyproject.toml:30 floors autonerves>=2026.8.22.1, and that is currently the newest release on PyPI — i.e. the newest legitimately-resolvable autonerves predates the stamp. This is not a blocker: with an older autonerves the writer emits no card, readers see absent → unknown → the existing shape fallback, which is exactly the designed degradation. But until a stamped autonerves is released, this change has nothing to read and the interferometer case stays open.

The header key is duplicated here rather than imported for that same reason — an import would hard-fail against a legitimately-resolved older autonerves, whereas an absent card degrades safely. It is a cross-repo wire-format contract with no compile-time link; any rename must be treated as such.

The tests deliberately force stamps with _set_stamp rather than relying on aa.output_to_fits to write one, so this suite tests this repo's reader and passes under either autonerves version. The writer is tested in test_autonerves/test_fitsable.py.

Known gap, stated rather than implied

This reads <dataset_path>/data.fits and nothing else — roughly 228 of the 253 should_simulate call sites in autolens_workspace. The rest get no verdict and fail safe to keep: datacube (channel_XXX/ subdirs), multi_dataset ({waveband}_data.fits), sample (dataset_N/), and the two FITS-less directories.

Widening the lookup is deliberately not done here. Every trap recorded in autolens_workspace_test#260 was a widened match hitting a file it should not have — a bare *.fits glob would delete every PSF-carrying dataset on every run. Growing a destructive predicate's reach is its own change with its own review.

Two corrections to the original issue text, both verified: point-source datasets are not JSON-only — they write a top-level data.fits and are covered normally. And dataset/weak/simple, the family the issue named, is regime-invariant (nothing in its write path reads the env var), so there is no bug there to fix. The one genuine JSON exposure, point_source/multiple_sources, is excluded from harness execution pending PyAutoLens#480 and is filed as a follow-up with both expiring justifications written down as explicit re-check triggers.

Full API Changes (for automation & release notes)

Removed / Renamed / Changed Signature

None.

Added (private)

• autoarray.util.dataset_util._small_datasets_stamp_on_disk(dataset_path) — tri-state True/False/None read of SMALLDAT from data.fits. Returns None for missing, unreadable, absent-card and any card that is not a genuine FITS boolean.
• autoarray.util.dataset_util._stamp_contradicted_by_shape(dataset_path) — True when the card claims capped but the shape says otherwise (larger than the cap in both axes). Unknown shape counts as not contradicted, since this guard only ever blocks a deletion.
• autoarray.util.dataset_util.SMALL_DATASETS_HEADER_KEY — "SMALLDAT", duplicated from autonerves by design (see Merge ordering).

Changed Behaviour

• should_simulate(dataset_path) — full-regime branch prefers the stamp, corroborates a destructive T against the data, and falls back to the shape heuristic when unstamped. Small-regime branch unchanged.

Migration

None. Unstamped datasets behave exactly as before this change.

Generated by the PyAutoLabs agent workflow.

---

Generated by Claude Code