prompt: jax-grad-local-vs-ci-assertions (#260) — route, root cause, fix

active.md

@@ -24,3 +24,56 @@
   separate /repo_cleanup sweep so a destructive branch delete never rides a code diff.
 - repos:
   - PyAutoHands: feature/hands-hygiene-leftovers
+
+## jax-grad-local-vs-ci-assertions
+- issue: https://github.com/PyAutoLabs/autolens_workspace_test/issues/260 (issued 2026-08-22)
+- session: https://claude.ai/code/session_01VEHLT33XpVcRt5YCJGLRMJ (web-github; no local worktree yet)
+- status: library-dev (RECLASSIFIED from workspace-dev — the fix is upstream in PyAutoArray;
+  autolens_workspace_test needs no change and no tolerance change)
+- worktree: ~/Code/PyAutoLabs-wt/jax-grad-local-vs-ci-assertions
+- prompt: active/jax_grad_local_assertions_fail_but_pass_in_ci.md
+- classification: workspace (single repo) — routes to /start_workspace
+- strategy: investigate-first. Bug Agent: severity=critical, scope=single-repo,
+  type=wrong-result, confidence=LOW. Reproduce and confirm root cause BEFORE patching.
+  Brain sizing disagreement: declared medium, derived large.
+- control: imaging/jax_grad/lp.py is the discriminator — the only script known to PASS in
+  CI and FAIL locally. Every A/B runs against it first (~41s).
+- ROOT CAUSE FOUND 2026-08-22 (no laptop needed; findings on issue #260):
+  PyAutoArray `util/dataset_util.py:72` `should_simulate()` is existence-only and asymmetric —
+  it force-regenerates under PYAUTO_SMALL_DATASETS=1 but never under full_datasets. `dataset/**`
+  is gitignored, so CI always simulates fresh and CANNOT hit this; locally the dir persists and
+  is never refreshed. Any prior smoke run (SMALL_DATASETS=1 is the default for every OTHER
+  script) rewrites the FITS at 15x15, and the next jax_grad run silently loads them under
+  full-resolution settings. All three failures reproduced exactly from a clean checkout
+  (pixelization eager/jit matches the report to 11 s.f.; regularization's tolerance vector
+  matches exactly). Full chain proven: fresh=PASS -> one SMALL run -> stale=FAIL -> rm -rf =PASS.
+- numpy FALSIFIED: lp.py byte-identical across numpy 2.2.6 / 2.4.6 / 2.5.2 (and across
+  1-core vs 4-core). The likelihood runs through JAX/XLA; numpy only does the FD bookkeeping.
+- NO tolerance change is warranted — all three asserts did their job on a genuinely invalid
+  dataset. assert_eager_jit_consistent's rtol=1e-10 is vindicated, not under-specified.
+- FIX SHIPPED to a branch 2026-08-22: PyAutoArray `claude/jax-grad-assertions-ci-hontn3` @ 5ef6eea.
+  `should_simulate` now also regenerates on the small->full transition, inferring the regime from
+  data.fits's shape (the cap emits EXACTLY 16x16). Design went through an adversarial review that
+  corrected three things: no pixel-scale check (not in the FITS header), `==` not `<=` (the cap
+  cannot emit 12x12), and data.fits BY NAME not first-FITS-in-dir (PSFs are legitimately 11x11 and
+  dataset/cluster/test/psf.fits is 5760 B — a glob would have deleted real data every run).
+  Verified: control-tested, 1168 tests pass, 12 new tests over all four regime transitions, and
+  end-to-end — poison then full-datasets run now PASSES where it raised. Steady state 0.64 ms.
+  NO PR opened (not requested); branch is pushed and ready.
+- SCOPE: imaging manifestation only. Point-source/weak-lensing (JSON, no FITS) and interferometer
+  (shape-invariant under the cap, so it fails SILENTLY) remain exposed; stated in the docstring.
+- follow-ups filed (not absorbed):
+  - PyAutoNerves#153 — stamp the regime at the single FITS writer funnel (fitsable.py:89); the only
+    discriminant that can catch the silent interferometer case. Kept out of the fix deliberately: it
+    changes a header card on every FITS the stack writes (round-trip tests, file-hash pins).
+  - PyAutoArray#470 — separate live defect: the small-datasets branch rmtree's
+    dataset/point_source/simple, which is COMMITTED and allowlisted in .gitignore:13, replacing it
+    with capped-solver garbage. Recoverable via git checkout but violates the allowlist invariant.
+- follow-up (separate defect, not this bug): autolens_workspace_test
+  `.github/scripts/smoke_install.sh:9` `pip install "jax<0.7" "jaxlib<0.7"` downgrades jax to
+  0.6.2 and conflicts with autonerves' jax>=0.7,<0.11; the run only lands on the intended
+  0.10.2 because the next line's [optional] extras pull it back up. CI is right by accident.
+- out-of-bounds: moving lp.py's evaluation point, adding skip_indices, or widening a
+  tolerance without a measured basis. All three mask the trap instead of removing it.
+- repos:
+  - PyAutoArray: claude/jax-grad-assertions-ci-hontn3

dashboard.html

@@ -38,7 +38,7 @@
 <body>
 <h1>📋 PyAutoMind Dashboard</h1>
 <p>Every task the Mind is holding. Tap a task's 📋 and its <code>/start_dev</code> command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.</p>
-<p class="muted">In flight 2 · Parked 3 · Planned 6 · Backlog 144 · <a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/dashboard.md">markdown version</a></p>
+<p class="muted">In flight 3 · Parked 3 · Planned 6 · Backlog 143 · <a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/dashboard.md">markdown version</a></p>
 <h2>Start here</h2>
 <h3>Highest priority <span class="facets">(filed as high) — showing 12 of 18</span></h3>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/triage/jax_zero_contour.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/triage/jax_zero_contour.md">TRIAGE: needs manual review before routing</a> — <span class="facets">medium · safe · high</span></p></div>
@@ -59,6 +59,7 @@ <h2>In flight <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/bl
 <p class="muted">Issued — each has an open GitHub issue and usually a branch.</p>
 <div class="task"><button class="copy" data-cmd="/start_dev active/16_transformed_message_factor_gradient_unpack.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/16_transformed_message_factor_gradient_unpack.md"><code>@PyAutoFit</code> <code>TransformedMessage.factor_gradient</code> crashes on first call</a> — <a href="https://github.com/PyAutoLabs/PyAutoFit/issues/1501">issue #1501</a> — <span class="facets">HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev active/hands_hygiene_leftovers.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/hands_hygiene_leftovers.md">Hands hygiene: expired announcement dead code, unregistered modules</a> — <a href="https://github.com/PyAutoLabs/PyAutoHands/issues/249">issue #249</a> — <span class="facets">library-dev</span></p></div>
+<div class="task"><button class="copy" data-cmd="/start_dev active/jax_grad_local_assertions_fail_but_pass_in_ci.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/active/jax_grad_local_assertions_fail_but_pass_in_ci.md">jax_grad scripts fail assertions locally that PASS in CI</a> — <a href="https://github.com/PyAutoLabs/autolens_workspace_test/issues/260">issue #260</a> — <span class="facets">library-dev (RECLASSIFIED from workspace-dev — the fix is upstream in PyAutoArray;</span></p></div>
 <h2>Parked <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/parked.md">markdown version</a></h2>
 <details>
 <summary>3 task(s)</summary>
@@ -77,7 +78,7 @@ <h2>Planned <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/blob
 <div class="task"><button class="copy" data-cmd="/route start the planned PyAutoMind task latent-nan-guard-honest-run — its record is in planned.md" aria-label="Copy the Claude command">📋</button><p><b>latent-nan-guard-honest-run</b></p></div>
 </details>
 <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree/main/draft">markdown version</a></h2>
-<p class="muted">144 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.</p>
+<p class="muted">143 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.</p>
 <details>
 <summary>feature — 28</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md">Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix</a> — <span class="facets">autoarray · medium · supervised · high</span></p></div>
@@ -110,15 +111,14 @@ <h2>Backlog <a class="mdsrc" href="https://github.com/PyAutoLabs/PyAutoMind/tree
 <div class="task"><button class="copy" data-cmd="/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/feature/pyautomind/repos-sync-config-stamper.md">Teach repos_sync --write to stamp organ config surfaces</a> — <span class="facets">pyautomind · hard · supervised · low</span></p></div>
 </details>
 <details>
-<summary>bug — 32</summary>
+<summary>bug — 31</summary>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jax_runtime_and_parity.md">Fix release JAX runtime compatibility and likelihood parity</a> — <span class="facets">health_fixes · too-large · supervised · high</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/jit_visualization_outputs.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/jit_visualization_outputs.md">Fix JIT quick-update visualization output regressions</a> — <span class="facets">health_fixes · too-large · supervised · high</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/numerical_inversion_failures.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/numerical_inversion_failures.md">Fix release-profile numerical inversion failures</a> — <span class="facets">health_fixes · too-large · supervised · high</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/health_fixes/samples_parameter_paths.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/health_fixes/samples_parameter_paths.md">Fix release result/sample parameter-path regressions</a> — <span class="facets">health_fixes · too-large · supervised · high</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md"><code>pixel_scales</code> given as an <code>int</code> (or <code>np.float64</code>) is never widened</a> — <span class="facets">autoarray · small · supervised · medium</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md">jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object</a> — <span class="facets">autofit · small · supervised · medium</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md">Heart script_timing baselines are orphaned by path moves and filled</a> — <span class="facets">pyautoheart · small · supervised · medium</span></p></div>
-<div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autolens_workspace_test/jax_grad_local_assertions_fail_but_pass_in_ci.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autolens_workspace_test/jax_grad_local_assertions_fail_but_pass_in_ci.md">jax_grad scripts fail assertions locally that PASS in CI</a> — <span class="facets">autolens_workspace_test · medium · supervised · medium</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/numba_kernel_shift_axes_swapped.md">Numba PSF gathers derive the y/x kernel shifts from the</a> — <span class="facets">autoarray · low · supervised · medium</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md">PyNUFFT dev extra is incompatible with current SciPy on Python</a> — <span class="facets">autoarray · small · supervised · normal</span></p></div>
 <div class="task"><button class="copy" data-cmd="/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md" aria-label="Copy the Claude command">📋</button><p><a href="https://github.com/PyAutoLabs/PyAutoMind/blob/main/draft/bug/autofit/loggaussian_prior_declares_own_support.md"><code>LogGaussianPrior</code> misreports its own support as <code>(-inf, inf)</code></a> — <span class="facets">autofit · small · supervised · normal</span></p></div>

dashboard.md

@@ -8,10 +8,10 @@ Every task the Mind is holding, on one page: what is in flight, what is parked,
 
 | Where | Count |
 |-------|------:|
-| [In flight](#in-flight) (`active/`) | 2 |
+| [In flight](#in-flight) (`active/`) | 3 |
 | [Parked](#parked) (`parked.md`) | 3 |
 | [Planned](#planned) (`planned.md`) | 6 |
-| [Backlog](#backlog) (`draft/`) | 144 |
+| [Backlog](#backlog) (`draft/`) | 143 |
 
 ## Start here
 
@@ -137,6 +137,14 @@ Issued — each has an open GitHub issue and usually a branch. The full record f
 
 </details>
 
+<details><summary>📋 <a href="active/jax_grad_local_assertions_fail_but_pass_in_ci.md">jax_grad scripts fail assertions locally that PASS in CI</a> — <a href="https://github.com/PyAutoLabs/autolens_workspace_test/issues/260">issue #260</a> — library-dev (RECLASSIFIED from workspace-dev — the fix is upstream in PyAutoArray;</summary>
+
+```
+/start_dev active/jax_grad_local_assertions_fail_but_pass_in_ci.md
+```
+
+</details>
+
 ## Parked
 
 Started or scoped, not currently in flight — resume by moving the row back to `active.md`. Full detail in [`parked.md`](parked.md).
@@ -229,7 +237,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
 
 ## Backlog
 
-**144** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below.
+**143** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below.
 
 <details>
 <summary><b>feature</b> — 28</summary>
@@ -461,7 +469,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
 </details>
 
 <details>
-<summary><b>bug</b> — 32</summary>
+<summary><b>bug</b> — 31</summary>
 
 <details><summary>📋 <a href="draft/bug/health_fixes/jax_runtime_and_parity.md">Fix release JAX runtime compatibility and likelihood parity</a> — health_fixes · too-large · supervised · high</summary>
 
@@ -519,14 +527,6 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
 
 </details>
 
-<details><summary>📋 <a href="draft/bug/autolens_workspace_test/jax_grad_local_assertions_fail_but_pass_in_ci.md">jax_grad scripts fail assertions locally that PASS in CI</a> — autolens_workspace_test · medium · supervised · medium</summary>
-
-```
-/start_dev draft/bug/autolens_workspace_test/jax_grad_local_assertions_fail_but_pass_in_ci.md
-```
-
-</details>
-
 <details><summary>📋 <a href="draft/bug/autoarray/numba_kernel_shift_axes_swapped.md">Numba PSF gathers derive the y/x kernel shifts from the</a> — autoarray · low · supervised · medium</summary>
 
 ```