From 00232aaedf8c739ae355acd575378c7c1903945b Mon Sep 17 00:00:00 2001
From: Claude Every task the Mind is holding. Tap a task's 📋 and its In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version TRIAGE: needs manual review before routing — medium · safe · high Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high Fix release-profile numerical inversion failures — health_fixes · too-large · supervised · high Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium Every task the Mind is holding. Tap a task's 📋 and its In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version In flight 2 · Parked 3 · Planned 6 · Backlog 145 · markdown version TRIAGE: needs manual review before routing — medium · safe · high The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high (none right now) latent-nan-guard-honest-run 144 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below. 145 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below. Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium Every task the Mind is holding. Tap a task's 📋 and its In flight 2 · Parked 3 · Planned 6 · Backlog 145 · markdown version In flight 3 · Parked 3 · Planned 6 · Backlog 144 · markdown version TRIAGE: needs manual review before routing — medium · safe · high📋 PyAutoMind Dashboard
/start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.Start here
-Highest priority (filed as high) — showing 12 of 18
+Highest priority (filed as high) — showing 12 of 17
Backlog
pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · mediumreconstruction_noise_map_with_covariance sqrt-NaNs every off-diagonal by construction — autoarray · small · supervised · medium📋 TRIAGE: needs manual review before routing — medium · safe · high
@@ -479,26 +479,26 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
📋 Fix release-profile numerical inversion failures — health_fixes · too-large · supervised · high
+📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
```
-/start_dev draft/bug/health_fixes/numerical_inversion_failures.md
+/start_dev draft/bug/health_fixes/samples_parameter_paths.md
```
📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
+📋
```
-/start_dev draft/bug/health_fixes/samples_parameter_paths.md
+/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md
```
pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium📋
+pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium📋
```
-/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md
+/start_dev draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md
```
reconstruction_noise_map_with_covariance sqrt-NaNs every off-diagonal by construction — autoarray · small · supervised · medium📋 PyAutoMind Dashboard
/start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.Start here
-Highest priority (filed as high) — showing 12 of 17
+Highest priority (filed as high) — showing 12 of 18
Highest priority (filed as high) — showing 12 of 17
einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · highQuick wins (small enough, and safe enough to run unattended)
In flight markdown version
@@ -77,7 +77,7 @@ Planned
Backlog markdown version
-feature — 28
Backlog
bug — 32
+bug — 33
+pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · mediumreconstruction_noise_map_with_covariance sqrt-NaNs every off-diagonal by construction — autoarray · small · supervised · mediumreconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — autoarray · small · supervised · medium📋 TRIAGE: needs manual review before routing — medium · safe · high
@@ -25,6 +25,14 @@ Every task the Mind is holding, on one page: what is in flight, what is parked,
📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
+
+```
+/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md
+```
+
+📋 Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high
```
@@ -105,14 +113,6 @@ Every task the Mind is holding, on one page: what is in flight, what is parked,
📋 Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high
-
-```
-/start_dev draft/test/autolens_workspace_developer/mge_jit_regression_rebaseline.md
-```
-
-feature — 28
@@ -461,7 +461,15 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
bug — 32
+bug — 33
+
+📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
+
+```
+/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md
+```
+
+📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high
@@ -495,7 +503,7 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned.
📋
+reconstruction_noise_map_with_covariance sqrt-NaNs every off-diagonal by construction — autoarray · small · supervised · medium📋
```
/start_dev draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md
diff --git a/draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md b/draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md
index 5088353c..ca4e1a70 100644
--- a/draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md
+++ b/draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md
@@ -1,4 +1,4 @@
-# `reconstruction_noise_map_with_covariance` sqrt-NaNs every off-diagonal by construction
+# `reconstruction_noise_map_with_covariance` — form the covariance properly, fix the sqrt
Type: bug
Target: autoarray
@@ -14,133 +14,159 @@ Status: draft
Found during the 2026-08-21 reproduction gate for
`draft/bug/health_fixes/numerical_inversion_failures.md` (record:
`complete/2026/08/numerical-inversion-failures.md`). That prompt alleged
-non-positive-definite inversion matrices; it was **refuted** (2/2 scripts pass).
-This warning was the one real defect the gate turned up, and the prompt itself
-flagged it as "worth its own PyAutoArray prompt".
+non-positive-definite inversion matrices and was **refuted**; this was the real
+defect the gate turned up.
-**Read this first if you are chasing an inversion conditioning bug:** these NaNs
-look exactly like evidence of a non-positive-definite matrix and are not. They
-are unconditional — they appear for *any* input matrix, however well-conditioned.
-This finding has now been mistaken for a conditioning symptom once; don't let it
-happen twice.
+**Scope note (2026-08-22).** Deeper research found the noise map is wrong in
+*four* distinct ways. This prompt owns the two that need no science decision —
+the numerics and the semantics. The estimator-level defects (the noise map
+describes a different estimator than the default solver, and ignores edge-zeroed
+pixels) are **`draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md`**,
+which is the larger and more consequential of the two. Do that one second; this
+one first, because it is small, safe and unblocks the other.
-## The defect
+## Defect A — elementwise sqrt NaNs every off-diagonal
-`autoarray/inversion/inversion/abstract.py:839-859`, verified on `main` @ `a6b07cd`
-(2026-08-22):
+`autoarray/inversion/inversion/abstract.py:839-859`, verified on `main` @ `a6b07cd`:
```python
@property
def reconstruction_noise_map_with_covariance(self) -> np.ndarray:
- """
- Returns the noise-map of the reconstruction as a two dimension matrix which accounts for the covariance
- of the noise between pixels.
-
- The diagonal of this matrix is the noise-map of the reconstruction, ...
- """
+ """... a two dimension matrix which accounts for the covariance of the noise between pixels."""
return np.sqrt(np.linalg.inv(self.curvature_reg_matrix))
```
-`np.sqrt` is applied **elementwise to the entire inverse matrix**. That inverse is
-the reconstruction covariance matrix `C = inv(curvature_reg_matrix)`, whose
-off-diagonal entries are covariances and are **generally negative**. So every
-negative off-diagonal becomes `NaN`, and each call emits
-`RuntimeWarning: invalid value encountered in sqrt`.
+`np.sqrt` is applied **elementwise to the entire inverse**. That inverse is the
+covariance matrix `C`, whose off-diagonals are covariances and are generally
+negative — so they become `NaN`, and each call emits
+`RuntimeWarning: invalid value encountered in sqrt`. Unconditional: any matrix
+with an anti-correlated pixel pair NaNs, however well-conditioned.
+
+**This does NOT affect the 1D noise map.** `np.sqrt` is elementwise, so it
+commutes with taking the diagonal — `np.diagonal(np.sqrt(C))[i] == sqrt(C[i,i])`.
+The off-diagonal NaNs never reach `reconstruction_noise_map`. Do not cite this
+defect as the cause of unreliable 1D noise maps; that is the sibling prompt.
+
+## Defect B — `np.linalg.inv` is the wrong routine, and the repo already says so
+
+The same file, 50 lines up, documents exactly this hazard for the log-det path
+(`abstract.py:805-806`):
-This is unconditional, not data-dependent: any covariance matrix with an
-anti-correlated pixel pair NaNs. Observed as 4x `RuntimeWarning` from a clean,
-passing run of
-`autogalaxy_workspace/scripts/interferometer/features/pixelization/galaxy_reconstruction.py`.
+> the analytically exact `pixels * log(coeff) - log det C` from a single Cholesky
+> of their covariance `C`, **avoiding the round-off of factorizing the explicitly
+> formed inverse** (which reaches ~1e-6 absolute in the evidence at
+> **cond(C) ~ 1e9 on clustered traced mesh vertices**)
-## Why it is worth fixing
+That reasoning was applied to the log-det and never to the noise map, which still
+forms the explicit inverse — of the same matrix, at the same conditioning, on the
+same clustered-mesh geometry.
-The property is public API and its docstring promises a matrix that "accounts for
-the covariance of the noise between pixels". It returns NaN in exactly the entries
-that carry that covariance — so the documented purpose of the property is the part
-that is broken. Any consumer reading off-diagonals gets NaN.
+Three consequences, all pointing the same way:
-Severity is bounded, and the bound should be stated honestly: **the science path is
-correct.** The 1D `reconstruction_noise_map` takes `np.diagonal(...)` of this
-property, and elementwise-sqrt commutes with taking the diagonal, so
-`diag(sqrt(C)) == sqrt(diag(C))` — the right answer. Only the covariance-aware
-consumer and the warning spam are hit.
+1. `np.linalg.inv` is LU-based. It exploits neither symmetry nor
+ positive-definiteness, both of which this matrix has (when it is well-posed).
+2. It **raises only on exactly-singular input.** Near-singular passes through with
+ amplified error, so a diagonal entry can come back negative — impossible for a
+ true PD inverse — and `sqrt` turns it into NaN, or leaves it barely positive and
+ yields a wildly wrong RMS. Silently.
+3. The reconstruction path never inverts: `reconstruction_positive_negative_from`
+ uses `xp.linalg.solve` and `fnnls_cholesky` uses
+ `slg.solve(..., assume_a="pos")`. The noise map is the only place in the
+ inversion that forms an explicit inverse.
-## The trap — read before changing the return value
+**This is already biting users.** `inversion_plots.py:395` wraps the noise map in
+`except np.linalg.LinAlgError` and writes the CSV column as NaN with a warning —
+a guard that exists because this fails in practice.
-`reconstruction_noise_map` (line 881) is **derived from this property's diagonal**:
+## The fix
+
+Option 1 from the original draft, chosen 2026-08-22: the property should return
+the actual covariance matrix, computed via Cholesky. `scipy` is already a hard
+dependency (`pyproject.toml`).
```python
-return np.diagonal(self.reconstruction_noise_map_with_covariance)
+from scipy.linalg import cho_factor, cho_solve
+
+@property
+def reconstruction_covariance_matrix(self) -> np.ndarray:
+ """The covariance matrix C = [F + λH]^-1 of the reconstruction."""
+ matrix = np.asarray(self.curvature_reg_matrix)
+ covariance = cho_solve(cho_factor(matrix), np.eye(matrix.shape[0]))
+ return 0.5 * (covariance + covariance.T) # remove rounding asymmetry
+
+@property
+def reconstruction_noise_map(self) -> np.ndarray:
+ """1D RMS noise: sqrt of the diagonal of the covariance matrix."""
+ return np.sqrt(np.diag(self.reconstruction_covariance_matrix))
```
-If the fix changes the diagonal's meaning — e.g. returning the raw covariance
-matrix `C` — then `reconstruction_noise_map` silently changes from **standard
-deviation to variance**. That is a science regression in the one path that is
-currently correct, and no existing test would catch it in a way that names the
-cause. Any fix must either keep this property's diagonal as standard deviations,
-or decouple `reconstruction_noise_map` from it and compute `sqrt(diag(C))`
-directly.
-
-## The decision this needs (do not guess it inline)
-
-What the off-diagonals *should* be is an API/science call, not an obvious repair.
-The plausible options:
-
-1. **Return `C` unchanged** (a true covariance matrix; diagonal = variances).
- Cleanest semantics, but breaks the "diagonal is the noise-map" docstring claim
- and springs the trap above — `reconstruction_noise_map` must be rewritten in
- the same change.
-2. **Sqrt the diagonal only**, leave off-diagonals as covariances. Preserves both
- docstring claims and every current caller, but the matrix has mixed units
- (std devs on the diagonal, variances off it), which is a strange object to hand
- a user.
-3. **Signed sqrt**, `sign(C) * sqrt(|C|)`. Keeps a consistent "root" scale
- throughout and preserves the diagonal, but is a non-standard quantity that
- needs a documented justification.
-
-Option 2 is the smallest, least disruptive change and is the natural reading of
-the existing docstring. Option 1 is the most defensible statistically. Put the
-choice to a human — this is a public-API contract, and the property's name should
-probably change with it if the units do.
+Why this shape:
+
+- **`cho_factor` raises `LinAlgError` on a non-PD matrix**, so the noise map now
+ fails loudly on exactly the matrices the reconstruction already rejects. Today
+ the two disagree: `solve` raises and resamples, `inv` returns garbage.
+- **`reconstruction_noise_map` is decoupled** and computes `sqrt(diag(C))`
+ directly. Today it is correct only *incidentally*, because sqrt happens to be
+ elementwise — change the matrix and it silently becomes a variance. Decoupling
+ removes that trap permanently.
+- **Off-diagonals become real covariances**, so the docstring's promise holds.
+
+**Naming.** `..._with_covariance` returning a covariance matrix should be
+`reconstruction_covariance_matrix`. Keep the old name as a `DeprecationWarning`
+alias returning the new matrix. Its values *do* change — diagonal from std-dev to
+variance, off-diagonals from NaN to covariances — but every off-diagonal consumer
+was reading NaN, so nothing correct can break. Note the change in the release
+notes regardless.
+
+Optional, only if profiling asks for it: if just the diagonal is needed,
+`diag(C)` is available from the Cholesky factor as the squared row-norms of
+`L^-1`, avoiding the full `n x n` product. Not worth the complexity up front —
+this is computed once per fit, not per-likelihood.
## Verification
-- Off-diagonals are finite for a well-conditioned `curvature_reg_matrix` with an
- anti-correlated pixel pair. **There is currently no such test** — the only
- assertion on this property is
- `test_autoarray/inversion/inversion/test_abstract.py:684`, which checks `[0, 0]`,
- a *diagonal* element. The off-diagonal NaNs are entirely untested; that gap is
- why this shipped.
-- No `RuntimeWarning: invalid value encountered in sqrt` is emitted. Consider
- running the regression test under `-W error::RuntimeWarning` so a regression
- fails loudly rather than warning quietly.
-- `reconstruction_noise_map` still returns `sqrt(diag(C))` — assert this
- explicitly, against a hand-computed value, in the same test. The existing
- assertion at `test_abstract.py:686` covers the values but not the invariant.
+- **Off-diagonals finite** for a well-conditioned matrix with an anti-correlated
+ pixel pair. **No such test exists today** — the only assertion on this property
+ (`test_autoarray/inversion/inversion/test_abstract.py:684`) checks `[0, 0]`, a
+ *diagonal* element. That gap is why this shipped.
+- **No `RuntimeWarning`.** Run the regression test under `-W error::RuntimeWarning`
+ so a regression fails rather than warns.
+- **`reconstruction_noise_map` still returns `sqrt(diag(C))`** — assert against a
+ hand-computed value, and assert the invariant explicitly, not just the numbers.
+- **The `inv`-vs-`cho_solve` A/B is required, not assumed.** The claim that `inv`
+ yields negative/inaccurate diagonals at realistic conditioning was reasoned from
+ the numerical properties and the repo's own `~1e-6 at cond ~ 1e9` note — it was
+ **not** measured. Probe real `curvature_reg_matrix` instances from a Delaunay
+ source fit and count negative and near-zero diagonal entries under each method.
+ If `inv` turns out to be fine at realistic conditioning, say so and drop the
+ claim — keep the Cholesky change on the raise-loudly argument alone.
- `test_autoarray/inversion/plot/test_inversion_plotters.py:82,110` monkeypatch
- this property to a singular matrix to force a `LinAlgError` and check plots/CSV
- degrade gracefully. Those must still pass — confirm the fix does not alter which
- exception escapes.
-- Check downstream consumers before shipping: this sweep covered **PyAutoArray
- only**, where the sole in-repo consumer is `reconstruction_noise_map`. Grep
- @PyAutoGalaxy and @PyAutoLens for `reconstruction_noise_map_with_covariance`
- before assuming the change is contained.
-
-Repro environment: `PYAUTO_SKIP_WORKSPACE_VERSION_CHECK=1`,
-`NUMBA_CACHE_DIR=/tmp/numba_cache`, `MPLCONFIGDIR=/tmp/matplotlib`,
-`PYAUTO_DISABLE_JAX=1`.
-
-## Also worth folding in
-
-`reconstruction_noise_map`'s docstring says it "is computed as the square root of
-the diagonal of the `reconstruction_noise_map_with_covariance` matrix". The code
-takes the diagonal of an already-square-rooted matrix — no second sqrt. The two
-are numerically equal today only because sqrt is elementwise, which is precisely
-the bug. Whichever option is chosen, that sentence needs rewriting to match.
+ this property to force a `LinAlgError` and check plots/CSV degrade gracefully.
+ Confirm the same exception still escapes — `cho_factor` also raises
+ `LinAlgError`, so this should hold, but assert it.
+- **Downstream:** this sweep covered PyAutoArray only, where the sole in-repo
+ consumer is `reconstruction_noise_map`. Grep @PyAutoGalaxy and @PyAutoLens for
+ `reconstruction_noise_map_with_covariance` before assuming containment.
+
+## Also fold in
+
+`reconstruction_noise_map`'s docstring claims it "is computed as the square root
+of the diagonal of the `reconstruction_noise_map_with_covariance` matrix". The
+code takes the diagonal of an already-square-rooted matrix — no second sqrt. The
+two agree today only because sqrt is elementwise, which is precisely the bug.
+Rewrite the sentence to match whatever ships.
+
+## Note on the JAX path
+
+This property uses bare `np`, not `self._xp`, so it is already numpy-only even
+under a JAX fit — a JAX `curvature_reg_matrix` is coerced via `__array__`, forcing
+a device→host sync. The scipy fix does not regress that (there was no JAX support
+to lose) but it does make it explicit. Add `np.asarray` at the boundary, as above,
+and note the limitation in the docstring rather than leaving it implicit.
## Provenance
- Found during: `complete/2026/08/numerical-inversion-failures.md` (2026-08-22)
-- **Not** a symptom of the refuted non-positive-definite hypothesis in that
- prompt, nor of the earlier `complete/2026/07/pix-inversion-not-positive-definite.md`
- cluster (also refuted). Independent of both.
+- Sibling: `draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md`
+- **Not** a symptom of the refuted non-positive-definite hypothesis in that record,
+ nor of `complete/2026/07/pix-inversion-not-positive-definite.md` (also refuted).
diff --git a/draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md b/draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md
new file mode 100644
index 00000000..95a29c66
--- /dev/null
+++ b/draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md
@@ -0,0 +1,183 @@
+# The reconstruction noise map describes a different estimator than the default solver
+
+Type: bug
+Target: autoarray
+Repos:
+- @PyAutoArray
+Difficulty: medium
+Autonomy: human-required
+Priority: high
+Status: draft
+
+## Why this exists
+
+Found 2026-08-22 while researching
+`draft/bug/autoarray/reconstruction_noise_map_covariance_sqrt.md` (the numerics
+and semantics half). The question that started it: *why has the noise map on
+source reconstructions not always been reliable?*
+
+The answer is not the elementwise-sqrt bug — that only touches off-diagonals and
+provably cannot reach the 1D noise map. It is this: **the noise map is computed
+from a formula that describes an estimator PyAutoArray no longer uses by
+default.** `Autonomy: human-required` because the primary fix is a statistical
+decision about what the reported uncertainty *means*, not a code repair.
+
+## Defect 1 — the covariance formula assumes an unconstrained solve; the default is NNLS
+
+`abstract.py:859` computes `C = inv(curvature_reg_matrix)` = `[F + λH]^-1`. That
+is the posterior covariance of the **classical semi-linear inversion** — the
+unconstrained linear-Gaussian solution of Warren & Dye (2003) eq. 12, which is
+what `reconstruction_positive_negative_from` computes.
+
+But `config/general.yaml` ships:
+
+```yaml
+use_positive_only_solver: true # DEFAULT
+```
+
+So the default reconstruction is `fnnls_cholesky` — a **non-negative least
+squares** solve, minimising `||Zs - x||²` subject to `s >= 0`. That is a
+*constrained* estimator with an active set: `fnnls` maintains a passive set `P`
+and solves `slg.solve(ZTZ[P][:,P], ..., assume_a="pos")` on the free pixels only,
+pinning the rest at exactly zero.
+
+Imposing `s >= 0` is equivalent to truncating the Gaussian prior to the
+non-negative orthant. The posterior is then a **truncated** multivariate Gaussian,
+whose covariance is *not* `[F + λH]^-1`:
+
+- For pixels well inside the positive region, the constraint is inactive and
+ `[F + λH]^-1` is a good approximation.
+- For pixels near the boundary, truncation **reduces** the variance, so the
+ reported noise is systematically **overstated**.
+- For pixels pinned at exactly zero, the marginal posterior is not Gaussian at
+ all — it piles up at the boundary. The reported number is meaningless.
+
+**Why this bites source reconstructions hardest.** A lensed source is compact; most
+of the mesh is empty sky. So NNLS pins a *large fraction* of pixels at zero, the
+active set is large, and the unconstrained formula is worst exactly where it is
+most used. That matches the reported symptom precisely.
+
+The docstring makes an uncertainty claim, not a diagnostic one — "the RMS standard
+deviation of the noise in every pixel ... should be used for any scientific
+analysis (e.g. source reconstructions of strong lenses)" — so this matters.
+
+**The honest counter-argument, which a human should weigh:** if the noise map is
+meant only as a *diagnostic* of how well each pixel is constrained by data plus
+regularization, `[F + λH]^-1` is defensible for any solver, and the fix is to
+rewrite the docstring rather than the maths. Decide which of the two it is before
+writing code. That decision is the point of this prompt.
+
+## Defect 2 — the noise map ignores edge-zeroed pixels the reconstruction excluded
+
+`config/general.yaml` also ships `use_edge_zeroed_pixels: true`. Under it, the
+reconstruction (`abstract.py:511-539`) subsets the system:
+
+```python
+curvature_reg_matrix = self.curvature_reg_matrix[self.zeroed_ids_to_keep][:, self.zeroed_ids_to_keep]
+```
+
+solves the reduced problem, then scatters back with **exact zeros** at the zeroed
+pixels. `reconstruction_noise_map_with_covariance` inverts `self.curvature_reg_matrix`
+— the **full** matrix, respecting neither this reduction nor the separate
+`mapper_indices` reduction that `curvature_reg_matrix_reduced` applies for the
+log-det.
+
+Those excluded rows are, by the Delaunay mesh's own docstring, the
+"poorly constrained boundary vertices" whose zeroing exists to "stabilize the
+linear inversion" and "prevent poorly constrained boundary vertices from absorbing
+flux". **The noise map re-admits into an explicit inverse precisely the degenerate
+rows the reconstruction deliberately dropped to stay stable.**
+
+The user-visible result: a pixel whose reconstruction reads exactly `0` (meaning
+"not solved for") gets a noise value computed as though it had been solved.
+Reconstruction and noise map disagree about what those pixels mean.
+
+Scope: only bites when `zeroed_pixels > 0`. `Delaunay.__init__` defaults it to `0`,
+so this is opt-in per mesh — check `rectangular_rtu_adapt_density` and any
+workspace configs before sizing the blast radius.
+
+## Defect 3 — `use_edge_zeroed_pixels` is silently ignored unless the positive-only solver is on
+
+This is the "does the edge-pixel handling make sense next to positive-only?"
+question, and the answer is no. The control flow (`abstract.py:509-554`):
+
+```python
+if self.settings.use_positive_only_solver: # default True
+ if self.settings.use_edge_zeroed_pixels and self.has(cls=Mapper):
+ ...subset, fnnls, scatter back...
+ else:
+ return reconstruction_positive_only_from(FULL matrix)
+return reconstruction_positive_negative_from(FULL matrix) # edge-zeroing never consulted
+```
+
+`use_edge_zeroed_pixels` is nested **inside** the positive-only branch. Setting
+`use_positive_only_solver: false` — a reasonable thing to do, for speed or to
+permit negative values — **silently disables edge-zeroing too**, with no warning.
+The poorly-constrained boundary vertices come straight back into the solve and
+results change for a reason the config does not express.
+
+These are orthogonal concerns. Which parameters are *solvable* (edge-zeroing) is a
+statement about the mesh; which solver walks them is a separate choice. Edge-zeroing
+should apply to both branches, or the coupling should be made explicit and
+documented.
+
+## Suggested direction (not a decision — see Defect 1)
+
+If the noise map is to describe the estimator actually used, the covariance should
+be formed on the **same index set the reconstruction solved**, and scattered back:
+
+1. Determine the kept set exactly as `reconstruction` does — `zeroed_ids_to_keep`
+ under edge-zeroing, and, for the NNLS answer to Defect 1, further restricted to
+ the free set (pixels with `reconstruction > 0`).
+2. Cholesky-invert that submatrix (per the sibling prompt's `cho_factor` /
+ `cho_solve` fix).
+3. Scatter back into full shape. **Decide what the excluded pixels report** — `0`
+ matches the reconstruction's own convention and keeps plots working; `NaN` is
+ more honest ("never estimated") but breaks colourbars and the CSV. Recommend
+ `0` with an explicit docstring statement, since the reconstruction already
+ reports `0` there and consumers handle it.
+
+Restricting to the NNLS free set gives the covariance *conditional on the active
+set* — standard practice for constrained least squares, and a defensible,
+documentable choice. It is still an approximation: it ignores the uncertainty in
+the active set itself. Say so in the docstring rather than implying exactness.
+
+## Verification
+
+- **Reproduce the symptom first.** Take a real Delaunay source fit, compute the
+ noise map under the current code and under the free-set-restricted covariance,
+ and compare. Quantify how many mesh pixels are pinned at zero by NNLS — the
+ claim that this fraction is large for compact sources is **reasoned, not
+ measured**, and the whole prompt rests on it. If the fraction turns out small,
+ Defect 1 is a much smaller problem than stated here and should be re-graded.
+- Confirm reconstruction and noise map agree on which pixels were solved: every
+ pixel the reconstruction reports as an exact structural zero should be
+ identifiable in the noise map by the documented convention.
+- With `zeroed_pixels > 0`, assert the covariance is formed on the reduced matrix
+ — regression-test the shape and the scatter-back, not just values.
+- For Defect 3, assert `use_edge_zeroed_pixels: true` + `use_positive_only_solver:
+ false` either applies edge-zeroing or raises/warns. It must not silently ignore
+ the setting.
+- Check whether `curvature_reg_matrix_reduced`'s `mapper_indices` reduction should
+ apply to the noise map too. The log-det uses it; the noise map does not. Decide
+ deliberately — this is a third, separate index set and the inconsistency between
+ all three is itself a finding.
+
+## Prior art — read before starting
+
+- `complete/2026/08/numerical-inversion-failures.md` — this cluster's refutation.
+- `complete/2026/07/pix-inversion-not-positive-definite.md` — an earlier
+ non-positive-definite hypothesis, also refuted; documents the `GaussianKernel`
+ PD-guarantee `f1817af0`.
+- `autoarray/util/cholesky_funcs.py:50-80` — near-coincident mesh vertices make the
+ Schur pivot's sign depend on BLAS thread count. The degeneracy is real and
+ documented; this prompt is about not feeding it into an explicit inverse.
+- `abstract.py:805` — the repo already documents `~1e-6` evidence round-off from
+ "factorizing the explicitly formed inverse" at `cond(C) ~ 1e9` on clustered
+ traced mesh vertices.
+
+## Provenance
+
+- Found during: research for the sibling prompt, 2026-08-22.
+- Do the sibling first — it is small, needs no science decision, and its Cholesky
+ covariance helper is the building block this prompt reuses.
From c2fb5a5e931e42fa5e8832a55470584f0dd203d9 Mon Sep 17 00:00:00 2001
From: Claude reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — autoarray · small · supervised · medium📋 PyAutoMind Dashboard
/start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.Start here
Highest priority (filed as high) — showing 12 of 18
In flight Issued — each has an open GitHub issue and usually a branch.
@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract
reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — issue #468 — library-dev
latent-nan-guard-honest-run
145 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.
+144 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.
Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high
Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high
Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high
Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium
reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — autoarray · small · supervised · medium
jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium
Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium
jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium
reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — issue #468 — library-devreconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — autoarray · small · supervised · mediumEvery task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.
In flight 3 · Parked 3 · Planned 6 · Backlog 144 · markdown version
+In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version
TRIAGE: needs manual review before routing — medium · safe · high
@PyAutoFit TransformedMessage.factor_gradient crashes on first call — issue #1501 — HOLD — do not start dev. Fix-or-delete hangs off the PyAutoFit#1498 logpdf-contract
reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — issue #468 — library-dev
reconstruction_noise_map_with_covariance — form the covariance properly, fix the sqrt — issue #468 — library-devEvery task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.
In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version
TRIAGE: needs manual review before routing — medium · safe · high
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high
Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high
Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high
Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high
einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high
Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high
Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high
(none right now)
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high
Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high
Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium
jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium
Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · medium
jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium
Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium
PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal
Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.
In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version
TRIAGE: needs manual review before routing — medium · safe · high
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high
Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high
Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high
Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high
einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high
Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high
Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high
(none right now)
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high
Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high
Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium
jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium
Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · medium
jax_grad scripts fail assertions locally that PASS in CI — autolens_workspace_test · medium · supervised · medium
Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium
PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal
Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.
In flight 2 · Parked 3 · Planned 6 · Backlog 144 · markdown version
TRIAGE: needs manual review before routing — medium · safe · high
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high
Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high
Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high
Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high
einstein_radius_jit_from: replace static init_guess with a JAX-native seed finder — autogalaxy · too-large · supervised · high
Split lensing regimes: multi_galaxy / group / cluster (epic plan) — autolens · too-large · supervised · high
Re-baseline the MGE imaging JIT profiling regression value — autolens_workspace_developer · too-large · supervised · high
(none right now)
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · high
Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high
Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high
Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high
Priors & Messages cleanup — tracker — priors · too-large · supervised · normal
autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal
status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low
The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low