From d8a3ec2e4b4f31ad10e7697350d7f9ee5ecfac2b Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 22 Aug 2026 19:31:51 +0000 Subject: [PATCH] prompt: file the multi_dataset/jax_likelihood XLA compile stall MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit multi_dataset/jax_likelihood/mge.py was parked out of the PR smoke gate (autolens_workspace_test#262) to unblock the pynufft removal, after hanging to the 300s cap in 4/4 jobs on a PR whose diff that gate never executes. Parking removes the symptom, so this prompt is what keeps the cause visible. It is not a new bug: no_run.yaml already records the sibling delaunay.py as an "intermittent XLA compile stall, family-wide" (autolens_workspace_test#245), and mge.py's signature matches exactly — 32s when it passes, otherwise no progress at all. The real cost is the accumulation: the whole multi_dataset/jax_likelihood family is now quarantined somewhere, three times over, without anyone having diagnosed the stall. The prompt asks for the stall to be characterised rather than the scripts sped up, and for the runner to leave a stack trace on timeout — today a hang produces no evidence, which is why it keeps recurring. --- dashboard.html | 69 ++--- dashboard.md | 264 +++++++++--------- .../multi_dataset_jax_likelihood_xla_stall.md | 87 ++++++ 3 files changed, 258 insertions(+), 162 deletions(-) create mode 100644 draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md diff --git a/dashboard.html b/dashboard.html index 88b55d6a..1168076e 100644 --- a/dashboard.html +++ b/dashboard.html @@ -38,7 +38,7 @@

📋 PyAutoMind Dashboard

Every task the Mind is holding. Tap a task's 📋 and its /start_dev command is on your clipboard — paste it into a Claude Code chat to route Claude straight to that task.

-

In flight 2 · Parked 3 · Planned 6 · Backlog 148 · markdown version

+

In flight 2 · Parked 3 · Planned 6 · Backlog 149 · markdown version

Start here

Highest priority (filed as high) — showing 12 of 18

TRIAGE: needs manual review before routing — medium · safe · high

@@ -77,40 +77,9 @@

Planned

latent-nan-guard-honest-run

Backlog markdown version

-

148 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

+

149 filed prompts, not started — sorted most-pickable first (priority, then size). 23 of them belong to an epic and are listed only under Epics below.

-feature — 28 -

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

-

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

-

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

-

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

-

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

-

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

-

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

-

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

-

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

-

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

-

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

-

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

-

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

-

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

-

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

-

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

-

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

-

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

-

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

-

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

-

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

-

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

-

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

-

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

-

Release board: local run_logs enrichment — pyautohands · small · supervised · low

-

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

-

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

-

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

-
-
-bug — 33 +bug — 34

Stamp the small-datasets regime at the FITS writer funnel — pyautonerves · large · supervised · high

Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high

Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high

@@ -136,6 +105,7 @@

Backlog

interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal

JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal

JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal

+

multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal

@PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal

Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal

@PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal

@@ -146,6 +116,37 @@

Backlog

interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens

+feature — 28 +

Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high

+

Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high

+

Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high

+

Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high

+

Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high

+

Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium

+

Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium

+

Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal

+

Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal

+

Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal

+

The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal

+

The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal

+

Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal

+

EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal

+

Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal

+

Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal

+

Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal

+

Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal

+

PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal

+

autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal

+

Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal

+

Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal

+

dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low

+

Release board: local run_logs enrichment — pyautohands · small · supervised · low

+

Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low

+

Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low

+

Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low

+
+
research — 17

Deep research: Can we speed up Delaunay in PyAutoArray? — autoarray · too-large · supervised · high

Census of priors and messages — confirmed bugs + redesign — autofit · too-large · supervised · high

diff --git a/dashboard.md b/dashboard.md index aefe2999..50d6f330 100644 --- a/dashboard.md +++ b/dashboard.md @@ -11,7 +11,7 @@ Every task the Mind is holding, on one page: what is in flight, what is parked, | [In flight](#in-flight) (`active/`) | 2 | | [Parked](#parked) (`parked.md`) | 3 | | [Planned](#planned) (`planned.md`) | 6 | -| [Backlog](#backlog) (`draft/`) | 148 | +| [Backlog](#backlog) (`draft/`) | 149 | ## Start here @@ -235,500 +235,508 @@ Scoped but not started; some are not yet prompt files. Full detail in [`planned. ## Backlog -**148** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below. +**149** filed prompts, not started. Each section is sorted most-pickable first (priority, then size). **23** of them belong to an epic and are listed only under [Epics](#epics) below.
-feature — 28 +bug — 34 -
📋 Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high +
📋 Stamp the small-datasets regime at the FITS writer funnel — pyautonerves · large · supervised · high ``` -/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md +/start_dev draft/bug/pyautonerves/small_datasets_regime_stamp_at_writer_funnel.md ```
-
📋 Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high +
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autoarray/rectangular_bilinear_rtu_mesh_split.md +/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md ```
-
📋 Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high +
📋 Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autoarray/numba_cpu_likelihood_kernel_cdf_fast_path.md +/start_dev draft/bug/health_fixes/jit_visualization_outputs.md ```
-
📋 Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high +
📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high ``` -/start_dev draft/feature/autolens_profiling/jax_compile_time_profiling.md +/start_dev draft/bug/health_fixes/samples_parameter_paths.md ```
-
📋 Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high +
📋 pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium ``` -/start_dev draft/feature/profiling/profiling_agent_jax_compile_time_scope.md +/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md ```
-
📋 Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium +
📋 jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium ``` -/start_dev draft/feature/autofit/prior_exit_sampler_coverage.md +/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md ```
-
📋 Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium +
📋 Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium ``` -/start_dev draft/feature/autofit/search_seed_reproducibility.md +/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md ```
-
📋 Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal +
📋 should_simulate rmtree's committed, gitignore-allowlisted datasets — autoarray · medium · supervised · medium ``` -/start_dev draft/feature/autoarray/multiwavelength_inversion.md +/start_dev draft/bug/autoarray/small_datasets_rmtree_of_committed_data.md ```
-
📋 Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal +
📋 Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium ``` -/start_dev draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md +/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md ```
-
📋 Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal +
📋 PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal ``` -/start_dev draft/feature/pyautomind/complete_archive_wiki.md +/start_dev draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md ```
-
📋 The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal +
📋 LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal ``` -/start_dev draft/feature/workspaces/group_los_halos.md +/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md ```
-
📋 The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal +
📋 autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal ``` -/start_dev draft/feature/workspaces/group_subhalo_sensitivity.md +/start_dev draft/bug/autofit/plot_functions_discard_kwargs.md ```
-
📋 Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal +
📋 TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal ``` -/start_dev draft/feature/autoarray/arcsecond_to_decimal.md +/start_dev draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md ```
-
📋 EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal +
📋 point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal ``` -/start_dev draft/feature/autofit/ep_analytic_updates.md +/start_dev draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md ```
-
📋 Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal +
📋 Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal ``` -/start_dev draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md +/start_dev draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md ```
-
📋 Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal +
📋 HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal ``` -/start_dev draft/feature/autolens_profiling/search_settings_estimation_infrastructure.md +/start_dev draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md ```
-
📋 Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal +
📋 PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal ``` -/start_dev draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md +/start_dev draft/bug/pyautobrain/pyautoconf_rename_functional_leftovers.md ```
-
📋 Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal +
📋 generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal ``` -/start_dev draft/feature/autoarray/rectangular_multi_submesh.md +/start_dev draft/bug/pyautohands/generate_rejects_autocti_after_deleting_notebooks.md ```
-
📋 PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal +
📋 autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal ``` -/start_dev draft/feature/autogalaxy/piemass_potential.md +/start_dev draft/bug/pyautoreduce/published_autoreduce_09_missing_312_floor.md ```
-
📋 autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal +
📋 aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal ``` -/start_dev draft/feature/autolens_jax_joss/autolens_jax_joss_benchmark_repo.md +/start_dev draft/bug/workspaces/aplt_output_drift_remaining_repos.md ```
-
📋 Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal +
📋 Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal ``` -/start_dev draft/feature/jax_substructure/5_prng_key_vmap_noise.md +/start_dev draft/bug/workspaces/jax_likelihood_pins_stale_by_1e4.md ```
-
📋 Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal +
📋 PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal ``` -/start_dev draft/feature/jax_substructure/6_deflection_equivalence_test.md +/start_dev draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md ```
-
📋 Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal +
📋 interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal ``` -/start_dev draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md +/start_dev draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md ```
-
📋 dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low +
📋 JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal ``` -/start_dev draft/feature/autogalaxy/dpie_sigma0_parameterization.md +/start_dev draft/bug/autolens/jax_point_source_point_smoke_sentinel.md ```
-
📋 Release board: local run_logs enrichment — pyautohands · small · supervised · low +
📋 JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal ``` -/start_dev draft/feature/pyautohands/release_board_run_logs_enrichment.md +/start_dev draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md ```
-
📋 Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low +
📋 multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) — autolens_workspace_test · medium · supervised · normal ``` -/start_dev draft/feature/workspaces/gallery_runner_missing_tiers.md +/start_dev draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md ```
-
📋 Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low +
📋 @PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal ``` -/start_dev draft/feature/autonomy/10_scheduled_runs.md +/start_dev draft/bug/priors/15_transformed_message_logpdf_jacobian.md ```
-
📋 Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low +
📋 Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal ``` -/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md +/start_dev draft/bug/health_fixes/release_timeout_policy.md ```
-
+
📋 @PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal -
-bug — 33 +``` +/start_dev draft/bug/priors/14_replace_transform_stack_with_bijectors.md +``` -
📋 Stamp the small-datasets regime at the FITS writer funnel — pyautonerves · large · supervised · high +
+ +
📋 Priors & Messages cleanup — tracker — priors · too-large · supervised · normal ``` -/start_dev draft/bug/pyautonerves/small_datasets_regime_stamp_at_writer_funnel.md +/start_dev draft/bug/priors/z_features.md ```
-
📋 Fix release JAX runtime compatibility and likelihood parity — health_fixes · too-large · supervised · high +
📋 autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal ``` -/start_dev draft/bug/health_fixes/jax_runtime_and_parity.md +/start_dev draft/bug/autolens/jax_likelihood_smoke_pins_stale.md ```
-
📋 Fix JIT quick-update visualization output regressions — health_fixes · too-large · supervised · high +
📋 status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low ``` -/start_dev draft/bug/health_fixes/jit_visualization_outputs.md +/start_dev draft/bug/pyautomind/status_sh_repos_missing_source.md ```
-
📋 Fix release result/sample parameter-path regressions — health_fixes · too-large · supervised · high +
📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low ``` -/start_dev draft/bug/health_fixes/samples_parameter_paths.md +/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md ```
-
📋 pixel_scales given as an int (or np.float64) is never widened — autoarray · small · supervised · medium +
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens ``` -/start_dev draft/bug/autoarray/pixel_scales_int_not_widened_to_tuple.md +/start_dev draft/bug/autolens/interferometer_release_leg_oom.md ```
-
📋 jax 0.11 breaks beta/gamma message log_partition under jit ('tuple' object — autofit · small · supervised · medium +
+ +
+feature — 28 + +
📋 Numba CPU likelihood phase 1: batched MGE convolution + operated-matrix — autoarray · medium · supervised · high ``` -/start_dev draft/bug/autofit/jax_011_message_log_partition_tuple_shape.md +/start_dev draft/feature/autoarray/numba_cpu_likelihood_mge_convolution_and_caching.md ```
-
📋 Heart script_timing baselines are orphaned by path moves and filled — pyautoheart · small · supervised · medium +
📋 Rectangular mesh split: Bilinear (fast CPU default) vs RTU (advanced/GPU) — autoarray · medium · supervised · high ``` -/start_dev draft/bug/pyautoheart/script_timing_baselines_orphaned_and_window_filled.md +/start_dev draft/feature/autoarray/rectangular_bilinear_rtu_mesh_split.md ```
-
📋 should_simulate rmtree's committed, gitignore-allowlisted datasets — autoarray · medium · supervised · medium +
📋 Numba CPU likelihood phase 2: kernel-CDF numba fast path (the — autoarray · large · supervised · high ``` -/start_dev draft/bug/autoarray/small_datasets_rmtree_of_committed_data.md +/start_dev draft/feature/autoarray/numba_cpu_likelihood_kernel_cdf_fast_path.md ```
-
📋 Numba PSF gathers derive the y/x kernel shifts from the — autoarray · low · supervised · medium +
📋 Profile and speed up JAX likelihood-function compile times (all use — autolens_profiling · large · supervised · high ``` -/start_dev draft/bug/autoarray/numba_kernel_shift_axes_swapped.md +/start_dev draft/feature/autolens_profiling/jax_compile_time_profiling.md ```
-
📋 PyNUFFT dev extra is incompatible with current SciPy on Python — autoarray · small · supervised · normal +
📋 Give the Profiling Agent a compile-time axis — the arc — profiling · large · supervised · high ``` -/start_dev draft/bug/autoarray/pynufft_scipy_pinv2_dev_extra.md +/start_dev draft/feature/profiling/profiling_agent_jax_compile_time_scope.md ```
-
📋 LogGaussianPrior misreports its own support as (-inf, inf) — autofit · small · supervised · normal +
📋 Which other searches need prior-support handling — coverage audit after — autofit · medium · supervised · medium ``` -/start_dev draft/bug/autofit/loggaussian_prior_declares_own_support.md +/start_dev draft/feature/autofit/prior_exit_sampler_coverage.md ```
-
📋 autofit.plot functions accept **kwargs and silently discard them — autofit · small · supervised · normal +
📋 Give PyAutoFit searches a seed — today no search can — autofit · medium · supervised · medium ``` -/start_dev draft/bug/autofit/plot_functions_discard_kwargs.md +/start_dev draft/feature/autofit/search_seed_reproducibility.md ```
-
📋 TEST_MODE bypass crashes on ordered-parameter assertion ties — autofit · small · supervised · normal +
📋 Can create a list of InversionMatrix objects for each dataset — autoarray · medium · supervised · normal ``` -/start_dev draft/bug/autofit/test_mode_bypass_ordered_assertion_ties.md +/start_dev draft/feature/autoarray/multiwavelength_inversion.md ```
-
📋 point.py JAX-vmap parity assert is non-deterministic under the smoke env — autolens · small · supervised · normal +
📋 Tune cluster-scale JOSS benchmarks toward their 5-minute targets — autolens_workspace · medium · supervised · normal ``` -/start_dev draft/bug/autolens/point_jax_vmap_parity_nondeterministic.md +/start_dev draft/feature/autolens_workspace/joss_cluster_benchmark_tuning.md ```
-
📋 Scripts derive geometry from a hardcoded pixel_scale while the dataset — autolens_workspace · small · supervised · normal +
📋 Token-light wiki index over the complete/ archive — pyautomind · medium · supervised · normal ``` -/start_dev draft/bug/autolens_workspace/script_local_pixel_scale_vs_dataset_pixel_scales.md +/start_dev draft/feature/pyautomind/complete_archive_wiki.md ```
-
📋 HowToGalaxy small API drifts: ellipse kwargs + plot_grid_lines (parked NEEDS_FIX) — howtogalaxy · small · supervised · normal +
📋 The imaging features/advanced/los_halos example needs improving and padding out before — workspaces · medium · safe · normal ``` -/start_dev draft/bug/howtogalaxy/small_api_drift_ellipse_and_plot_grid_lines.md +/start_dev draft/feature/workspaces/group_los_halos.md ```
-
📋 PyAutoConf rename leftovers in Brain functional surfaces — pyautobrain · small · supervised · normal +
📋 The imaging features/advanced/subhalo/sensitivity example needs improving and padding out before — workspaces · medium · safe · normal ``` -/start_dev draft/bug/pyautobrain/pyautoconf_rename_functional_leftovers.md +/start_dev draft/feature/workspaces/group_subhalo_sensitivity.md ```
-
📋 generate.py deletes notebooks/ before rejecting an unknown project — pyautohands · small · supervised · normal +
📋 Claude Development Prompt: Arcsecond Tick Label Decimal Placement — autoarray · large · supervised · normal ``` -/start_dev draft/bug/pyautohands/generate_rejects_autocti_after_deleting_notebooks.md +/start_dev draft/feature/autoarray/arcsecond_to_decimal.md ```
-
📋 autoreduce 0.9 on PyPI never got the Python 3.12 floor — pyautoreduce · small · supervised · normal +
📋 EP analytic updates — implement the four planned work packages — autofit · large · supervised · normal ``` -/start_dev draft/bug/pyautoreduce/published_autoreduce_09_missing_312_floor.md +/start_dev draft/feature/autofit/ep_analytic_updates.md ```
-
📋 aplt.Output stale-API drift in the remaining workspace repos — workspaces · small · supervised · normal +
📋 Remote-MCP deployment tiers (2 + 3) for the results-inspector server — autofit_assistant · large · human-required · normal ``` -/start_dev draft/bug/workspaces/aplt_output_drift_remaining_repos.md +/start_dev draft/feature/autofit_assistant/remote_mcp_deployment_tiers.md ```
-
📋 Three jax_likelihood pins are stale by ~1.24e-4 and fail the — workspaces · small · supervised · normal +
📋 Search settings-estimation + profiling infrastructure (n_starts / batch_size / n_batch) — autolens_profiling · large · supervised · normal ``` -/start_dev draft/bug/workspaces/jax_likelihood_pins_stale_by_1e4.md +/start_dev draft/feature/autolens_profiling/search_settings_estimation_infrastructure.md ```
-
📋 PROBE: is Adapt's 4th-power coefficient dependence (double square) intentional? — autoarray · medium · supervised · normal +
📋 Adopt oversampled PSFs in the start-here dataset chain (option a) — autolens_workspace · large · supervised · normal ``` -/start_dev draft/bug/autoarray/PROBE_adapt_double_square_coefficient.md +/start_dev draft/feature/autolens_workspace/oversampled_psf_dataset_adoption.md ```
-
📋 interferometer Delaunay pixelization — non-PD FitException in test-mode bypass — autolens · medium · supervised · normal +
📋 Follow-up to rectangular_adapt_cdf.md (issue #322) and Path A — autoarray · too-large · supervised · normal ``` -/start_dev draft/bug/autolens/interferometer_delaunay_nonpd_fitexception.md +/start_dev draft/feature/autoarray/rectangular_multi_submesh.md ```
-
📋 JAX point-source smoke sentinel: point.py returns -1e99 instead of -83.38 — autolens · medium · supervised · normal +
📋 PIEMass.potential_2d_from: implement the missing lensing potential — autogalaxy · too-large · supervised · normal ``` -/start_dev draft/bug/autolens/jax_point_source_point_smoke_sentinel.md +/start_dev draft/feature/autogalaxy/piemass_potential.md ```
-
📋 JIT cache not hit in modeling_visualization delaunay/rectangular scripts — autolens · medium · supervised · normal +
📋 autolens_jax_joss benchmark repo + real-data start_here pairing — autolens_jax_joss · too-large · supervised · normal ``` -/start_dev draft/bug/autolens/jit_cache_not_hit_modeling_visualization.md +/start_dev draft/feature/autolens_jax_joss/autolens_jax_joss_benchmark_repo.md ```
-
📋 @PyAutoFit TransformedMessage.logpdf/pdf omit the transform Jacobian — priors · medium · supervised · normal +
📋 Context: PyAutoLens issue #542 follow-up (Gap 1, deferred during the — jax_substructure · too-large · supervised · normal ``` -/start_dev draft/bug/priors/15_transformed_message_logpdf_jacobian.md +/start_dev draft/feature/jax_substructure/5_prng_key_vmap_noise.md ```
-
📋 Resolve release-profile timeout scripts deliberately — health_fixes · too-large · supervised · normal +
📋 Context: PyAutoLens issue #542 follow-up (Gap 2, deferred during the — jax_substructure · too-large · supervised · normal ``` -/start_dev draft/bug/health_fixes/release_timeout_policy.md +/start_dev draft/feature/jax_substructure/6_deflection_equivalence_test.md ```
-
📋 @PyAutoFit Refactor: replace hand-rolled AbstractDensityTransform with tfp.bijectors / numpyro.distributions.transforms — priors · too-large · supervised · normal +
📋 Once https://github.com/PyAutoLabs/PyAutoLens/issues/480 is fixed (PointSolver — workspaces · too-large · supervised · normal ``` -/start_dev draft/bug/priors/14_replace_transform_stack_with_bijectors.md +/start_dev draft/feature/workspaces/restore_multiple_sources_lensing_of_lens.md ```
-
📋 Priors & Messages cleanup — tracker — priors · too-large · supervised · normal +
📋 dPIE: optional central-dispersion (sigma_0) parameterization — autogalaxy · small · supervised · low ``` -/start_dev draft/bug/priors/z_features.md +/start_dev draft/feature/autogalaxy/dpie_sigma0_parameterization.md ```
-
📋 autolens_workspace_test jax_likelihood pins: 4 scripts fail smoke on main — autolens · low · supervised · normal +
📋 Release board: local run_logs enrichment — pyautohands · small · supervised · low ``` -/start_dev draft/bug/autolens/jax_likelihood_smoke_pins_stale.md +/start_dev draft/feature/pyautohands/release_board_run_logs_enrichment.md ```
-
📋 status.sh --repos sources a file that no longer exists — pyautomind · small · supervised · low +
📋 Gallery runner: add visualization_upper + decide the modeling_visualization_jit tier — workspaces · small · supervised · low ``` -/start_dev draft/bug/pyautomind/status_sh_repos_missing_source.md +/start_dev draft/feature/workspaces/gallery_runner_missing_tiers.md ```
-
📋 The reconstruction noise map describes a different estimator than the — autoarray · medium · human-required · low +
📋 Scheduled runs — overnight queue passes with a morning report — autonomy · medium · supervised · low ``` -/start_dev draft/bug/autoarray/reconstruction_noise_map_solver_mismatch.md +/start_dev draft/feature/autonomy/10_scheduled_runs.md ```
-
📋 interferometer/start_here.py OOM in nightly release-validation integrate leg — autolens +
📋 Teach repos_sync --write to stamp organ config surfaces — pyautomind · hard · supervised · low ``` -/start_dev draft/bug/autolens/interferometer_release_leg_oom.md +/start_dev draft/feature/pyautomind/repos-sync-config-stamper.md ```
diff --git a/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md b/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md new file mode 100644 index 00000000..8cf34472 --- /dev/null +++ b/draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md @@ -0,0 +1,87 @@ +# multi_dataset/jax_likelihood scripts hang to the timeout cap (XLA compile stall) + +Type: bug +Target: autolens_workspace_test +Repos: +- @autolens_workspace_test +- @PyAutoFit +Difficulty: medium +Autonomy: supervised +Priority: normal +Status: draft + +## Why this is filed now + +Filed 2026-08-22 after `multi_dataset/jax_likelihood/mge.py` was parked out of +the PR smoke gate (autolens_workspace_test#262) to unblock the pynufft removal +(#261). Parking removes the symptom from CI; **this prompt is what stops the +underlying stall from going quiet.** The whole +`multi_dataset/jax_likelihood/` family is now absent from the PR gate. + +## The bug + +Scripts in this family intermittently stop making progress and run to whatever +timeout cap applies, instead of completing in their normal tens of seconds. + +Already recorded for the sibling, in `config/build/no_run.yaml:87`: + +> `multi_dataset/jax_likelihood/delaunay.py` — NEEDS_FIX 2026-08-01 — hangs to +> the 1800s release cap in 3 of 5 release-integrate runs since 07-31 (18s when +> it passes; **intermittent XLA compile stall, family-wide** — ag_test +> `rectangular.py` also hit); quarantined to unblock the 2026-08-01 release; +> see autolens_workspace_test#245 + +`mge.py` now shows the identical signature at the 300s smoke cap: + +- 4/4 job failures on #261 — Python 3.12 and 3.13, twice each including a + re-run — always `TIMEOUT (300s) multi_dataset/jax_likelihood/mge.py`, + always 23/24 passed. +- Runs green standalone in **32 s** under its declared `ENV: jax + full_datasets` profile, simulating its dataset from scratch as CI does. So + it is not cap margin; it stops progressing. +- Not caused by #261: that diff is three files, none in `smoke_tests.txt`, and + `mge.py` references nothing it changed. +- Not dataset contamination: `mge.py` is the only gate script touching + `dataset/multi_dataset/lens_sersic`, and the runner gives each script its own + subprocess. + +Reproduced only in CI so far — never locally in isolation — which points at the +compile stall being load- or environment-sensitive rather than a code defect in +any one script. + +## Scope + +The family is now almost entirely quarantined, which is the real cost: + +- `delaunay.py` — NEEDS_FIX in `no_run.yaml` (release), #245 +- `delaunay_mge` — SLOW in `no_run.yaml` +- `shared_preloads.py` — SLOW in `no_run.yaml`, and disabled in + `smoke_tests.txt` since 2026-07-22 +- `mge.py` — disabled in `smoke_tests.txt` 2026-08-22 (this filing). + Deliberately **not** added to `no_run.yaml`: every observed failure is at the + 300 s smoke cap and there is no release-cap evidence, so release coverage is + intentionally left intact. + +## Task + +1. Characterise the stall rather than the scripts. It is family-wide and has + also hit `ag_test rectangular.py`, so a per-script speedup is not the fix. + Establish whether it is an XLA/jaxlib compile deadlock, a runner resource + limit, or a threading interaction — a hung run's stack (py-spy or + `faulthandler` on SIGTERM at the cap) would settle it far faster than more + timing data. +2. Consider making the runner dump a stack trace on timeout instead of just + reporting `TIMEOUT`. Today a hang produces no evidence at all, which is why + this has been quarantined three times without a diagnosis. +3. Once fixed, un-quarantine the family: re-enable `mge.py` and + `shared_preloads.py` in `smoke_tests.txt` and clear the `no_run.yaml` + entries that cite #245. + +## Acceptance + +- A root cause for the stall, or an explicit decision that it is an + infrastructure limit to be worked around rather than fixed. +- A hung script leaves diagnostic evidence behind. +- The `multi_dataset/jax_likelihood/` family is back under CI coverage, or its + absence is a recorded deliberate choice rather than an accumulation of + one-off quarantines.