prompt: file the multi_dataset/jax_likelihood XLA compile stall - #266
Merged
Conversation
multi_dataset/jax_likelihood/mge.py was parked out of the PR smoke gate (autolens_workspace_test#262) to unblock the pynufft removal, after hanging to the 300s cap in 4/4 jobs on a PR whose diff that gate never executes. Parking removes the symptom, so this prompt is what keeps the cause visible. It is not a new bug: no_run.yaml already records the sibling delaunay.py as an "intermittent XLA compile stall, family-wide" (autolens_workspace_test#245), and mge.py's signature matches exactly — 32s when it passes, otherwise no progress at all. The real cost is the accumulation: the whole multi_dataset/jax_likelihood family is now quarantined somewhere, three times over, without anyone having diagnosed the stall. The prompt asks for the stall to be characterised rather than the scripts sped up, and for the runner to leave a stack trace on timeout — today a hang produces no evidence, which is why it keeps recurring.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Files
draft/bug/autolens_workspace_test/multi_dataset_jax_likelihood_xla_stall.md.multi_dataset/jax_likelihood/mge.pywas parked out of the PR smoke gate in PyAutoLabs/autolens_workspace_test#262, to unblock the pynufft removal (#261) after it hung to the 300s cap in 4/4 jobs on a PR whose diff that gate never executes. Parking removes the symptom — this prompt is what keeps the cause visible.It is not a new bug.
config/build/no_run.yaml:87already records the siblingdelaunay.pyas an "intermittent XLA compile stall, family-wide" (autolens_workspace_test#245, which also hitag_test rectangular.py).mge.py's signature matches exactly: 32s standalone when it passes, otherwise no progress at all until the cap.The cost worth surfacing is the accumulation. The whole
multi_dataset/jax_likelihood/family is now quarantined somewhere —delaunay.pyanddelaunay_mgeandshared_preloads.pyinno_run.yaml,shared_preloads.pyand nowmge.pyinsmoke_tests.txt— across three separate incidents, and nobody has diagnosed the stall itself.So the prompt asks for two things a per-script speedup would not give:
faulthandleron SIGTERM at the cap) would settle it faster than more timing data.TIMEOUT, which is exactly why this has been quarantined three times without a diagnosis.Test Plan
python3 scripts/lifecycle.py check→lifecycle check: OKdashboard.md/dashboard.htmlregenerated viapyauto-brain intake --apply dashboard(149 prompts), not hand-editeddraft/bug/<target>/per the taxonomy, with the target repo as the second folderGenerated by Claude Code