Plot per-step score magnitude at the end of a MAGIC run - #437
Open
luciaquirke wants to merge 7 commits into
Open
Plot per-step score magnitude at the end of a MAGIC run#437luciaquirke wants to merge 7 commits into
luciaquirke wants to merge 7 commits into
Conversation
`bergson score_trajectory <run>` reads a finished MAGIC run's scores and config, groups the rows into batch_size-sized optimizer steps, and writes score_vs_step.png: the per-step median log10|score| against training step. Steps a MAGIC backward left with no score at all are marked rather than plotted as absence. matplotlib is an optional `viz` extra, imported lazily so the rest of the CLI never requires it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Quantize the three figures to a 64-colour palette (249 KB -> 73 KB, visually identical), tighten the magic.rst section, and fold three near-identical plot-writes-a-png tests into one parametrized test. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
One for per_step_level's arithmetic, one end-to-end from a yaml step to the written PNG over a pipeline-shaped run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Every run that saves scores now writes score_vs_step.png beside them, so there is no score_trajectory command and no CLI registry entry. batch_size comes from the run config rather than being recovered from a finished run's config.yaml, which retires batch_size_from_config. matplotlib stays optional -- the `plot` extra -- and a run without it prints a hint at startup and skips the plot. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Matches the wandb.init fallback in utils/logging.py: the scores are already saved, so a plotting failure must not end the run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every MAGIC run that saves scores now writes
<run_path>/score_vs_step.pngbeside them: the per-step medianlog10|score|against training step. A MAGIC run takes the shuffled training documentsbatch_sizeat a time, so batchsis optimizer stepsand the scores group back into per-step buckets. Steps a MAGIC backward left with no score are marked with a red band rather than plotted as absence.matplotlib is optional, as the
plotextra. A plotting failure warns and lets the run finish, since the scores are already saved. A run without matplotlib prints "Hint: runpip install matplotlibfor an optional trajectory plot of scores" at startup and skips the plot.The plot is emitted from
save_magic_scores, where both scoring paths converge underglobal_rank == 0and the scores are still in memory.batch_sizecomes from the run config, so nothing has to recover it from a finished run'sconfig.yaml. There is no CLI command and no registry entry —bergson/__main__.py,bergson/cli/commands.py, anddocs/cli.rstare untouched.Extracted from #421, which bundled this plot with a
--windowstep-normalisation feature. That feature is left out: its own LDS experiment found it is not a robust win (one run improved, one unchanged, one clearly hurt).Testing
pytest tests/test_score_trajectory.py— 2 tests cover the feature:per_step_level's arithmetic (known levels over trailing axes, steps with no score), andsave_magic_scoreswriting the PNG beside the scores.pytest tests/test_magic.py -k save_magic_scores— 2 passed, for the changed signature. The missing-matplotlib and nothing-to-plot paths were checked directly. pre-commit and pyright clean; Sphinx builds at 37 warnings, matchingorigin/main.🤖 Generated with Claude Code
https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC