Skip to content

Plot per-step score magnitude at the end of a MAGIC run - #437

Open
luciaquirke wants to merge 7 commits into
mainfrom
score-trajectory-plot
Open

Plot per-step score magnitude at the end of a MAGIC run#437
luciaquirke wants to merge 7 commits into
mainfrom
score-trajectory-plot

Conversation

@luciaquirke

@luciaquirke luciaquirke commented Aug 22, 2026

Copy link
Copy Markdown
Collaborator

Every MAGIC run that saves scores now writes <run_path>/score_vs_step.png beside them: the per-step median log10|score| against training step. A MAGIC run takes the shuffled training documents batch_size at a time, so batch s is optimizer step s and the scores group back into per-step buckets. Steps a MAGIC backward left with no score are marked with a red band rather than plotted as absence.

matplotlib is optional, as the plot extra. A plotting failure warns and lets the run finish, since the scores are already saved. A run without matplotlib prints "Hint: run pip install matplotlib for an optional trajectory plot of scores" at startup and skips the plot.

The plot is emitted from save_magic_scores, where both scoring paths converge under global_rank == 0 and the scores are still in memory. batch_size comes from the run config, so nothing has to recover it from a finished run's config.yaml. There is no CLI command and no registry entry — bergson/__main__.py, bergson/cli/commands.py, and docs/cli.rst are untouched.

Extracted from #421, which bundled this plot with a --window step-normalisation feature. That feature is left out: its own LDS experiment found it is not a robust win (one run improved, one unchanged, one clearly hurt).

Testing

pytest tests/test_score_trajectory.py — 2 tests cover the feature: per_step_level's arithmetic (known levels over trailing axes, steps with no score), and save_magic_scores writing the PNG beside the scores. pytest tests/test_magic.py -k save_magic_scores — 2 passed, for the changed signature. The missing-matplotlib and nothing-to-plot paths were checked directly. pre-commit and pyright clean; Sphinx builds at 37 warnings, matching origin/main.

🤖 Generated with Claude Code

https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC

davidoj and others added 4 commits August 22, 2026 12:38
`bergson score_trajectory <run>` reads a finished MAGIC run's scores and
config, groups the rows into batch_size-sized optimizer steps, and writes
score_vs_step.png: the per-step median log10|score| against training step.
Steps a MAGIC backward left with no score at all are marked rather than
plotted as absence.

matplotlib is an optional `viz` extra, imported lazily so the rest of the
CLI never requires it.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Quantize the three figures to a 64-colour palette (249 KB -> 73 KB, visually
identical), tighten the magic.rst section, and fold three near-identical
plot-writes-a-png tests into one parametrized test.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
One for per_step_level's arithmetic, one end-to-end from a yaml step to the
written PNG over a pipeline-shaped run.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Every run that saves scores now writes score_vs_step.png beside them, so there
is no score_trajectory command and no CLI registry entry. batch_size comes from
the run config rather than being recovered from a finished run's config.yaml,
which retires batch_size_from_config. matplotlib stays optional -- the `plot`
extra -- and a run without it prints a hint at startup and skips the plot.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
@luciaquirke luciaquirke changed the title Add a score_trajectory report for per-step score magnitude Plot per-step score magnitude at the end of a MAGIC run Aug 22, 2026
luciaquirke and others added 3 commits August 22, 2026 14:55
Matches the wandb.init fallback in utils/logging.py: the scores are already
saved, so a plotting failure must not end the run.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_011TfX3ZiFEKHT2W252jPJwC
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants