feat(magic): add score_trajectory report (per-step score-vs-step plot) - #421
Open
davidoj wants to merge 9 commits into
Open
feat(magic): add score_trajectory report (per-step score-vs-step plot)#421davidoj wants to merge 9 commits into
davidoj wants to merge 9 commits into
Conversation
`bergson score_trajectory <run>` reads a finished MAGIC run's scores/ dir and
config.yaml, groups the scores into training steps (batch_size rows per step),
and writes score_vs_step.png -- the per-step median log10|score| (the score
"level"). It is the visual companion to the metasmoothness score: a run whose
level sweeps tens of decades or oscillates ("rings") is not to be trusted.
`--window N` additionally overlays the step-normalisation curve the level would
be divided by, plus the residual level after normalising.
- new module bergson/magic/score_trajectory.py (plot + reusable per_step_level)
- Score_Trajectory CLI command; matplotlib added as an optional `viz` extra and
imported lazily so it is not required for the rest of the CLI
- docs: magic.rst "Score trajectory" section + cli.rst entry
- tests: CPU-only, covering per_step_level and PNG output
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
for more information, see https://pre-commit.ci
Illustrate the Score-trajectory section with two real pythia-160m MAGIC runs (per-step score level spans ~35-37 decades, generated by the same plotting code as `bergson score_trajectory`), and add a "Does normalising scores help?" section with the LDS results: step-normalisation helped one cell, was neutral on another, and hurt a third -- so it stays an opt-in `--window` overlay, not a default. Caveats (pythia-160m, N=5, no noise floor) noted. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…laim - Match the terse module / function / command docstring style used elsewhere in the package (one-line summary + short note, no Args/Returns sections). - Drop the claim that score instability tracks metasmoothness -- metasmoothness does not always predict it. The trajectory is a direct view of score magnitude, described as such in the module docstring and the docs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ults - Score trajectory: contrast a healthy run (SmolLM2-360M, ~6 decades, all steps live) with a pathological one (pythia-160m, ~37 decades, ~1550 dead steps); the pythia figure is now level-only. - Does normalising scores help?: motivate from the shuffled-document expectation; correct the run set (a ~7B deep-ignorance run + two pythia-160m cells, not "three pythia cells"); add the p1 trajectory (the run where it helped, least pathological, metasmoothness ~0.88); state we don't ship normalisation as a utility but it's easy to DIY; note it may help more in less-extreme cases; fix caveats to scope pathology to the pythia runs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Shuffling is the usual default but not universal ("if we train on shuffled
documents"), and the level only *sometimes* drifts far (metasmoothness knobs can
reduce it, and metasmoothness does not perfectly predict drift).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the per-run pathology rundown and N=5 caveats with a single summary: normalising more often helped when other health signs agreed (high metasmoothness score, scores that don't climb to NaN regions). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Contributor
Author
|
I added a plot to check the stability of the score as it's a useful health check on the attributed scores. I also added a documentation section explaining some brief results with score normalisation. I didn't add score normalisation as a feature because it seems like it's not that helpful and sometimes harmful, but it's straightforward to implement so I figure if it's in the docs people can just ask their assistant to normalise the scores if they want to try it. |
Add a deep-ignorance r32 trajectory: metasmoothness 0.99 (near-perfect) yet the level still drifts ~7 decades and rings periodically. Reinforces that the score-trajectory check and the metasmoothness score are related but distinct. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace SmolLM2-360M (which spikes mid-run and rings late) with magic_medmcqa_di_strong -- a deep-ignorance strong-filter MCQA run whose level holds a tight band near -6.5 dex for all 7k steps, dipping only in the final tail. It's also the `magic` example the pipeline docs build. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Collaborator
|
This is a good experiment but 1/3 wins and the win going from .7ish to .8ish seems maybe too low to merge to the main library if it would give people false hope hmmmm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
bergson score_trajectory <run>— an opt-in report you run after a MAGIC backward pass to see how attribution-score magnitude behaves across training.It reads a finished run's
scores/dir andconfig.yaml, groups the scores into training steps (batch_sizerows per step), and writes<run_path>/score_vs_step.png: the per-step medianlog10|score|(the score level) vs training step.Why
The docs' Metasmoothness section says the game is "identify and manage spikes", and
bergson metasmoothnessgives a scalar health score — but there was no way to look at the per-step score behaviour. On real runs the level sweeps ~tens of decades and can "ring"; this plot makes that visible at a glance, as the visual companion to the metasmoothness number.--window Noverlays the window-N step-normalisation curve the level would be divided by (and the residual level, which should sit near 0).Changes
bergson/magic/score_trajectory.py—plot_score_trajectory()+ a reusableper_step_level()(pools trailing token/query axes into the step).Score_TrajectoryCLI command wired into__main__.vizextra (and todev), imported lazily so the rest of the CLI never requires it.magic.rst+ acli.rstentry.per_step_level(known levels, trailing-axis pooling, dead steps) and end-to-end PNG output.Testing
pytest tests/test_score_trajectory.py— 4 passed;tests/test_cli.pystill green (command registry intact, 16 commands).--help, default, and--window) on a synthetic run dir.🤖 Generated with Claude Code