Skip to content

feat(magic): add score_trajectory report (per-step score-vs-step plot) - #421

Open
davidoj wants to merge 9 commits into
mainfrom
score-trajectory-report
Open

feat(magic): add score_trajectory report (per-step score-vs-step plot)#421
davidoj wants to merge 9 commits into
mainfrom
score-trajectory-report

Conversation

@davidoj

@davidoj davidoj commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What

Adds bergson score_trajectory <run> — an opt-in report you run after a MAGIC backward pass to see how attribution-score magnitude behaves across training.

It reads a finished run's scores/ dir and config.yaml, groups the scores into training steps (batch_size rows per step), and writes <run_path>/score_vs_step.png: the per-step median log10|score| (the score level) vs training step.

bergson score_trajectory runs/my-magic             # -> runs/my-magic/score_vs_step.png
bergson score_trajectory runs/my-magic --window 5  # + overlay the step-norm curve & residual

Why

The docs' Metasmoothness section says the game is "identify and manage spikes", and bergson metasmoothness gives a scalar health score — but there was no way to look at the per-step score behaviour. On real runs the level sweeps ~tens of decades and can "ring"; this plot makes that visible at a glance, as the visual companion to the metasmoothness number. --window N overlays the window-N step-normalisation curve the level would be divided by (and the residual level, which should sit near 0).

Changes

  • bergson/magic/score_trajectory.pyplot_score_trajectory() + a reusable per_step_level() (pools trailing token/query axes into the step).
  • Score_Trajectory CLI command wired into __main__.
  • matplotlib added as an optional viz extra (and to dev), imported lazily so the rest of the CLI never requires it.
  • Docs: a "Score trajectory" section in magic.rst + a cli.rst entry.
  • Tests: CPU-only, covering per_step_level (known levels, trailing-axis pooling, dead steps) and end-to-end PNG output.

Testing

  • pytest tests/test_score_trajectory.py — 4 passed; tests/test_cli.py still green (command registry intact, 16 commands).
  • pre-commit (black + ruff) clean; pyright (CI-pinned 1.1.378) clean on changed files.
  • CLI smoke-tested end-to-end (--help, default, and --window) on a synthetic run dir.

🤖 Generated with Claude Code

davidoj and others added 7 commits August 11, 2026 12:24
`bergson score_trajectory <run>` reads a finished MAGIC run's scores/ dir and
config.yaml, groups the scores into training steps (batch_size rows per step),
and writes score_vs_step.png -- the per-step median log10|score| (the score
"level"). It is the visual companion to the metasmoothness score: a run whose
level sweeps tens of decades or oscillates ("rings") is not to be trusted.
`--window N` additionally overlays the step-normalisation curve the level would
be divided by, plus the residual level after normalising.

- new module bergson/magic/score_trajectory.py (plot + reusable per_step_level)
- Score_Trajectory CLI command; matplotlib added as an optional `viz` extra and
  imported lazily so it is not required for the rest of the CLI
- docs: magic.rst "Score trajectory" section + cli.rst entry
- tests: CPU-only, covering per_step_level and PNG output

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Illustrate the Score-trajectory section with two real pythia-160m MAGIC runs
(per-step score level spans ~35-37 decades, generated by the same plotting code
as `bergson score_trajectory`), and add a "Does normalising scores help?"
section with the LDS results: step-normalisation helped one cell, was neutral on
another, and hurt a third -- so it stays an opt-in `--window` overlay, not a
default. Caveats (pythia-160m, N=5, no noise floor) noted.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…laim

- Match the terse module / function / command docstring style used elsewhere in
  the package (one-line summary + short note, no Args/Returns sections).
- Drop the claim that score instability tracks metasmoothness -- metasmoothness
  does not always predict it. The trajectory is a direct view of score
  magnitude, described as such in the module docstring and the docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ults

- Score trajectory: contrast a healthy run (SmolLM2-360M, ~6 decades, all steps
  live) with a pathological one (pythia-160m, ~37 decades, ~1550 dead steps);
  the pythia figure is now level-only.
- Does normalising scores help?: motivate from the shuffled-document expectation;
  correct the run set (a ~7B deep-ignorance run + two pythia-160m cells, not
  "three pythia cells"); add the p1 trajectory (the run where it helped, least
  pathological, metasmoothness ~0.88); state we don't ship normalisation as a
  utility but it's easy to DIY; note it may help more in less-extreme cases;
  fix caveats to scope pathology to the pythia runs.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Shuffling is the usual default but not universal ("if we train on shuffled
documents"), and the level only *sometimes* drifts far (metasmoothness knobs can
reduce it, and metasmoothness does not perfectly predict drift).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace the per-run pathology rundown and N=5 caveats with a single summary:
normalising more often helped when other health signs agreed (high metasmoothness
score, scores that don't climb to NaN regions).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@davidoj

davidoj commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

I added a plot to check the stability of the score as it's a useful health check on the attributed scores. I also added a documentation section explaining some brief results with score normalisation. I didn't add score normalisation as a feature because it seems like it's not that helpful and sometimes harmful, but it's straightforward to implement so I figure if it's in the docs people can just ask their assistant to normalise the scores if they want to try it.

@davidoj
davidoj requested a review from luciaquirke August 11, 2026 04:43
davidoj and others added 2 commits August 11, 2026 14:46
Add a deep-ignorance r32 trajectory: metasmoothness 0.99 (near-perfect) yet the
level still drifts ~7 decades and rings periodically. Reinforces that the
score-trajectory check and the metasmoothness score are related but distinct.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Replace SmolLM2-360M (which spikes mid-run and rings late) with
magic_medmcqa_di_strong -- a deep-ignorance strong-filter MCQA run whose level
holds a tight band near -6.5 dex for all 7k steps, dipping only in the final tail.
It's also the `magic` example the pipeline docs build.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@luciaquirke

Copy link
Copy Markdown
Collaborator

This is a good experiment but 1/3 wins and the win going from .7ish to .8ish seems maybe too low to merge to the main library if it would give people false hope hmmmm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants