perf(rerank): judgments persist and replay — the judge stops re-buying opinions it already owns - #568
Merged
Merged
Conversation
…g opinions it already owns Approved rec #2 (2026-08-31 set). Measured on the live corpus: 42% of rerank judgments over 48h (808 provenance rows / 466 distinct items / 17 passes) re-judged items an earlier pass had already judged with the SAME model identity and prompt version — the feed's top band is sticky by construction, so every ~10-minute cycle re-bought identical verdicts. Schema 116 adds advisor_judgments — one row per (source_item_id, identity_hash, prompt_version), newest wins, ON DELETE CASCADE per the Phase-115 precedent. apply_llm_reranking now: - loads fresh-enough memo rows (20h window: judged content is immutable post-ingest; what drifts is the user's context, which re-judges daily), - LLM-judges only the remainder, and REPLAYS the memoized judgments through the reconciler — same adjustments, same explanations, zero API calls, - stamps provenance and calibration samples for FRESH judgments only — a replay re-entering the calibration set would double-count the sample and feed the fitter its own output (2026-08-11 incident class), - runs the uniform-advisor circuit breaker on fresh judgments only (its job is catching an advisor broken NOW; replays passed it in the pass that produced them), and memoizes only breaker-surviving passes, - keys strictly on identity_hash + prompt_version: a swapped model or bumped prompt never replays a predecessor's opinions. Effect: a stable top band costs one judgment per item per day instead of ~9, and the freed (paced) budget rotates to items the judge has never seen — coverage up, spend down. Scores for replayed items are identical to what a fresh judgment would produce, so no PIPELINE_VERSION bump. The memo lives as a #[path] submodule of analysis_rerank (the analysis_dedup precedent) — no lib.rs churn. rerank log line now carries fresh= and replayed= counts. Verification: cargo test --lib 4748 passed / 0 failed (5 new memo tests: freshness aging, identity/prompt isolation, upsert semantics); clippy --all-features -D warnings clean; fmt clean; check-file-sizes warn-only. Schema 116 coordinated with the fleet (Lane F confirmed no competing migration; 115 = #564). Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_019RTCgzAcaopjz3KZnaqNrj
runyourempire
enabled auto-merge (squash)
August 31, 2026 04:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Approved rec #2 from the 2026-08-31 recommendation set (operator green-lit).
The measured waste
42% of rerank judgments over the last 48h (808 provenance rows / 466 distinct items / 17 passes; adjacent-pass overlap 13/48) re-judged items an earlier pass had already judged with the same model identity and prompt version. The feed's top band is sticky by construction, so every ~10-minute analysis cycle re-bought verdicts it already owned.
What changed
Schema 116 adds
advisor_judgments— one row per(source_item_id, identity_hash, prompt_version), newest wins,ON DELETE CASCADE(Phase-115 precedent). Coordinated on the board: Lane F confirmed no competing migration; 115 was taken by #564.apply_llm_rerankingnow partitions candidates: fresh-enough memo rows (20h window — judged content is immutable post-ingest; the user's context is what drifts, and it re-judges daily) replay through the reconciler with zero API calls; only the remainder is LLM-judged. Doctrinal guardrails:Replayed items get byte-identical adjustments to a fresh judgment, so there is deliberately no PIPELINE_VERSION bump. The memo mounts as a
#[path]submodule ofanalysis_rerank(theanalysis_dedupprecedent) — nolib.rschurn. The rerank completion log now reportsfresh=/replayed=.Effect
A stable top band costs one judgment per item per day instead of ~9; the freed (paced) token budget rotates to items the judge has never seen — coverage up, spend down. Compounds with #565's oldest-first drain per Lane F.
Verification
cargo test --lib: 4,748 passed / 0 failed (5 new memo tests: freshness aging, identity/prompt isolation, upsert newest-wins, empty no-op)--all-features -D warningsclean; fmt clean;check-file-sizeswarn-only (948/1000)🤖 Generated with Claude Code
https://claude.ai/code/session_019RTCgzAcaopjz3KZnaqNrj