fix(cascade): consider escalating everything, and price correct answers over priced rows - #80
Merged
Conversation
…rs over priced rows The cascade's candidate thresholds were 0 plus every observed score, and a row is covered when its score reaches the threshold, so the highest score still covered the rows that reached it: escalating every case was never a candidate. With every case wrong, an error at $100 and an escalation at $1, it chose 0.9 at $201 where escalating all three cost $3. ESCALATE_ALL (infinity) is now the last candidate, and the report says in words that escalating every case is cheapest rather than printing a threshold of inf. CostSummary.per_correct_usd divided the total over priced rows by the correct answers across all rows, so with unpriced rows it understated the cost of a correct answer: summarize([1.0, None], [True, True]) gave 0.5 where 1.0 is right. It now counts correct answers among priced rows. Fixes #34. Fixes #35. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
TMHSDigital
deleted the
fix/cascade-escalate-all-and-cost-per-correct
branch
September 23, 2026 18:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #34. Fixes #35.
#34: the cascade never considered escalating every case.
ESCALATE_ALL(infinity) is now the last candidate. When it wins, the report says in words that escalating every case is cheapest, instead of printing a threshold ofinf.#35:
per_correct_usdunderstated the cost of a correct answer.summarize([1.0, None], [True, True])gave 0.5 where 1.0 is right.Checked
mainand pass now:per_correct_usdis 1.0 in the case above;inf. This test forces the optimum to the new candidate, because the mock refuses an accuracy below chance.ESCALATE_ALL.ruff,mypy --strict, the fullpytestsuite, and the site build all pass.🤖 Generated with Claude Code