test: read LiteLLM's price table offline, not from the network - #352
Open
LivXue wants to merge 1 commit into
Open
test: read LiteLLM's price table offline, not from the network#352LivXue wants to merge 1 commit into
LivXue wants to merge 1 commit into
Conversation
LiteLLM fetches its price and context table over the network at import unless LITELLM_LOCAL_MODEL_COST_MAP is set beforehand. Nothing set it, so the unit suite read a remote file as a fixed input, and one row appearing upstream was enough to break it. token_rates and resolve_context_window consult LiteLLM's table first and only then the OpenRouter catalogue, which the autouse _no_openrouter_network fixture already keeps off the wire for exactly this reason. Several tests mock an OpenRouter catalogue and arrange for LiteLLM to miss, so the mocked number is the one under test; when a newly published row started answering that lookup first, the mock was never reached. The fixture's promise to keep the suite off the network covered one of the two doors. Three parts. conftest publishes the flag at module scope, before any test module imports LiteLLM, because setting it before first use is too late; setdefault, so a run can still be pointed at the live table deliberately. The two assertions that genuinely need a row the bundled table lacks pin minimax/MiniMax-M3 themselves, keyed on the direct prefix because _candidates maps the plan-billed one onto it before the table is consulted. And a third test asserts the offline guarantee, which nothing else in the suite would notice being lost. No file under raven/ changes, so production still fetches the table exactly as before. The trade-off is that these assertions now describe LiteLLM's bundled data rather than a live vendor claim, so a LiteLLM upgrade can still move them. Co-authored-by: Claude (claude-opus-5[1m]) <[email protected]>
gloryfromca
approved these changes
Aug 22, 2026
gloryfromca
left a comment
Contributor
There was a problem hiding this comment.
Verified the import-time LiteLLM catalog behavior, the pinned MiniMax fixture, and the OpenRouter tier isolation. The focused provider-rate and usage-sink suite passes locally (71 passed), and the full GitHub CI unit job is green. No blocking findings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LiteLLM fetches its price and context table over the network at import unless
LITELLM_LOCAL_MODEL_COST_MAPis set beforehand. Nothing in this repo sets it, so theunit suite reads a remote file as a fixed input. A
deepseek/deepseek-v4-prorowappeared upstream and turned 19 tests red on
main, on bytes no commit here hadtouched, minutes after the same bytes had passed.
token_ratesandresolve_context_windowconsult LiteLLM's table first and only thenthe OpenRouter catalogue, which the autouse
_no_openrouter_networkfixture alreadykeeps off the wire for exactly this reason. Several tests mock an OpenRouter catalogue
and arrange for the LiteLLM tier to miss, so the mocked number is the one under test.
When the new row started answering that lookup first, the mock was never reached: the
counting transport recorded 0 calls, no disk cache was written, and the assertions read
live figures instead (a window of 1048576 where the mock said 163840, a cost of 0.0033
where the mock said 0.00125). The fixture's promise to keep the suite off the network
covered one of the two doors.
The change is three small parts:
conftestpublishesLITELLM_LOCAL_MODEL_COST_MAPat module scope, before any testmodule imports LiteLLM. Setting it before first use is too late, because the remote
table is loaded at import.
setdefault, so a run can still be pointed at the livetable deliberately.
minimax/MiniMax-M3is the one MiniMax row absent from the bundled table, which does carry the other
direct
minimax/models. Both now pin the row, so what they exercise is theresolution ladder rather than what MiniMax published this morning. The key is
minimax/and notminimax-global/because_candidatesmaps the plan-billed prefixonto the direct one before the table is consulted.
notice it being lost, which is how it was lost in the first place.
Checked and deliberately not changed:
_litellm_price_tableand_try_litellm_context_windowdescribe that table as"static" and "offline" (four places in
raven/providers/rates.py). Both are wrongin production, where the flag is never set and the import does reach the network, and
they are the reason the tests were written believing it. Correcting them means
touching the provider hot path, and whether production should read a pinned table
instead of a fetched one is a behaviour question with its own trade-offs: a bundled
table is reproducible but goes stale, and it lacks rows the fetched one has, which is
the same gap the MiniMax pin works around. Left for a change that can weigh it
properly, because
mainis red right now and this is the smallest surface that clearsit.
allow_fetch=Falsetier is still order-dependent._try_litellm_context_windowreturns early only while LiteLLM is absent fromsys.modules, so whether that tier is shut depends on which test imported LiteLLMfirst. Pinning the table makes those cases deterministic, which is what this change is
for, but it does not make the contract they assert true.
Type
Verification
Full suite on the base commit and on this branch, same machine, failing test IDs
diffed rather than counts compared:
The two files that exercise the ladder, with no flag on the command line, which is what
proves the
conftestplacement is early enough:Not vacuous: the same two files with
LITELLM_LOCAL_MODEL_COST_MAP=False, whichpre-empts the
setdefaultand so removes the guarantee:Twenty are the nineteen original failures plus the new test that exists to notice
exactly this.
Repo gate and lint:
Disclosed gaps. The 81 failures that remain are present on the base commit too and are
unrelated to this change (58 in
test_tui_rpc_session.py, the rest across the CLI cron,import, onboard and TUI command suites); they are environment-dependent on this machine
and CI does not reproduce them, which is why the red count on CI was 19 and not 100. The
full suite segfaults at interpreter finalization on base and branch alike, before and
after.
make lintalso runslint-tuiandlint-bridge, which could not run herebecause their Node toolchains are not installed in this checkout; this change adds no
TypeScript. No docs changed: nothing outside
tests/mentions this flag, which is thesame grep that confirms production never sets it.
Risk
Test-only. No file under
raven/changes, so no runtime behaviour moves: in productionLiteLLM still fetches its table exactly as before, and this only stops the suite from
asserting against it.
The trade-off is that the suite now reads a table pinned to whatever LiteLLM bundles, so
a genuine upstream correction to a window or a price no longer reaches these assertions.
That is the intended direction -- a unit test should fail because the code changed -- but
it means the numbers in them describe LiteLLM's bundled data rather than a live vendor
claim, and a LiteLLM upgrade can still move them.
Two existing tests changed meaning and are worth a reviewer's eye:
test_a_plan_billed_provider_still_reports_a_windowandtest_the_window_those_families_report_is_the_vendors_ownpreviously passed by reading alive row, and now pass by reading a pinned one.
Rollback is reverting the commit; the suite returns to reading the live table, with the
failure mode described above.
Related Issues\n\nFixes #358