Skip to content

test: read LiteLLM's price table offline, not from the network - #352

Open
LivXue wants to merge 1 commit into
EverMind-AI:mainfrom
LivXue:test/litellm_price_table_offline
Open

test: read LiteLLM's price table offline, not from the network#352
LivXue wants to merge 1 commit into
EverMind-AI:mainfrom
LivXue:test/litellm_price_table_offline

Conversation

@LivXue

@LivXue LivXue commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

LiteLLM fetches its price and context table over the network at import unless
LITELLM_LOCAL_MODEL_COST_MAP is set beforehand. Nothing in this repo sets it, so the
unit suite reads a remote file as a fixed input. A deepseek/deepseek-v4-pro row
appeared upstream and turned 19 tests red on main, on bytes no commit here had
touched, minutes after the same bytes had passed.

token_rates and resolve_context_window consult LiteLLM's table first and only then
the OpenRouter catalogue, which the autouse _no_openrouter_network fixture already
keeps off the wire for exactly this reason. Several tests mock an OpenRouter catalogue
and arrange for the LiteLLM tier to miss, so the mocked number is the one under test.
When the new row started answering that lookup first, the mock was never reached: the
counting transport recorded 0 calls, no disk cache was written, and the assertions read
live figures instead (a window of 1048576 where the mock said 163840, a cost of 0.0033
where the mock said 0.00125). The fixture's promise to keep the suite off the network
covered one of the two doors.

The change is three small parts:

  • conftest publishes LITELLM_LOCAL_MODEL_COST_MAP at module scope, before any test
    module imports LiteLLM. Setting it before first use is too late, because the remote
    table is loaded at import. setdefault, so a run can still be pointed at the live
    table deliberately.
  • Two assertions genuinely needed a row that exists only remotely. minimax/MiniMax-M3
    is the one MiniMax row absent from the bundled table, which does carry the other
    direct minimax/ models. Both now pin the row, so what they exercise is the
    resolution ladder rather than what MiniMax published this morning. The key is
    minimax/ and not minimax-global/ because _candidates maps the plan-billed prefix
    onto the direct one before the table is consulted.
  • A third test asserts the offline guarantee itself. Nothing else in the suite would
    notice it being lost, which is how it was lost in the first place.

Checked and deliberately not changed:

  • _litellm_price_table and _try_litellm_context_window describe that table as
    "static" and "offline"
    (four places in raven/providers/rates.py). Both are wrong
    in production, where the flag is never set and the import does reach the network, and
    they are the reason the tests were written believing it. Correcting them means
    touching the provider hot path, and whether production should read a pinned table
    instead of a fetched one is a behaviour question with its own trade-offs: a bundled
    table is reproducible but goes stale, and it lacks rows the fetched one has, which is
    the same gap the MiniMax pin works around. Left for a change that can weigh it
    properly, because main is red right now and this is the smallest surface that clears
    it.
  • The allow_fetch=False tier is still order-dependent.
    _try_litellm_context_window returns early only while LiteLLM is absent from
    sys.modules, so whether that tier is shut depends on which test imported LiteLLM
    first. Pinning the table makes those cases deterministic, which is what this change is
    for, but it does not make the contract they assert true.

Type

  • Fix
  • Feature
  • Docs
  • CI / tooling
  • Refactor
  • Other -- tests only

Verification

Full suite on the base commit and on this branch, same machine, failing test IDs
diffed rather than counts compared:

base   (0e544ecb):  100 failed, 6379 passed, 46 skipped, 13 deselected
branch:              81 failed, 6399 passed, 46 skipped, 13 deselected

introduced: 0
fixed:     19   (exactly the set that went red on main)

The two files that exercise the ladder, with no flag on the command line, which is what
proves the conftest placement is early enough:

pytest tests/test_provider_rates.py tests/test_agent_loop_usage_sink.py -q
-> 71 passed

Not vacuous: the same two files with LITELLM_LOCAL_MODEL_COST_MAP=False, which
pre-empts the setdefault and so removes the guarantee:

-> 20 failed, 51 passed

Twenty are the nineteen original failures plus the new test that exists to notice
exactly this.

Repo gate and lint:

python scripts/check_commit_messages.py 0e544ecb..HEAD   -> exit 0
ruff check raven tests scripts                           -> All checks passed!
ruff format --check raven tests scripts                  -> 844 files already formatted
  • Relevant tests pass locally
  • Relevant lint / type checks pass locally
  • User-facing docs or screenshots are updated when needed

Disclosed gaps. The 81 failures that remain are present on the base commit too and are
unrelated to this change (58 in test_tui_rpc_session.py, the rest across the CLI cron,
import, onboard and TUI command suites); they are environment-dependent on this machine
and CI does not reproduce them, which is why the red count on CI was 19 and not 100. The
full suite segfaults at interpreter finalization on base and branch alike, before and
after. make lint also runs lint-tui and lint-bridge, which could not run here
because their Node toolchains are not installed in this checkout; this change adds no
TypeScript. No docs changed: nothing outside tests/ mentions this flag, which is the
same grep that confirms production never sets it.

Risk

Test-only. No file under raven/ changes, so no runtime behaviour moves: in production
LiteLLM still fetches its table exactly as before, and this only stops the suite from
asserting against it.

The trade-off is that the suite now reads a table pinned to whatever LiteLLM bundles, so
a genuine upstream correction to a window or a price no longer reaches these assertions.
That is the intended direction -- a unit test should fail because the code changed -- but
it means the numbers in them describe LiteLLM's bundled data rather than a live vendor
claim, and a LiteLLM upgrade can still move them.

Two existing tests changed meaning and are worth a reviewer's eye:
test_a_plan_billed_provider_still_reports_a_window and
test_the_window_those_families_report_is_the_vendors_own previously passed by reading a
live row, and now pass by reading a pinned one.

Rollback is reverting the commit; the suite returns to reading the live table, with the
failure mode described above.

  • Security impact considered
  • Backward compatibility considered
  • Rollback path is clear for risky changes

Related Issues\n\nFixes #358

LiteLLM fetches its price and context table over the network at import unless
LITELLM_LOCAL_MODEL_COST_MAP is set beforehand. Nothing set it, so the unit
suite read a remote file as a fixed input, and one row appearing upstream was
enough to break it.

token_rates and resolve_context_window consult LiteLLM's table first and only
then the OpenRouter catalogue, which the autouse _no_openrouter_network fixture
already keeps off the wire for exactly this reason. Several tests mock an
OpenRouter catalogue and arrange for LiteLLM to miss, so the mocked number is
the one under test; when a newly published row started answering that lookup
first, the mock was never reached. The fixture's promise to keep the suite off
the network covered one of the two doors.

Three parts. conftest publishes the flag at module scope, before any test module
imports LiteLLM, because setting it before first use is too late; setdefault, so
a run can still be pointed at the live table deliberately. The two assertions
that genuinely need a row the bundled table lacks pin minimax/MiniMax-M3
themselves, keyed on the direct prefix because _candidates maps the plan-billed
one onto it before the table is consulted. And a third test asserts the offline
guarantee, which nothing else in the suite would notice being lost.

No file under raven/ changes, so production still fetches the table exactly as
before. The trade-off is that these assertions now describe LiteLLM's bundled
data rather than a live vendor claim, so a LiteLLM upgrade can still move them.

Co-authored-by: Claude (claude-opus-5[1m]) <[email protected]>
@LivXue
LivXue requested a review from 0xKT August 21, 2026 08:31
@LivXue LivXue self-assigned this Aug 21, 2026

@gloryfromca gloryfromca left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the import-time LiteLLM catalog behavior, the pinned MiniMax fixture, and the OpenRouter tier isolation. The focused provider-rate and usage-sink suite passes locally (71 passed), and the full GitHub CI unit job is green. No blocking findings.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants