fleetdiff reads your LiteLLM database or export to show what changed in spend, who drove it, which models served each route, and where spend-log rows are missing.
Linux or macOS; needs curl, gh, shasum and tar. The installer checks the release before unpacking it.
bash -o pipefail -c 'curl --fail --location --proto "=https" --tlsv1.2 https://raw.githubusercontent.com/llm-measurement/fleetdiff/0d7db95c5bb3013f8c564c91fbb07b6235a2664e/scripts/install.sh | sh -s -- v0.12.1 ./fleetdiff-install'
./fleetdiff-install/fleetdiff --versionSave your read-only connection URL in a file only you can read (see the database setup), then run:
chmod 600 ~/.config/litellm-db-url
./fleetdiff-install/fleetdiff investigate --litellm-db --database-url-file ~/.config/litellm-db-url --group-by keyThe report compares the last two complete seven-day UTC periods. Add --daily
for row coverage or --routing-audit for route evidence. The
database guide covers setup, periods and saved exports.
Install fleetdiff using step 1, then investigate your request-level CSV, JSON or JSONL file offline:
export_dir="$HOME/litellm-export"
./fleetdiff-install/fleetdiff investigate \
--litellm-spend "$export_dir/spend.csv" --group-by keyWith enough history, fleetdiff compares the two most recent complete seven-day
UTC periods inside the file and prints their exact boundaries and weekdays.
For shorter exports or chosen periods, set --before-period and --after-period
to dates or explicit UTC start/end intervals.
The report shows request count versus tokens per request by model, the leading
contributors, and usage completeness. Choose --group-by team, user, end-user
or session for another view; add --format json for automation.
Export guide; compare recorded cache reuse.
For a single-period session-budget review, add
--session-budget-usd 1.00 instead of a two-period grouping.
To check for missing spend-log rows, export the daily counts too
and add --litellm-daily-spend "$export_dir/daily.csv"; totals stay as recorded.
For client certificates, PostgreSQL services and other native-unsupported connections, follow the read-only SQL recipe. Keep the export private; it excludes prompts and responses but retains identities.
Install fleetdiff using step 1, then run the synthetic example:
curl --fail --location --proto '=https' --tlsv1.2 \
https://raw.githubusercontent.com/llm-measurement/fleetdiff/v0.12.1/examples/litellm-spend/synthetic.csv \
--output synthetic.csv
./fleetdiff-install/fleetdiff investigate --litellm-spend synthetic.csv \
--before-period 2026-10-07 --after-period 2026-10-08Selected output:
Recorded tokens doubled (8,000 -> 16,000).
model-1 accounts for 100% of the net recorded increase (+8,000 tokens).
model-1: 4,000 -> 12,000 recorded tokens; 4 -> 6 requests.
1,000.00 -> 2,000.00 recorded tokens per request; +3,000 from request count, +5,000 from request size.
2 leading tracked keys account for 100% of the net recorded increase.
Zero-only records, origin unknown: 2 -> 4.
Failed records (overlap usage categories): 1 -> 2.
Two keys explain the increase. The zero-only and failed rows point to records worth reviewing in LiteLLM. The fixture and checks make these answers reproducible.
The summary-file path also shows session concentration:
Run the sessions demo | Read its transcript.
The verified v0.12.1 installer selects the platform binary for your machine.
For public release downloads, gh need not be signed in.
For manual downloads, use the v0.12.1 archives
and verification instructions.
An administrator can verify and transfer artifacts through an approved internal mirror.
With a security-patched Go 1.26 toolchain, install the same version from its module:
go install github.com/llm-measurement/fleetdiff/cmd/[email protected]The executable goes to GOBIN, or $(go env GOPATH)/bin when unset.
See source builds and version reporting.
- Spend reports describe the supplied logged model requests. Missing usage, ambiguous zero-only counts and failures remain visible; unsupported call types lead with an incomplete comparison. Provider records remain the source for billing.
- Use request-level exports. The LiteLLM Usage dashboard's daily aggregates receive instructions for getting those rows. The reader accepts up to 10M rows and 16 GiB per file, with per-record limits.
- By default, private names use aliases or pseudonymous hashes; public catalog model IDs are readable. Keep source files private and share reports under your organization's policy.
- Rankings carry deterministic bounds; distinct counts are statistical estimates. A flagged contributor identifies something to investigate, rather than its cause. Compatible summary files need aligned windows, keys and disjoint producer coverage.
fleetdiff also compares LLM usage summaries, flags unusual windows, and checks trace captures and collector configurations. File-based commands stay offline; the database path reads from your LiteLLM instance and reports locally.
Guides: LiteLLM spend | Routing | Row coverage.
| Commands | investigate, what-if, scan, compare, inspect, diagnose, export litellm |
| Platforms | Linux and macOS, AMD64 and ARM64 |
| Status | Pre-1.0; see the changelog |
| License | Apache-2.0 |
- Native LiteLLM export: setup, credentials and connection contract.
- LiteLLM spend exports: SQL, periods, groupings and source accounting.
- Session-budget review: single-period recorded totals and cap review.
- Model-mix what-if (v0.10.0): re-price covered usage under explicit assumptions.
- Single-application and session examples.
- Inspect your traces and capture recipes.
- Diagnose configurations, scan history and compare cache use.
- Continuous LiteLLM collection.
- Several operators and live collectors.
- Questions and contributor rankings.
- How summary comparison works, comparison contract and investigation contract.
- Operations, resource measurements and checks.
- FAQ, security, release notes and release procedure.
Questions or feedback: open an issue. See Contributing for checks and signed, signed-off commits. Code authors: Vijay and Codex.