Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
name: Docs

on:
push:
branches: [main]
paths:
- 'docs/**'
- 'mkdocs.yml'
- '.github/workflows/docs.yml'
pull_request:
paths:
- 'docs/**'
- 'mkdocs.yml'
- '.github/workflows/docs.yml'

permissions:
contents: read

concurrency:
group: docs-${{ github.ref }}
cancel-in-progress: true

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install MkDocs
# mkdocs is pinned too: a 2.0 is announced that drops the plugin system.
run: pip install mkdocs==1.6.1 mkdocs-material==9.5.49 mkdocs-redirects==1.2.2
- name: Build (strict; fails on broken internal links)
run: mkdocs build --strict
- uses: actions/upload-pages-artifact@v3
if: github.event_name == 'push'
with:
path: site

deploy:
if: github.event_name == 'push'
needs: build
runs-on: ubuntu-latest
permissions:
pages: write
id-token: write
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
/bin/
/dist/
/site/
*.db
*.db-wal
*.db-shm
Expand Down
13 changes: 9 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,9 @@ just test # go test ./...
just race # race detector, -count=1
just lint # golangci-lint run
just ui # rebuild the embedded web UI (requires node)
just stress # load harness (cmd/stress), see docs/performance.md
just stress # load harness (cmd/stress), see docs/reference/performance.md
just docker # docker build -t llmproxy .
just docs # preview the documentation site (requires uv)
```

Single test: `go test ./internal/server -run TestName`. Most server tests go through `testenv_test.go`, which boots a real server against a temp SQLite database.
Expand All @@ -20,7 +21,7 @@ The web UI (`ui/`, React + Vite) builds into `internal/server/uidist/`, which is

## What this is

A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API keys, curated model catalog, usage accounting. Exactly two upstream wire formats (OpenAI-compatible and Anthropic native); the non-goals table in docs/architecture.md caps it there deliberately. Read docs/architecture.md before structural changes; it states every design rule with its rationale.
A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API keys, curated model catalog, usage accounting. Exactly two upstream wire formats (OpenAI-compatible and Anthropic native); the non-goals table in docs/concepts/architecture.md caps it there deliberately. Read docs/concepts/architecture.md before structural changes; it states every design rule with its rationale.

## Hard constraints

Expand All @@ -33,7 +34,7 @@ A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API ke

- `cmd/llmproxy`: CLI entry point; subcommands `serve`, `key`, `relay-token`, `principal`, `version`.
- `cmd/stress`: self-contained load harness.
- `internal/config`: all configuration, environment-driven with `LLMPROXY_` prefix, defaults in `FromEnv()`. Document new variables in docs/configuration.md and expose them in `docker-compose.coolify.yml` as `${VAR:-default}`.
- `internal/config`: all configuration, environment-driven with `LLMPROXY_` prefix, defaults in `FromEnv()`. Document new variables in docs/reference/configuration.md and expose them in `docker-compose.coolify.yml` as `${VAR:-default}`.
- `internal/server`: HTTP layer. `openai.go` is the OpenAI-compatible ingress, `transparent.go` the Anthropic relay, `admin.go` the admin API, `sso.go`/`password.go`/`session.go` auth, `ui.go` serves the embedded UI, `litellm.go` compatibility surface.
- `internal/store`: all persistence; `schema.go` holds the DDL for both dialects.
- `internal/catalog`: model alias -> provider binding resolution and capability checks (checked in exactly one place; a miss is a typed 400 `endpoint_not_supported`).
Expand All @@ -49,6 +50,10 @@ A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API ke
- API keys are stored as HMAC-SHA256 under a secret outside the database (`LLMPROXY_KEY_SECRET` or the generated secret file); plaintext shown once at mint.
- Prometheus labels are aggregate-only (provider, model, endpoint, unit, outcome); never principals or keys.

## Documentation

`docs/` is organised by reader intent: `getting-started.md` (tutorial), `install/` (deploy), `guides/` (operate), `reference/` (look up), `concepts/` (understand). Task pages are imperative and step-numbered with expected output; rationale is one sentence plus a link, and lives in `concepts/architecture.md`. The writing rules are in CONTRIBUTING.md. A new page needs an entry in `mkdocs.yml`'s nav and in the `docs/README.md` router. `mkdocs build --strict` fails on broken internal links and runs in CI on any PR touching `docs/`.

## Deployment files

`docker-compose.yml` is the development setup (SQLite, loopback-only port). `docker-compose.coolify.yml` is the PaaS deployment (Postgres, no host port mapping, every `LLMPROXY_*` variable exposed as `${VAR:-default}` so Coolify lists them). Keep the two and docs/docker.md in sync when configuration changes.
`docker-compose.yml` is the development setup (SQLite, loopback-only port). `docker-compose.coolify.yml` is the PaaS deployment (Postgres, no host port mapping, every `LLMPROXY_*` variable exposed as `${VAR:-default}` so Coolify lists them). Keep the two and docs/install/docker.md in sync when configuration changes.
40 changes: 36 additions & 4 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ just test # go test ./...
just race # race detector, -count=1
just lint # golangci-lint
just ui # rebuild the embedded web UI (requires node)
just stress # load harness, see docs/performance.md
just stress # load harness, see docs/reference/performance.md
```

`just` on its own lists every recipe.
Expand All @@ -32,13 +32,45 @@ temporary SQLite database. `go test ./internal/server -run TestName` runs one.
purpose so a plain `go build` embeds the current UI without a node
toolchain; the `ui-drift` CI job fails if it is stale.
- If you changed configuration, document the variable in
`docs/configuration.md` and expose it in `docker-compose.coolify.yml` as
`docs/reference/configuration.md` and expose it in `docker-compose.coolify.yml` as
`${VAR:-default}`.
- New behaviour comes with a test.

## Writing documentation

`docs/` is organised by what the reader is trying to do: `getting-started.md`
to try it, `install/` to deploy it, `guides/` to operate it, `reference/` to
look something up, `concepts/` to understand why. Put a new page where its
reader would look for it, and add it to the nav in `mkdocs.yml` and the router
in `docs/README.md`.

Six rules keep the task pages consistent:

- **Second person, imperative, present tense.** "Create a key", not "a key can
be created".
- **One action per step.** Number the steps, and give each one its command and
the output that says it worked.
- **Happy path first.** Open with the goal and the prerequisites, then the
minimal path. Variations (TLS, Postgres, overrides) come after, under their
own headings, so they can be skipped.
- **Rationale is one sentence and a link.** The why belongs in
`concepts/architecture.md`, stated once.
- **Consequences stay, justifications go.** "Losing the secret invalidates
every key" survives the cut. "The alternative would buy microseconds" does
not, outside `concepts/`.
- **Reference pages are tables**, not narrative. The tutorial stays one tested
path.

`concepts/architecture.md` and the README are the exception: they explain
rather than instruct, and their voice is deliberately different.

Check a change with `just docs` (local preview) or `mkdocs build --strict`,
which fails on broken internal links. The `Docs` workflow runs the strict
build on every PR that touches `docs/`.

## Design rules that are not negotiable

Read [docs/architecture.md](docs/architecture.md) before any structural change.
Read [docs/concepts/architecture.md](docs/concepts/architecture.md) before any structural change.
It states every rule with its rationale. The ones that will get a patch
rejected outright:

Expand All @@ -58,7 +90,7 @@ rejected outright:
- **Prometheus labels stay aggregate-only** (provider, model, endpoint, unit,
outcome); never principals or keys.

The non-goals table in `docs/architecture.md` is deliberate. If a feature is
The non-goals table in `docs/concepts/architecture.md` is deliberate. If a feature is
listed there, open an issue to discuss before writing code.

## Commit messages
Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,15 +60,23 @@ client.chat.completions.create(model="qwen-72b",
messages=[{"role": "user", "content": "hello"}])
```

See [docs/docker.md](docs/docker.md) for compose, Postgres and the published image.
See [docs/install/docker.md](docs/install/docker.md) for compose, Postgres and the published image, and [docs/install/binary.md](docs/install/binary.md) for systemd.

## Documentation

Everything lives in [docs/](docs/README.md): a getting-started tutorial, how-to guides (installation, Docker, SSO, providers and models, keys and usage, the Anthropic relay), reference (configuration, the HTTP API, an OpenAPI 3.1 spec, performance) and an architecture explanation.
Rendered at **[greyhavenhq.github.io/llmproxy](https://greyhavenhq.github.io/llmproxy/)**, source in [docs/](docs/README.md).

| | |
|---|---|
| Try it | [Getting started](docs/getting-started.md) |
| Deploy it | [binary](docs/install/binary.md), [Docker](docs/install/docker.md), [upgrades](docs/install/upgrades.md) |
| Operate it | [providers](docs/guides/providers.md), [models](docs/guides/models.md), [API keys](docs/guides/api-keys.md), [pricing](docs/guides/pricing.md), [usage](docs/guides/usage.md), [SSO](docs/guides/sso.md), [Claude Code](docs/guides/claude-code.md) |
| Look it up | [configuration](docs/reference/configuration.md), [API](docs/reference/api.md), [OpenAPI spec](docs/reference/openapi.yaml), [performance](docs/reference/performance.md) |
| Understand it | [architecture](docs/concepts/architecture.md) |

## Performance

The built-in stress harness sustains around 8,500 req/s at concurrency 200 on SQLite, every request accounted, heap under 50 MiB. Numbers, method and caveats in [docs/performance.md](docs/performance.md); reproduce with `just stress`.
The built-in stress harness sustains around 8,500 req/s at concurrency 200 on SQLite, every request accounted, heap under 50 MiB. Numbers, method and caveats in [docs/reference/performance.md](docs/reference/performance.md); reproduce with `just stress`.

## Development

Expand Down
4 changes: 2 additions & 2 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,10 @@ Things that are known and deliberate, not vulnerabilities:
box grants access, which is why the server refuses to bind a non-loopback
address without `-allow-nonlocal`. Running it exposed with that flag is a
configuration choice, documented in
[docs/installation.md](docs/installation.md).
[docs/install/binary.md](docs/install/binary.md).
- **The proxy does not inspect request bodies.** It forwards them byte-for-byte.
Content filtering and prompt-level policy are out of scope by design; see the
non-goals table in [docs/architecture.md](docs/architecture.md).
non-goals table in [docs/concepts/architecture.md](docs/concepts/architecture.md).
- **Losing the key secret invalidates keys, not confidentiality of past
traffic.** The secret (`LLMPROXY_KEY_SECRET` or the generated secret file)
is what makes stored key hashes and encrypted provider credentials usable.
Expand Down
110 changes: 65 additions & 45 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -1,47 +1,67 @@
# llmproxy documentation

The documentation follows the [Diátaxis](https://diataxis.fr/) structure:
a tutorial to learn by doing, how-to guides for specific goals, reference
material for looking things up, and explanation for understanding the design.

## Tutorial

- [getting-started.md](getting-started.md): from `git clone` to a first chat
completion, including provider registration, model binding and your first
API key.

## How-to guides

- [installation.md](installation.md): release binaries, build from source,
state layout, first key, systemd, upgrades.
- [docker.md](docker.md): the image, docker compose, switching to Postgres, deploying on Coolify.
- [sso.md](sso.md): OIDC login, sessions, group-based roles, an Authentik
walkthrough.
- [providers-and-models.md](providers-and-models.md): register providers,
discover and bind models, aliases, capabilities, dry-run resolution.
- [keys-and-usage.md](keys-and-usage.md): key lifecycle, usage accounting,
pricing feeds, team statistics, metrics.
- [transparent-relay.md](transparent-relay.md): meter Claude Code and other
Anthropic-native tools without touching their credentials.

## Reference

- [configuration.md](configuration.md): every environment variable, database
URLs, the secret, the admin password, CLI flags.
- [api.md](api.md): the full HTTP surface, endpoint by endpoint, with error
codes and headers.
- [openapi.yaml](openapi.yaml): the same surface as an OpenAPI 3.1 spec.
- [performance.md](performance.md): throughput and latency measurements and
how to reproduce them.

## Explanation

- [architecture.md](architecture.md): scope and non-goals, the three endpoint
families, and the design rules with their rationale, including the
no-content-persistence guarantee.

## Contributing

- [CONTRIBUTING.md](../CONTRIBUTING.md): development setup, the commit
convention, the review expectations and how a release is cut.
- [SECURITY.md](../SECURITY.md): how to report a vulnerability privately.
A self-hosted, OpenAI-compatible LLM proxy: one static Go binary, per-user API
keys, a curated model catalog and usage accounting. No request or response
content is ever persisted.

## Start here

| I want to | Go to |
|---|---|
| See it work, end to end | [Getting started](getting-started.md) |
| Deploy it on a server | [Install the binary](install/binary.md) |
| Deploy it with containers | [Install with Docker](install/docker.md) |
| Give my team access | [Set up SSO](guides/sso.md) |
| Meter Claude Code | [Meter Claude Code](guides/claude-code.md) |
| Look up a variable or endpoint | [Configuration](reference/configuration.md), [API](reference/api.md) |
| Understand why it is built this way | [Architecture](concepts/architecture.md) |

## Deploy it

- [install/binary.md](install/binary.md): release install, state layout, first
login, the loopback guard, a systemd unit.
- [install/docker.md](install/docker.md): the published image, compose,
Postgres, Coolify.
- [install/upgrades.md](install/upgrades.md): backup, upgrade, what migrations
do.

## Operate it

- [guides/providers.md](guides/providers.md): register upstreams, rotate
credentials, TLS, per-endpoint overrides.
- [guides/models.md](guides/models.md): discover, bind, alias, hide, resolve.
- [guides/api-keys.md](guides/api-keys.md): keys for people and services, and
how to revoke them.
- [guides/pricing.md](guides/pricing.md): per-model prices, the feed format,
what unpriced means.
- [guides/usage.md](guides/usage.md): what is recorded, application tags,
the usage and stats endpoints, Prometheus metrics.
- [guides/sso.md](guides/sso.md): OIDC login, group-derived roles, an
Authentik walkthrough, offboarding.
- [guides/claude-code.md](guides/claude-code.md): meter Anthropic-native tools
without touching their credentials.

## Look it up

- [reference/configuration.md](reference/configuration.md): every environment
variable, database URLs, the secret, CLI flags.
- [reference/api.md](reference/api.md): the HTTP surface endpoint by endpoint,
with error codes and headers.
- [reference/openapi.yaml](reference/openapi.yaml): the same surface as an
OpenAPI 3.1 spec.
- [reference/performance.md](reference/performance.md): throughput and latency
measurements, and how to reproduce them.

## Understand it

- [concepts/architecture.md](concepts/architecture.md): scope and non-goals,
the endpoint families, and every design rule with its rationale, including
the no-content-persistence guarantee.

## Contribute

- [CONTRIBUTING.md](https://github.com/greyhavenhq/llmproxy/blob/main/CONTRIBUTING.md):
development setup, the writing rules for these pages, the commit convention,
how a release is cut.
- [SECURITY.md](https://github.com/greyhavenhq/llmproxy/blob/main/SECURITY.md):
how to report a vulnerability privately.
27 changes: 26 additions & 1 deletion docs/architecture.md → docs/concepts/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,31 @@ quantity as unpriced and flags the event; summaries show `cost: null` and
turns missing configuration into "free", which is exactly the failure a
spend-accounting system exists to prevent.

### Usage numbers are normalised at read time

The two supported wire formats disagree about what "input tokens" counts. The
OpenAI shape reports cached tokens as a subset of the prompt count
(`prompt_tokens_details.cached_tokens` ⊆ `prompt_tokens`), while Anthropic
reports cache reads and writes *outside* `input_tokens`. Summing either shape
naively produces a number that means something different per provider.

Stored quantities stay raw as reported, because pricing multiplies each unit
by its own rate and cache reads and writes have their own rates. Every
aggregate (`/my/usage*`, `/admin/v1/usage*`, `/stats/*`) then normalises
`input_tokens` to the non-cached input: the OpenAI cached subset is subtracted
back out, and the relay's `cache_creation_tokens` (fresh input billed at a
premium) folds in while cache reads stay their own unit.

So `input_tokens` always means "input processed at the full input rate" and is
comparable across providers, and `cached_input_tokens` is always the cheap
cache-read count on top of it. Normalising at write time instead would destroy
the raw numbers pricing needs; normalising in each caller would let two
dashboards disagree.

An upstream that never reports `cached_tokens` shows no cached quantity.
Absence means "not reported", never "free", for the same reason unpriced is
not zero.

### The schema is content-free by construction

This is the hard privacy guarantee: **no prompt, completion, embedding input
Expand Down Expand Up @@ -209,4 +234,4 @@ request starts; the relay holds no store handle, and usage is written after
the fact, detached from the request path. Holding a pooled connection for the
duration of a streamed response pins one connection per in-flight stream and
collapses the pool at modest concurrency. This rule is what lets a small pool
serve thousands of concurrent streams; see [performance.md](performance.md).
serve thousands of concurrent streams; see [performance](../reference/performance.md).
Loading
Loading