From 18937fd898ae4a5cd32700136779e481f295e709 Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:06:58 -0600 Subject: [PATCH 1/6] docs: move pages into the journey tree Pure renames, no content changes. installation.md -> install/binary.md, docker.md -> install/docker.md, sso.md and transparent-relay.md -> guides/, providers-and-models.md and keys-and-usage.md -> guides/ (split in the following commits), configuration/api/openapi/performance -> reference/, architecture.md -> concepts/. --- docs/{ => concepts}/architecture.md | 0 docs/{keys-and-usage.md => guides/api-keys.md} | 0 docs/{transparent-relay.md => guides/claude-code.md} | 0 docs/{providers-and-models.md => guides/providers.md} | 0 docs/{ => guides}/sso.md | 0 docs/{installation.md => install/binary.md} | 0 docs/{ => install}/docker.md | 0 docs/{ => reference}/api.md | 0 docs/{ => reference}/configuration.md | 0 docs/{ => reference}/openapi.yaml | 0 docs/{ => reference}/performance.md | 0 11 files changed, 0 insertions(+), 0 deletions(-) rename docs/{ => concepts}/architecture.md (100%) rename docs/{keys-and-usage.md => guides/api-keys.md} (100%) rename docs/{transparent-relay.md => guides/claude-code.md} (100%) rename docs/{providers-and-models.md => guides/providers.md} (100%) rename docs/{ => guides}/sso.md (100%) rename docs/{installation.md => install/binary.md} (100%) rename docs/{ => install}/docker.md (100%) rename docs/{ => reference}/api.md (100%) rename docs/{ => reference}/configuration.md (100%) rename docs/{ => reference}/openapi.yaml (100%) rename docs/{ => reference}/performance.md (100%) diff --git a/docs/architecture.md b/docs/concepts/architecture.md similarity index 100% rename from docs/architecture.md rename to docs/concepts/architecture.md diff --git a/docs/keys-and-usage.md b/docs/guides/api-keys.md similarity index 100% rename from docs/keys-and-usage.md rename to docs/guides/api-keys.md diff --git a/docs/transparent-relay.md b/docs/guides/claude-code.md similarity index 100% rename from docs/transparent-relay.md rename to docs/guides/claude-code.md diff --git a/docs/providers-and-models.md b/docs/guides/providers.md similarity index 100% rename from docs/providers-and-models.md rename to docs/guides/providers.md diff --git a/docs/sso.md b/docs/guides/sso.md similarity index 100% rename from docs/sso.md rename to docs/guides/sso.md diff --git a/docs/installation.md b/docs/install/binary.md similarity index 100% rename from docs/installation.md rename to docs/install/binary.md diff --git a/docs/docker.md b/docs/install/docker.md similarity index 100% rename from docs/docker.md rename to docs/install/docker.md diff --git a/docs/api.md b/docs/reference/api.md similarity index 100% rename from docs/api.md rename to docs/reference/api.md diff --git a/docs/configuration.md b/docs/reference/configuration.md similarity index 100% rename from docs/configuration.md rename to docs/reference/configuration.md diff --git a/docs/openapi.yaml b/docs/reference/openapi.yaml similarity index 100% rename from docs/openapi.yaml rename to docs/reference/openapi.yaml diff --git a/docs/performance.md b/docs/reference/performance.md similarity index 100% rename from docs/performance.md rename to docs/reference/performance.md From f0a2f3bf08f66b5a96bc23578a5edace9951b996 Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:09:21 -0600 Subject: [PATCH 2/6] docs: rewrite the tutorial and the install guides getting-started.md now starts from the release binary instead of git clone, with numbered steps and expected output at each one. install/binary.md keeps the release install, state layout, first login, the loopback guard (now stated once, here) and a step-by-step systemd setup. install/docker.md leads with the commands and moves the Dockerfile narration into a facts table; the two-stage build description is cut. install/upgrades.md is new: backup first, then stop-replace-start, with the migration transaction mechanics reduced to five bullets. Deleted: build-recipe listing in the tutorial, Dockerfile stage design, the loopback guard rationale in docker.md (now a link). --- docs/getting-started.md | 160 ++++++++++--------- docs/install/binary.md | 326 +++++++++++++++++++-------------------- docs/install/docker.md | 218 +++++++++++++------------- docs/install/upgrades.md | 69 +++++++++ 4 files changed, 433 insertions(+), 340 deletions(-) create mode 100644 docs/install/upgrades.md diff --git a/docs/getting-started.md b/docs/getting-started.md index a93ea0a..321128d 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -1,83 +1,87 @@ # Getting started -This tutorial takes you from a fresh clone to a first successful chat -completion through the proxy. At the end you will have a running llmproxy, a -registered upstream provider, a bound model, your own API key, and a working -call from both curl and the Python OpenAI SDK. +Take a running proxy from nothing to a first chat completion in about ten +minutes. At the end you have llmproxy running, one upstream provider +registered, one model bound, your own API key, and a working call from curl +and the Python OpenAI SDK. You need: -- A recent Go toolchain. `go.mod` pins the exact version and Go's default - `GOTOOLCHAIN=auto` fetches it, so any reasonably current install works. -- [`just`](https://github.com/casey/just), the command runner the project - uses for its recipes. +- A Linux or macOS machine. - An OpenAI-compatible upstream to route to: a local vLLM or SGLang node, or any endpoint serving the OpenAI API shape (including `https://api.openai.com/v1` with your own OpenAI key). -No node toolchain is needed: the web UI's compiled assets are committed and -embedded at build time. - -## 1. Build +## 1. Install the binary ```bash -git clone https://github.com/greyhavenhq/llmproxy.git -cd llmproxy -just build # -> bin/llmproxy (static, roughly 14 MiB) +curl -fsSL https://github.com/greyhavenhq/llmproxy/releases/latest/download/llmproxy_Linux_x86_64.tar.gz | tar xz +sudo install -m 0755 llmproxy /usr/local/bin/llmproxy +llmproxy version +``` + +``` +llmproxy v1.1.0 ``` -The result is one static binary (no CGO, pure-Go SQLite). The other recipes, -for later: `just ui` (rebuild the embedded web UI), `just test`, `just race`, -`just stress` and `just lint`. +Other platforms and checksum verification are in +[install/binary.md](install/binary.md). To build from source instead: +`git clone https://github.com/greyhavenhq/llmproxy.git && cd llmproxy && just build`. -## 2. Run +## 2. Start the server ```bash -bin/llmproxy serve -# llmproxy v1.1.0 listening on http://127.0.0.1:4000 +llmproxy serve ``` -Without any OIDC configuration the proxy runs in local single-admin mode and -binds loopback only. State appears in the working directory on first boot: -`llmproxy.db` (SQLite, WAL mode), the key-hashing secret in -`.llmproxy/secret`, and a generated admin password in +``` +llmproxy v1.1.0 listening on http://127.0.0.1:4000 +``` + +State appears in the working directory on first boot: `llmproxy.db`, the +key-hashing secret in `.llmproxy/secret`, and a generated admin password in `.llmproxy/admin-password`. ## 3. Sign in -Open and sign in with the admin password: +Read the generated password: ```bash cat .llmproxy/admin-password ``` -(Set `LLMPROXY_ADMIN_PASSWORD` before starting the server if you want to -choose it yourself.) The UI covers everything this tutorial does: provider -registration, model binding, keys and the usage dashboard. The rest of the -tutorial uses the HTTP API so every step is copy-pasteable; feel free to do -the same steps in the UI instead. +Open and sign in with it. + +The UI does everything the rest of this tutorial does: providers, models, +keys and the usage dashboard. The steps below use the HTTP API so they are +copy-pasteable. Follow either. + +## 4. Mint an admin key -For the terminal steps, mint an admin API key. Run this from the same -directory as the server, in a second terminal: +Run this in a second terminal, from the same directory as the server: ```bash -bin/llmproxy key create -label tutorial -# key id: 3f2a... -# principal: local-admin -# api key: lp_... -# Store it now; it will not be shown again. +llmproxy key create -label tutorial ``` +``` +key id: 3f2a... +principal: local-admin +api key: lp_... +Store it now; it will not be shown again. +``` + +Set the two variables the remaining steps use: + ```bash ADMIN=lp_... # the key you just minted P=http://127.0.0.1:4000 ``` -## 4. Register a provider +## 5. Register a provider -A provider is an upstream endpoint speaking a wire format. `base_url` -includes `/v1`. The upstream key is optional (many self-hosted nodes are -unauthenticated) and is stored encrypted, never returned by any API. +A provider is an upstream endpoint. `base_url` includes `/v1`. Adjust both +values for your upstream; drop `api_key` if it needs no authentication. ```bash curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ @@ -89,18 +93,30 @@ curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ }' ``` -Adjust `base_url` (and `api_key`) for your upstream. You can ask the upstream -what it serves; discovery is read-only and never binds anything: +The response echoes the provider with `has_credential: true`. The credential +is stored encrypted and no endpoint ever returns it. + +Ask the upstream what it serves: ```bash curl -s $P/admin/v1/providers/vllm-1/discover -H "authorization: Bearer $ADMIN" ``` -## 5. Bind a model +```json +{ + "provider": "vllm-1", + "models": [ + {"upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", "bound_alias": null} + ] +} +``` + +Discovery is read-only and binds nothing. -A binding maps a caller-facing alias to one (provider, upstream model) pair -and serves as soon as it exists. Use an upstream name from the discovery -output: +## 6. Bind a model + +Use an `upstream_name` from the discovery output. The alias is the name your +callers will use. ```bash curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ @@ -112,16 +128,26 @@ curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ }' ``` -Check what the alias resolves to without calling anything: +The binding serves callers as soon as it exists. Check what it resolves to +without calling the upstream: ```bash curl -s "$P/admin/v1/resolve?model=qwen-72b" -H "authorization: Bearer $ADMIN" ``` -## 6. Create your key +```json +{ + "alias": "qwen-72b", + "provider": "vllm-1", + "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", + "url": "http://10.0.0.5:8000/v1/chat/completions" +} +``` + +## 7. Create a caller key -The admin key works for inference too, but day-to-day callers use their own -keys. Any key or session holder can mint keys for themselves: +The admin key works for inference too, but day-to-day callers use their own. +Any key or session holder mints keys for themselves: ```bash curl -s $P/my/keys -H "authorization: Bearer $ADMIN" \ @@ -135,9 +161,7 @@ show only `***xxxx`. KEY=lp_... # the new key ``` -## 7. First chat completion - -With curl: +## 8. Make a call ```bash curl -s $P/v1/chat/completions -H "authorization: Bearer $KEY" \ @@ -147,10 +171,11 @@ curl -s $P/v1/chat/completions -H "authorization: Bearer $KEY" \ }' ``` -The response is the upstream's own body, untouched; the -`x-llmproxy-provider` and `x-llmproxy-model` headers tell you who served it. +The response is the upstream's own body, untouched. The +`x-llmproxy-provider` and `x-llmproxy-model` response headers name who served +it. -With the Python OpenAI SDK, one line changed from stock usage: +From the Python OpenAI SDK, one line differs from stock usage: ```python from openai import OpenAI @@ -174,17 +199,14 @@ for chunk in client.chat.completions.create( print(chunk.choices[0].delta.content or "", end="") ``` -Back in the UI, the usage dashboard now shows the requests, tokens and (once -you set prices) cost. +## 9. See the usage + +Back in the UI, the Usage tab now shows the requests and tokens. Cost appears +once you set prices; see [guides/pricing.md](guides/pricing.md). ## Where next -- [installation.md](installation.md) for systemd, state layout and upgrades; - [docker.md](docker.md) for the container route. -- [providers-and-models.md](providers-and-models.md) for capabilities, - aliases, per-endpoint overrides and TLS options. -- [keys-and-usage.md](keys-and-usage.md) for pricing, usage accounting and - metrics. -- [sso.md](sso.md) to move from the local admin password to OIDC SSO. -- [transparent-relay.md](transparent-relay.md) to meter Claude Code and other - Anthropic-native tools. +- Deploy it properly: [install/binary.md](install/binary.md) for systemd, or + [install/docker.md](install/docker.md) for containers. +- Give it to your team: [guides/sso.md](guides/sso.md). +- Meter Claude Code: [guides/claude-code.md](guides/claude-code.md). diff --git a/docs/install/binary.md b/docs/install/binary.md index 09c9de1..573589b 100644 --- a/docs/install/binary.md +++ b/docs/install/binary.md @@ -1,42 +1,47 @@ -# Installation +# Install the binary -llmproxy is a single static Go binary with no CGO and a pure-Go SQLite driver. -There is nothing else to install: no separate runtime, no external migration -tool, no sidecar. +llmproxy is one static Go binary with no CGO and a pure-Go SQLite driver. +There is no runtime to install, no migration tool and no sidecar. -## Install a release binary +## Install a release Every tag publishes archives for linux/amd64, linux/arm64 and darwin/arm64 on -the [releases page](https://github.com/greyhavenhq/llmproxy/releases), each -containing the binary, the README and the licence, plus a `checksums.txt` -signed off by the release workflow. +the [releases page](https://github.com/greyhavenhq/llmproxy/releases). -```bash -curl -fsSL https://github.com/greyhavenhq/llmproxy/releases/latest/download/llmproxy_Linux_x86_64.tar.gz | tar xz -sudo install -m 0755 llmproxy /usr/local/bin/llmproxy -llmproxy version -``` +1. Download and install: -The archive names follow `llmproxy__.tar.gz`, with `Os` in -`Linux`/`Darwin` and `Arch` in `x86_64`/`arm64`. Verify a download against -`checksums.txt`: + ```bash + curl -fsSL https://github.com/greyhavenhq/llmproxy/releases/latest/download/llmproxy_Linux_x86_64.tar.gz | tar xz + sudo install -m 0755 llmproxy /usr/local/bin/llmproxy + llmproxy version + ``` -```bash -curl -fsSLO https://github.com/greyhavenhq/llmproxy/releases/latest/download/checksums.txt -sha256sum -c --ignore-missing checksums.txt -``` + ``` + llmproxy v1.1.0 + ``` + + Archive names follow `llmproxy__.tar.gz`, with `Os` in + `Linux`/`Darwin` and `Arch` in `x86_64`/`arm64`. + +2. Verify the download: + + ```bash + curl -fsSLO https://github.com/greyhavenhq/llmproxy/releases/latest/download/checksums.txt + sha256sum -c --ignore-missing checksums.txt + ``` + + ``` + llmproxy_Linux_x86_64.tar.gz: OK + ``` -For containers, use the published image instead; see -[docker.md](docker.md). +For containers, use the published image instead; see [docker.md](docker.md). -## Build from source +## Build from source instead -The module's `go.mod` pins the exact Go version, so any reasonably recent -toolchain will fetch the right compiler automatically (Go's default -`GOTOOLCHAIN=auto` behavior). The web UI's compiled assets are committed in -`internal/server/uidist/` and embedded at build time, so building the binary -needs no node toolchain; `just ui` rebuilds them from `ui/` when you change -the frontend. +You need a recent Go toolchain (`go.mod` pins the version and Go's default +`GOTOOLCHAIN=auto` fetches it) and [just](https://just.systems). No node +toolchain is needed: the web UI's compiled assets are committed and embedded +at build time. ```bash git clone https://github.com/greyhavenhq/llmproxy @@ -44,173 +49,164 @@ cd llmproxy just build # -> bin/llmproxy (static, roughly 14 MiB) ``` -`just build` runs: +Cross-compile for another target: ```bash -CGO_ENABLED=0 go build -ldflags="-s -w" -o bin/llmproxy ./cmd/llmproxy +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o llmproxy ./cmd/llmproxy ``` -Cross-compile the same way, for example for a Linux server: +## Where state lives -```bash -CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -o llmproxy ./cmd/llmproxy -``` +Three pieces of state live outside the process. All are created on first use, +relative to the working directory unless configured otherwise. -The binary has five subcommands: `serve`, `key create|list|delete`, -`relay-token create|list|delete`, `principal create` and `version`. The `key`, -`relay-token` and `principal` commands operate directly on the database, which -is how you bootstrap before the HTTP API has any keys. +| What | Default location | Set with | +|---|---|---| +| Database | `llmproxy.db` (SQLite, WAL mode, plus `-wal` and `-shm` files) | `LLMPROXY_DATABASE_URL`, or a `postgres://` URL | +| Key secret | `.llmproxy/secret` (32 random bytes, mode 0600) | `LLMPROXY_KEY_SECRET` or `LLMPROXY_SECRET_FILE` | +| Admin password | `.llmproxy/admin-password` (mode 0600) | `LLMPROXY_ADMIN_PASSWORD` or `LLMPROXY_ADMIN_PASSWORD_FILE` | -## Where state lives +Two consequences to plan for: -Three pieces of state exist outside the process, all created on first use -relative to the working directory unless configured otherwise: - -- The database. `LLMPROXY_DATABASE_URL` defaults to `llmproxy.db`, a SQLite - file opened in WAL mode (so you will also see `llmproxy.db-wal` and - `llmproxy.db-shm` alongside it). Point it at a `postgres://` URL for - Postgres instead; see [configuration.md](configuration.md). -- The secret. If `LLMPROXY_KEY_SECRET` is not set, a random 32-byte secret is - generated once and written to `LLMPROXY_SECRET_FILE` (default - `.llmproxy/secret`, mode 0600). This secret is what makes API key hashes - and encrypted provider credentials usable; losing it invalidates every key - and stored upstream credential. Back it up together with the database, and - keep it out of the database's backup destination if you want the "stolen DB - is useless" property to hold. Details in - [configuration.md](configuration.md#the-secret). -- The admin password. Unless `LLMPROXY_ADMIN_PASSWORD` is set (or password - login is disabled), a random password is generated on first boot and written - to `LLMPROXY_ADMIN_PASSWORD_FILE` (default `.llmproxy/admin-password`, mode - 0600). It signs the local admin into the browser UI and is never stored in - the database. Losing it is harmless: delete the file and restart to get a - new one. +- **Losing the key secret invalidates every API key and every stored upstream + credential.** Back it up with the database, and keep the backup out of the + database's backup destination if you want a stolen database to stay useless. + See [the secret](../reference/configuration.md#the-secret). +- **Losing the admin password is harmless.** Delete the file and restart to + get a new one. Run the CLI from the same working directory (or with the same environment -variables) as the server, otherwise `llmproxy key create` will mint keys -against a different database and secret than the server is using. +variables) as the server. Otherwise `llmproxy key create` mints keys against a +different database and secret than the server reads. -## First login (local mode) +## First login Without any `LLMPROXY_OIDC_*` configuration the proxy runs in local single-admin mode: on boot it creates a `local-admin` principal with the admin role (name configurable via `LLMPROXY_LOCAL_ADMIN_NAME`). -The fastest path is the browser: start the server, read the generated password -from `.llmproxy/admin-password`, open `http://127.0.0.1:4000` and sign in. The -UI covers provider registration, model bindings, API keys and usage. +1. Start the server: -To bootstrap over the API instead, mint the first key with the CLI: + ```bash + llmproxy serve + ``` -```bash -bin/llmproxy key create -label bootstrap -# key id: 3f2a... -# principal: local-admin -# api key: lp_... -# Store it now; it will not be shown again. -``` - -The plaintext is printed once; only a keyed HMAC-SHA256 hash is stored. After -this, everything else can be done over the HTTP API with that key. + ``` + llmproxy v1.1.0 listening on http://127.0.0.1:4000 + ``` -Then run the server: +2. Read the generated password: -```bash -bin/llmproxy serve # llmproxy v1.1.0 listening on http://127.0.0.1:4000 -``` + ```bash + cat .llmproxy/admin-password + ``` -## The loopback guard +3. Open and sign in with it. The UI covers providers, + model bindings, API keys and usage. -In local mode any key minted on the box grants access, so the server refuses -to bind to a non-loopback address (`127.0.0.1`, `::1` and `localhost` are the -allowed hosts) unless you explicitly override: +To bootstrap over the API instead, mint the first key with the CLI. This works +before any HTTP credential exists, because the CLI writes to the database +directly: ```bash -bin/llmproxy serve -host 0.0.0.0 # refuses with an error -bin/llmproxy serve -host 0.0.0.0 -allow-nonlocal # binds, prints a loud warning +llmproxy key create -label bootstrap ``` -`LLMPROXY_ALLOW_NONLOCAL=1` is the environment equivalent of -`-allow-nonlocal`. The guard does not apply once SSO is configured -(`LLMPROXY_OIDC_ISSUER` set), because access is then gated by the identity -provider. If you need local mode reachable from other machines, prefer keeping -the bind on loopback and fronting it with a reverse proxy you control. - -## systemd unit example - -```ini -[Unit] -Description=llmproxy -After=network-online.target -Wants=network-online.target - -[Service] -User=llmproxy -Group=llmproxy -WorkingDirectory=/var/lib/llmproxy -ExecStart=/usr/local/bin/llmproxy serve -Restart=on-failure -Environment=LLMPROXY_DATABASE_URL=/var/lib/llmproxy/llmproxy.db -Environment=LLMPROXY_SECRET_FILE=/var/lib/llmproxy/secret -# Environment=LLMPROXY_OIDC_ISSUER=https://auth.example.com/application/o/llmproxy/ -# ... see docs/configuration.md and docs/sso.md - -# Hardening (optional but cheap; the process only needs its state directory) -NoNewPrivileges=true -ProtectSystem=strict -ProtectHome=true -ReadWritePaths=/var/lib/llmproxy -PrivateTmp=true - -[Install] -WantedBy=multi-user.target +``` +key id: 3f2a... +principal: local-admin +api key: lp_... +Store it now; it will not be shown again. ``` -Create the state directory and user first: +The plaintext is printed once; only a keyed hash is stored. -```bash -useradd --system --home /var/lib/llmproxy --shell /usr/sbin/nologin llmproxy -mkdir -p /var/lib/llmproxy && chown llmproxy:llmproxy /var/lib/llmproxy -``` +The binary has five subcommands: `serve`, `key create|list|delete`, +`relay-token create|list|delete`, `principal create` and `version`. -Bootstrap the first key as that user so the CLI and the service share state: +## The loopback guard + +In local mode any key minted on the box grants access, so the server refuses +to bind a non-loopback address. The allowed hosts are `127.0.0.1`, `::1` and +`localhost`. ```bash -sudo -u llmproxy env LLMPROXY_DATABASE_URL=/var/lib/llmproxy/llmproxy.db \ - LLMPROXY_SECRET_FILE=/var/lib/llmproxy/secret \ - /usr/local/bin/llmproxy key create -label bootstrap +llmproxy serve -host 0.0.0.0 # refuses with an error +llmproxy serve -host 0.0.0.0 -allow-nonlocal # binds, prints a loud warning ``` -The server shuts down cleanly on SIGINT/SIGTERM: it stops accepting -connections, gives in-flight requests up to 10 seconds, and drains the -detached usage-accounting goroutines so no usage records are lost. - -## Upgrades - -Upgrading is: stop the service, replace the binary, start the service. There -is no separate migration step and no external migration tool. - -On every boot the server applies the base schema with idempotent -`CREATE TABLE IF NOT EXISTS` / `CREATE INDEX IF NOT EXISTS` statements, which -brings a fresh database fully up to date and creates any wholly-missing table -on an existing one. It then runs the versioned migrations -(`internal/store/migrate.go`) that carry existing databases forward. Each -migration runs at most once per database: the change and its bookkeeping row -in the `schema_migration` table commit in the same transaction, so a database -is never left half-migrated. A migration that fails is a real startup error, -not a warning: the server refuses to serve rather than run against a schema it -cannot account for. - -Practical consequences: - -- The applied history is queryable: `SELECT version, applied_at FROM - schema_migration ORDER BY version`. -- Downgrades are not supported. There are no down migrations; roll back by - restoring the backup you took before upgrading. -- Take a database backup before upgrading regardless. For SQLite, copy the - `.db` file while the service is stopped, or run - `sqlite3 llmproxy.db ".backup ..."` while it is running. -- Data that a migration cannot invent stays empty rather than wrong. Keys - minted before `api_key.key_suffix` existed list an empty display suffix - until they are recreated. -- The secret file format is stable and is never rewritten by the server; do - not regenerate it during upgrades. +`LLMPROXY_ALLOW_NONLOCAL=1` is the environment equivalent of +`-allow-nonlocal`. The guard does not apply once `LLMPROXY_OIDC_ISSUER` is +set, because the identity provider then gates access. + +To reach local mode from other machines, keep the bind on loopback and front +it with a reverse proxy you control. + +## Run as a systemd service + +1. Create the user and the state directory: + + ```bash + useradd --system --home /var/lib/llmproxy --shell /usr/sbin/nologin llmproxy + mkdir -p /var/lib/llmproxy && chown llmproxy:llmproxy /var/lib/llmproxy + ``` + +2. Write `/etc/systemd/system/llmproxy.service`: + + ```ini + [Unit] + Description=llmproxy + After=network-online.target + Wants=network-online.target + + [Service] + User=llmproxy + Group=llmproxy + WorkingDirectory=/var/lib/llmproxy + ExecStart=/usr/local/bin/llmproxy serve + Restart=on-failure + Environment=LLMPROXY_DATABASE_URL=/var/lib/llmproxy/llmproxy.db + Environment=LLMPROXY_SECRET_FILE=/var/lib/llmproxy/secret + # Environment=LLMPROXY_OIDC_ISSUER=https://auth.example.com/application/o/llmproxy/ + + # Hardening: the process only needs its state directory + NoNewPrivileges=true + ProtectSystem=strict + ProtectHome=true + ReadWritePaths=/var/lib/llmproxy + PrivateTmp=true + + [Install] + WantedBy=multi-user.target + ``` + +3. Start it: + + ```bash + systemctl daemon-reload + systemctl enable --now llmproxy + systemctl status llmproxy + ``` + +4. Mint the first key as the service user, so the CLI and the service share + state: + + ```bash + sudo -u llmproxy env LLMPROXY_DATABASE_URL=/var/lib/llmproxy/llmproxy.db \ + LLMPROXY_SECRET_FILE=/var/lib/llmproxy/secret \ + /usr/local/bin/llmproxy key create -label bootstrap + ``` + +On SIGINT and SIGTERM the server stops accepting connections, gives in-flight +requests up to 10 seconds and drains the usage-accounting goroutines, so a +restart loses no usage records. + +## Where next + +- [upgrades.md](upgrades.md) to keep it current. +- [../guides/providers.md](../guides/providers.md) to register your first + upstream. +- [../guides/sso.md](../guides/sso.md) to replace the admin password with + OIDC. +- [../reference/configuration.md](../reference/configuration.md) for every + environment variable. diff --git a/docs/install/docker.md b/docs/install/docker.md index f78fd8a..09b3fd7 100644 --- a/docs/install/docker.md +++ b/docs/install/docker.md @@ -1,109 +1,89 @@ -# Docker - -## The published image +# Install with Docker Tagged releases push a multi-arch (amd64 and arm64) image to the GitHub -Container Registry: +Container Registry. + +## Run the image ```bash -docker pull ghcr.io/greyhavenhq/llmproxy:latest # or :1.1.0, or :1 docker run -p 127.0.0.1:4000:4000 -v llmproxy-data:/data \ - -e LLMPROXY_ALLOW_NONLOCAL=1 ghcr.io/greyhavenhq/llmproxy:latest + -e LLMPROXY_ALLOW_NONLOCAL=1 ghcr.io/greyhavenhq/llmproxy:1.1.0 ``` -Pin the exact version in production (`:1.1.0`); `:latest` and the major tag -`:1` move under you, and a prerelease tag never moves either of them. The image -is built from the same `Dockerfile` described below, with the version stamped -into the binary. +``` +llmproxy v1.1.0 listening on http://0.0.0.0:4000 +``` -## Building the image +Two things this command does deliberately: -```bash -just docker # docker build -t llmproxy . -``` +- **Mounts `/data`.** All state lives there. Without a volume, the database, + the key secret and the admin password vanish with the container. +- **Sets `LLMPROXY_ALLOW_NONLOCAL=1`.** The container binds `0.0.0.0`, which + trips the loopback guard in local (no-SSO) mode; see [the loopback + guard](binary.md#the-loopback-guard). Inside a container the real exposure + boundary is the port mapping, so publish the port narrowly, as above. In SSO + mode the guard does not apply and the variable is unnecessary. -The Dockerfile is a two-stage build: a `golang` Alpine stage compiles the -static binary with the committed web UI embedded (`CGO_ENABLED=0`), and the -runtime stage is plain `alpine` with just the binary. The UI is not rebuilt in -the image; the committed `internal/server/uidist/` is embedded as-is, and the -`ui-drift` CI job guarantees it matches the `ui/` source. What the runtime image -sets up: - -- A non-root user `llmproxy` (uid 1000); the process never runs as root. -- A `/data` volume owned by that user, and environment defaults that put all - state there: `LLMPROXY_DATABASE_URL=/data/llmproxy.db`, - `LLMPROXY_SECRET_FILE=/data/secret` and - `LLMPROXY_ADMIN_PASSWORD_FILE=/data/admin-password`. Mount something at - `/data` or the database, the secret and the admin password vanish with the - container. -- `LLMPROXY_HOST=0.0.0.0` and `LLMPROXY_PORT=4000`, with port 4000 exposed. - Binding loopback inside a container would make the service unreachable from - outside it, so the container binds all interfaces and leaves exposure - control to Docker's port publishing. -- `ENTRYPOINT ["llmproxy"]` with `CMD ["serve"]`, so the container runs the - server by default and `docker run llmproxy key create ...` style overrides - run the CLI against the same `/data` state. - -Because the container binds `0.0.0.0`, running in local (no-SSO) mode trips -the loopback guard: the server refuses to start unless -`LLMPROXY_ALLOW_NONLOCAL=1` is set. That is intentional friction. Inside a -container the guard's premise ("loopback means only this machine") no longer -maps to who can reach the socket; the real exposure boundary is the port -mapping, so you have to opt in explicitly and then publish the port narrowly. -In SSO mode the guard does not apply and the variable is unnecessary. - -## docker compose - -The repository ships two compose files: `docker-compose.yml` for local -development and `docker-compose.coolify.yml` for deployment behind a PaaS -reverse proxy (see [Deploying on Coolify](#deploying-on-coolify)). - -`docker-compose.yml` encodes the safe local-mode setup: SQLite state in a -named volume, `LLMPROXY_ALLOW_NONLOCAL=1`, and the port published only on -the host's loopback (`127.0.0.1:4000:4000`) so the proxy stays local to the -host despite the container-internal `0.0.0.0` bind. Widen the port mapping -deliberately, ideally only after switching to SSO. +Pin the exact version in production. `:latest` and the major tag `:1` move +under you; a prerelease tag moves neither. -```bash -docker compose up -d -curl -s http://127.0.0.1:4000/healthz -# {"status":"ok"} -``` +## Use docker compose -### First login +`docker-compose.yml` in the repository is the local-mode setup: SQLite state +in a named volume, `LLMPROXY_ALLOW_NONLOCAL=1`, and the port published only on +the host's loopback. -Read the generated admin password and sign in at `http://127.0.0.1:4000`: +1. Start it: -```bash -docker compose exec llmproxy cat /data/admin-password -``` + ```bash + docker compose up -d + curl -s http://127.0.0.1:4000/healthz + ``` + + ```json + {"status":"ok"} + ``` + +2. Read the generated admin password: -Or set `LLMPROXY_ADMIN_PASSWORD` in the compose file to choose it yourself. -From the UI you can register providers, bind models and mint keys. + ```bash + docker compose exec llmproxy cat /data/admin-password + ``` -### Minting the first key over the CLI +3. Open and sign in. From the UI, register providers, + bind models and mint keys. -The CLI shares the container's `/data`, so bootstrap through `exec`: +To choose the password yourself, set `LLMPROXY_ADMIN_PASSWORD` in the compose +file instead of step 2. + +To bootstrap over the API, mint a key through the container, which shares the +same `/data`: ```bash docker compose exec llmproxy llmproxy key create -label bootstrap -# key id: 9d0b... -# principal: local-admin -# api key: lp_... -# Store it now; it will not be shown again. ``` -(First `llmproxy` is the compose service name, second is the binary.) +``` +key id: 9d0b... +principal: local-admin +api key: lp_... +Store it now; it will not be shown again. +``` + +(The first `llmproxy` is the compose service name, the second is the binary.) + +Widen the port mapping only after switching to SSO. -### SSO mode +## Switch to SSO Uncomment and fill in the `LLMPROXY_OIDC_*` variables in the compose file -(issuer, client id, client secret, redirect URL, optionally admin and required -groups) and remove `LLMPROXY_ALLOW_NONLOCAL`. The redirect URL must be the -externally reachable `https://your-proxy/auth/callback`, which implies a TLS -reverse proxy in front; see [sso.md](sso.md). +(issuer, client id, client secret, redirect URL, optionally the admin and +required groups), then remove `LLMPROXY_ALLOW_NONLOCAL`. The redirect URL must +be the externally reachable `https://your-proxy/auth/callback`, which implies +a TLS reverse proxy in front. Full walkthrough in +[../guides/sso.md](../guides/sso.md). -## Switching to Postgres +## Switch to Postgres The compose file contains a commented-out `db` service. Uncomment it, the `llmproxy-pg` volume, and the `LLMPROXY_DATABASE_URL` line: @@ -128,33 +108,59 @@ volumes: llmproxy-pg: ``` -The schema is created automatically on boot, same as with SQLite. Note there -is no data migration between backends; switching starts from an empty -database, so plan the switch before you accumulate keys and usage you care -about. The secret file in `/data` stays authoritative either way: keep the -`llmproxy-data` volume (or set `LLMPROXY_KEY_SECRET`), because existing API -keys and encrypted provider credentials are only valid under that secret. +The schema is created on boot, same as with SQLite. + +Two consequences: -## Deploying on Coolify +- **There is no data migration between backends.** Switching starts from an + empty database, so switch before you accumulate keys and usage you care + about. +- **The `/data` volume still matters.** It holds the key secret, and existing + API keys and encrypted provider credentials are only valid under it. Keep + the volume, or set `LLMPROXY_KEY_SECRET` explicitly. + +## Deploy on Coolify `docker-compose.coolify.yml` targets Coolify or any compose-based PaaS. It -differs from the development file in two ways: - -- Postgres is the default backend, as a `db` service with a healthcheck the - app waits on. -- There is no `ports` mapping. The platform's reverse proxy routes to the - container on `LLMPROXY_PORT` (default 4000); nothing is published on the - host directly. - -Every supported `LLMPROXY_*` variable appears in the file as -`${VAR:-default}`. Coolify parses these placeholders and lists each one as -an editable environment variable in its UI, so configuration happens there -rather than by editing the file. At minimum set `POSTGRES_PASSWORD` (it -feeds both the `db` service and the connection URL). Once the deployment is -reachable over HTTPS, configure the `LLMPROXY_OIDC_*` variables for SSO; -until then the file defaults to local mode with `LLMPROXY_ALLOW_NONLOCAL=1`, -which means anyone who can reach the deployment can use its keys. - -The `/data` volume still matters with Postgres: it holds the generated key -secret and admin password files. Alternatively pin `LLMPROXY_KEY_SECRET` and +differs from the development file in two ways: Postgres is the default +backend, as a `db` service with a healthcheck the app waits on, and there is +no `ports` mapping, because the platform's reverse proxy routes to the +container on `LLMPROXY_PORT` (default 4000). + +1. Point Coolify at the repository and select + `docker-compose.coolify.yml`. Every supported `LLMPROXY_*` variable appears + in the file as `${VAR:-default}`, so Coolify lists each one as an editable + environment variable. +2. Set `POSTGRES_PASSWORD`. It feeds both the `db` service and the connection + URL. +3. Deploy, and confirm the deployment answers on `/healthz`. +4. Configure the `LLMPROXY_OIDC_*` variables once the deployment is reachable + over HTTPS. + +Until step 4, the file defaults to local mode with +`LLMPROXY_ALLOW_NONLOCAL=1`, which means anyone who can reach the deployment +can use its keys. + +The `/data` volume still holds the generated key secret and admin password +with Postgres. Alternatively pin `LLMPROXY_KEY_SECRET` and `LLMPROXY_ADMIN_PASSWORD` through the environment. + +## What the image does + +| | | +|---|---| +| User | `llmproxy`, uid 1000; the process never runs as root | +| State | `/data`, owned by that user | +| Env defaults | `LLMPROXY_DATABASE_URL=/data/llmproxy.db`, `LLMPROXY_SECRET_FILE=/data/secret`, `LLMPROXY_ADMIN_PASSWORD_FILE=/data/admin-password` | +| Bind | `LLMPROXY_HOST=0.0.0.0`, `LLMPROXY_PORT=4000`, port 4000 exposed | +| Entrypoint | `llmproxy`, with `CMD ["serve"]`, so `docker run llmproxy key create ...` runs the CLI against the same `/data` | + +Build it yourself with `just docker` (`docker build -t llmproxy .`). + +## Where next + +- [upgrades.md](upgrades.md) for backup and version pinning. +- [../guides/providers.md](../guides/providers.md) to register your first + upstream. +- [../reference/configuration.md](../reference/configuration.md) for every + environment variable. diff --git a/docs/install/upgrades.md b/docs/install/upgrades.md new file mode 100644 index 0000000..5323b76 --- /dev/null +++ b/docs/install/upgrades.md @@ -0,0 +1,69 @@ +# Upgrades and backup + +Upgrading is: back up, stop, replace the binary, start. There is no separate +migration step and no external migration tool. + +## Back up + +Take a backup before every upgrade. There are no down migrations, so a backup +is the only way back. + +Two things to copy, both of them: + +```bash +# SQLite, while the service is running +sqlite3 /var/lib/llmproxy/llmproxy.db ".backup /backup/llmproxy-$(date +%F).db" + +# The key secret +cp /var/lib/llmproxy/secret /backup/llmproxy-secret +``` + +With the service stopped, copying the `.db` file directly works too. For +Postgres, use `pg_dump` as usual; the secret still lives on disk and still +needs its own copy. + +The database without the secret is not a usable backup: every API key stops +validating and every stored upstream credential becomes undecryptable. Keep +the two backups in different places. + +## Upgrade + +```bash +systemctl stop llmproxy +curl -fsSL https://github.com/greyhavenhq/llmproxy/releases/latest/download/llmproxy_Linux_x86_64.tar.gz | tar xz +sudo install -m 0755 llmproxy /usr/local/bin/llmproxy +systemctl start llmproxy +systemctl status llmproxy +``` + +For containers, change the image tag and redeploy; see +[docker.md](docker.md). + +The server applies any pending schema migrations on boot. A migration that +fails is a startup error, not a warning: the server refuses to serve rather +than run against a schema it cannot account for. Check the logs if it does not +come back up. + +## What to know about migrations + +- **Each migration runs at most once.** The change and its bookkeeping row + commit in the same transaction, so a database is never left half-migrated. +- **The applied history is queryable:** + `SELECT version, applied_at FROM schema_migration ORDER BY version`. +- **Downgrades are not supported.** Roll back by restoring the backup. +- **Data a migration cannot invent stays empty rather than wrong.** Keys + minted before `api_key.key_suffix` existed show an empty display suffix + until they are recreated. +- **Never regenerate the secret file during an upgrade.** Its format is stable + and the server never rewrites it. + +## Version pinning + +Pin the exact version in production. For binaries, install a specific release +rather than `latest`. For containers, use `:1.1.0` rather than `:latest` or +`:1`, both of which move under you. + +## Where next + +- [binary.md](binary.md) for the systemd setup. +- [docker.md](docker.md) for the container route. From 8ee8512f65d07174c63724b98ed7b920727b21f7 Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:12:53 -0600 Subject: [PATCH 3/6] docs: split and rewrite the operator guides providers-and-models.md becomes providers.md (register, edit, rotate, TLS, overrides, disable, delete) and models.md (discover, bind, alias targets, edit, hide, resolve, delete), each opening with the task rather than the concept. keys-and-usage.md becomes api-keys.md, pricing.md and usage.md. sso.md leads with the config table and the walkthrough, and ends with an offboarding checklist. transparent-relay.md becomes claude-code.md, setup first. Deleted: the capability design rationale, the alias-as-promise passage, the cached-token normalisation treatise (a two-sentence definition remains, pointing at concepts/architecture.md), the relay-token justification, the UI feature inventory in the stats section. The admin audit trail and pagination sections move to reference/api.md in the next commit. --- docs/guides/api-keys.md | 434 +++++-------------------------------- docs/guides/claude-code.md | 268 ++++++++++++----------- docs/guides/models.md | 238 ++++++++++++++++++++ docs/guides/pricing.md | 114 ++++++++++ docs/guides/providers.md | 414 +++++++---------------------------- docs/guides/sso.md | 257 +++++++++++----------- docs/guides/usage.md | 228 +++++++++++++++++++ 7 files changed, 978 insertions(+), 975 deletions(-) create mode 100644 docs/guides/models.md create mode 100644 docs/guides/pricing.md create mode 100644 docs/guides/usage.md diff --git a/docs/guides/api-keys.md b/docs/guides/api-keys.md index 75f7602..46ec0a6 100644 --- a/docs/guides/api-keys.md +++ b/docs/guides/api-keys.md @@ -1,4 +1,10 @@ -# Keys and usage +# Manage API keys + +Every API key belongs to a principal (a person or a service) and inherits that +principal's role. Keys start with `lp_`, are generated server-side, and are +shown exactly once: the database stores only a keyed hash plus the last 4 +characters for display. There is no way to view a key again, only to mint a +new one. Examples assume: @@ -8,20 +14,10 @@ KEY=lp_... # any key ADMIN=lp_... # an admin key ``` -## Key lifecycle - -Every API key belongs to a principal (a user or a service) and inherits that -principal's role. Keys are random 32-byte values with an `lp_` prefix, -generated server-side; the database stores only an HMAC-SHA256 digest under -the server secret plus the plaintext's last 4 characters for display -(listings show `***xxxx`), so the plaintext exists exactly once, in the -creation response (or CLI output). There is no way to view a key again, only -to mint a new one. - -Four ways keys come into existence: +## Mint a key for yourself -**Self-service.** Any key or session holder manages their own keys, either in -the built-in UI at `/` or directly: +Any key or session holder manages their own keys, in the UI under Keys or +directly: ```bash curl -s $P/my/keys -H "authorization: Bearer $KEY" \ @@ -39,408 +35,76 @@ curl -s $P/my/keys -H "authorization: Bearer $KEY" \ } ``` -`GET /my/keys` lists metadata only (no `key` field, up to 500 keys), and -`DELETE /my/keys/{id}` deletes the key outright: +Copy `key` now. Listings show only `***Mq3w`. ```bash -curl -s -X DELETE $P/my/keys/8c1de2f4a6b84c33a1f09e7d5b246810 \ - -H "authorization: Bearer $KEY" -# {"deleted": "8c1de2f4a6b84c33a1f09e7d5b246810"} +curl -s $P/my/keys -H "authorization: Bearer $KEY" # metadata only, max 500 ``` -**Admin-issued.** An admin sees and revokes every key in the UI (Admin, All -keys) and mints keys for any principal by name over the API: +## Mint a key for someone else + +An admin sees and revokes every key in the UI under Admin, All keys, and mints +keys for any principal by name: ```bash curl -s $P/admin/v1/keys -H "authorization: Bearer $ADMIN" \ -H 'content-type: application/json' -d '{"principal": "alice", "label": "onboarding"}' ``` -The response is the same shape plus a `principal` field. Admins list keys -across principals (`GET /admin/v1/keys?principal=alice`) and delete any key -(`DELETE /admin/v1/keys/{id}`). +The response is the same shape plus a `principal` field. List across +principals with `GET /admin/v1/keys?principal=alice`. + +## Create a key for a service -**Service principals.** For workloads, create a `service` principal so the -key survives staff turnover and shows up under its own name in usage. In the -UI (Admin, Services) one step creates the service with its key; over the API: +For workloads, create a `service` principal first. The key then survives staff +turnover and shows up under its own name in usage. In the UI, Admin, Services +does both steps at once. Over the API: ```bash curl -s $P/admin/v1/principals -H "authorization: Bearer $ADMIN" \ -H 'content-type: application/json' -d '{"name": "batch-service", "kind": "service"}' + curl -s $P/admin/v1/keys -H "authorization: Bearer $ADMIN" \ -H 'content-type: application/json' -d '{"principal": "batch-service", "label": "prod"}' ``` -**CLI.** `llmproxy key create|list|delete` and `llmproxy principal create` -operate directly on the database; this is how the first admin key is minted -before any HTTP credential exists. See [installation.md](installation.md). - -Deletion is the revocation mechanism and takes effect immediately (the -database is checked on every request). A deleted key authenticates like any -unknown key: 401 `invalid_api_key`. Usage history survives deletion; usage -events reference the key id, which is never reused. `last_used_at` is -refreshed at most once per minute per key, so treat it as coarse. - -## Usage model - -Every `/v1` request that reaches resolution produces one `usage_event` row -carrying identifiers, flags and numbers: principal, key, provider, alias, -upstream name, endpoint, client, status code, outcome (`ok`, `upstream_error`, -`unreachable`, `cancelled`), error kind, cancelled and streamed flags, -duration, cost. `error_kind` classifies a failure without storing any of it: -the transport class on `unreachable` (`timeout`, `connection_error`, -`response_too_large`), the upstream's `error.type`/`error.code` token on -`upstream_error` (`rate_limit_error`, `overloaded_error`, ...), sanitised to -a short lowercase identifier and dropped whole when it does not look like -one. Provider error messages are never stored; they can echo request -content. -`client` is the caller's `User-Agent` header, truncated to 256 characters and -stored verbatim, so the tool behind a key is visible and can change over time -("claude-cli/2.0.13 (external, cli)", "openai-python/1.51.0", ...). Grouping -by product family happens at read time; the stored value stays exact. -Attached to it are per-unit `usage_quantity` rows: `(unit, quantity, -unit_price, priced)` with units `input_tokens`, `output_tokens`, -`cached_input_tokens` (from `prompt_tokens_details.cached_tokens` when -reported), `cache_creation_tokens` (Anthropic cache writes, via the -[transparent relay](transparent-relay.md)) and `audio_seconds` reserved for -transcription. - -Traffic on the [transparent Anthropic relay](transparent-relay.md) lands in -the same tables, attributed via relay token instead of API key, with the -sentinel provider `transparent:anthropic` and the model name as Anthropic -reports it. - -Quantities come from the upstream's `usage` object: read from the JSON body -on unary responses, and from SSE chunks on streams (the proxy forces -`stream_options.include_usage` so the final chunk carries usage). Cancelled -and errored requests are recorded too, flagged accordingly, with whatever -partial usage the upstream reported before the end. Accounting runs detached -from the request path, so a client hanging up cannot lose the record. - -One asymmetry is normalised at read time. The OpenAI shape reports cached -tokens as a subset of the prompt count (`prompt_tokens_details.cached_tokens -⊆ prompt_tokens`), while Anthropic reports cache reads and writes *outside* -`input_tokens`. Stored quantities stay raw as reported (pricing multiplies -each unit by its own price, so cache reads and writes keep their own rates), -but every aggregate (`/my/usage*`, `/admin/v1/usage*`, `/stats/*`) -normalises `input_tokens` to the *non-cached* input: the OpenAI cached -subset is subtracted back out, and the relay's `cache_creation_tokens` -(fresh input billed at a premium rate) folds in while cache reads stay -their own unit. So `input_tokens` always means "input processed at the full -input rate" and is comparable across providers, and `cached_input_tokens` -is always the cheap cache-read count on top of it. An OpenAI-compatible -upstream that never reports `cached_tokens` (no prefix caching, or caching -not surfaced) simply shows no cached quantity; absence means "not reported", -never "free". - -No content is ever persisted. The schema has no column that could hold a -prompt, completion or embedding input, and a test rejects content-shaped -column names in the DDL. - -## Application tags - -A key says who is calling; a tag says which of their applications. Callers -send `x-llmproxy-tags` with comma-separated `key:value` pairs: - -``` -x-llmproxy-tags: app:dataindex,context:search -``` +## Mint the first key -The proxy stores the pairs on the usage event and nothing else about them. -The header works on both ingresses, including the [transparent -relay](transparent-relay.md), where it is consumed by the proxy and stripped -before the request is forwarded. - -Values are normalised at capture into one canonical string, so grouping is -stable: each pair is trimmed and lowercased, must match -`[a-z0-9][a-z0-9._-]*:[a-z0-9][a-z0-9._-]*`, keys are unique (first -occurrence wins), pairs are sorted by key, and the list is capped at 8 pairs -and 256 bytes. Anything malformed is dropped silently: telemetry never fails -a request. - -The stats endpoints take a repeatable `tag=:` filter (exact -pairs, up to four, narrowing together), `/stats/summary` reports `tags` as a -dimension, and `/stats/requests/facets` lists the pairs seen in the window. -The Usage tab's **Apps** subtab draws it: spend, requests and tokens per app, -the contexts inside an app, and an app-by-model table. Events with no `app` -tag are grouped as `untagged`, so no traffic drops out of the breakdowns. - -## Pricing - -Prices belong to the model. Each binding carries a price per unit, per -million units, set when the model is bound or edited: - -```bash -curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"pricing": {"input_tokens": 0.4, "output_tokens": 1.2}}' -``` - -`pricing` is the model's complete price set: units left out are unpriced -afterwards, so the same call clears a price by omitting its unit. Every model -read (`GET /admin/v1/models`) returns the prices that apply to it: - -```json -{ - "alias": "qwen-72b", - "upstream_name": "Qwen/Qwen2.5-72B-Instruct", - "pricing": {"input_tokens": 0.4, "output_tokens": 1.2}, - "pricing_inherited": false -} -``` - -Prices resolve against the names a model is known by, most specific first: its -own name, the model it points at (see -[aliases](providers-and-models.md#point-a-name-at-another-model)), then the -upstream model name. So a name inherits a price without repeating it, and can -override it per unit by setting its own. `pricing_inherited` is true when the -price in force is not keyed on this name. - -Underneath, prices are a versioned feed keyed on `(model, unit)`, so a whole -price list can also be loaded at once via `POST /admin/v1/pricing` (or -`LLMPROXY_PRICING_FILE` at startup): +Before any HTTP credential exists, use the CLI. It operates directly on the +database: ```bash -curl -s $P/admin/v1/pricing -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{ - "version": "2026-07", - "entries": [ - {"model": "qwen-72b", "unit": "input_tokens", "price_per_million": 0.4}, - {"model": "qwen-72b", "unit": "output_tokens", "price_per_million": 1.2}, - {"model": "e5-embed", "unit": "input_tokens", "price_per_unit": 0.00000002} - ] - }' -# {"version": "2026-07", "count": 3} +llmproxy key create -label bootstrap +llmproxy key list +llmproxy key delete +llmproxy principal create -name batch-service -kind service ``` -The feed format: a required string `version`, and `entries` where each entry -has `model`, a `unit` from `input_tokens | output_tokens | -cached_input_tokens | cache_creation_tokens | audio_seconds`, and exactly one -of `price_per_unit` or -`price_per_million` (the latter is divided by 1,000,000 on load). `model` can -be either the public alias or the upstream model name; lookup tries the alias -first. Anything else is rejected with 400 `invalid_pricing_feed`. +Run it with the same working directory or environment as the server, or it +will write to a different database. See +[../install/binary.md](../install/binary.md#first-login). -Loading a feed replaces the active one atomically and applies to new requests -immediately; already-recorded events are not repriced. `GET /admin/v1/pricing` -returns the active version (`null` if none), the entry count and the entries -themselves (per-million prices). Editing one model's prices publishes a new -version too, carrying every other model's entries over unchanged. +## Revoke a key -A `(model, unit)` pair with no entry is recorded as **unpriced, never as -zero**: the quantity row gets `priced: false`, the event is flagged -`unpriced`, summaries show `cost: null` for fully unpriced groups, and -`/metrics` counts the volume under `llmproxy_unpriced_units_total`. Silence -never masquerades as "free". - -## Reading usage - -`GET /my/usage` returns the caller's own usage grouped by (model, endpoint); -`since` and `until` take ISO 8601 dates or datetimes (until is exclusive): +Deletion is the revocation mechanism and takes effect on the next request: ```bash -curl -s "$P/my/usage?since=2026-07-01&until=2026-08-01" -H "authorization: Bearer $KEY" -``` - -```json -{ - "usage": [ - { - "model": "qwen-72b", - "endpoint": "chat", - "requests": 412, - "cancelled": 3, - "cost": 1.20734, - "units": {"input_tokens": 1882340, "output_tokens": 378512} - } - ] -} -``` - -`GET /admin/v1/usage/summary` is the same aggregation across everyone, each -row carrying a `principal` name, optionally filtered with `?principal=`: - -```bash -curl -s "$P/admin/v1/usage/summary?since=2026-07-01" -H "authorization: Bearer $ADMIN" -``` - -```json -{ - "usage": [ - { - "principal": "batch-service", - "model": "e5-embed", - "endpoint": "embeddings", - "requests": 9120, - "cancelled": 0, - "cost": null, - "units": {"input_tokens": 44021833} - }, - { - "principal": "alice", - "model": "qwen-72b", - "endpoint": "chat", - "requests": 412, - "cancelled": 3, - "cost": 1.20734, - "units": {"input_tokens": 1882340, "output_tokens": 378512} - } - ] -} -``` - -## Usage over time - -`GET /my/usage/series` (and `GET /admin/v1/usage/series` across everyone, -optionally filtered with `?principal=`) is the same accounting bucketed -by time. `bucket` is `hour | day | week | month` (default `day`), `since` and -`until` bound the window as above: - -```bash -curl -s "$P/my/usage/series?bucket=day&since=2026-07-01" -H "authorization: Bearer $KEY" -``` - -```json -{ - "bucket": "day", - "series": [ - { - "start": "2026-07-01T00:00:00Z", - "requests": 41, - "ok": 38, - "cancelled": 1, - "failed": 2, - "unpriced_requests": 3, - "cost": 0.10734, - "units": {"input_tokens": 188234, "output_tokens": 37851} - }, - { - "start": "2026-07-02T00:00:00Z", - "requests": 0, - "ok": 0, - "cancelled": 0, - "failed": 0, - "unpriced_requests": 0, - "cost": null, - "units": {} - } - ] -} -``` - -Buckets are UTC (weeks start Monday) and the range is gap-filled: quiet -periods come back as empty buckets rather than as holes, so a series can be -plotted directly. `ok`, `cancelled` and `failed` partition `requests`, where -failed is any non-ok outcome the caller did not cancel (`upstream_error`, -`unreachable`). `cost` is null when nothing in the bucket had a price, and -`unpriced_requests` counts the requests that hit a missing price. Without -`since`, the series starts at the first recorded event; a window needing more -than 1000 buckets is refused with 400 `range_too_large`. - -## Team statistics - -The `/stats` endpoints serve the same accounting to **every authenticated -user**: the proxy's usage is team-visible by design, and the admin role gates -configuration, not visibility. They take the same optional filters: -`principal=`, `key=`, `provider=` (the sentinel -`transparent:anthropic` included; a deleted provider's events resolve to -`(deleted)`), `model=`, `client=` (a prefix match on the -stored User-Agent, so `client=claude-cli` covers every version), -`tag=:` (an exact pair, repeatable up to four times, all of which -must match), `outcome=` (`ok`, `upstream_error`, `unreachable`, -`cancelled`, or the meta value `failed` matching everything not ok; an -unknown value is a 400 `invalid_outcome`) and the `since`/`until` window. - -The `key` filter matches API keys only. Relay traffic authenticates with a -relay token, whose id occupies the same column, so relay events never match a -key filter and show no key; reach them through -`provider=transparent:anthropic` instead. - -* `GET /stats/series`: the bucketed series above, with the extra filters. -* `GET /stats/summary`: the window aggregated over every recorded dimension: - one row per `(principal, provider, model, endpoint, client, tags)`, so any - roll-up (by model, by user, by client, by app) is a client-side sum. **Completed - requests with a model only**: a failure's model and provider are whatever - the caller asked for, and model-less events (the relay's `count_tokens` - and `models` calls report no model and no usage) have no place in a - by-model breakdown. Both stay visible in the series and in the request - log. -* `GET /stats/requests`: one page of the request metadata log with - quantities, newest first, failures included. `?limit=` (max 500) and - `?offset=` page it; the response carries `{requests, limit, offset, - total}`, where `total` is the size of the whole filtered set. -* `GET /stats/requests/facets`: the distinct `principals`, `keys`, - `providers`, `models`, `clients` and `tags` present in the `since`/`until` - window, for the explorer's filter options. Derived from every event, not just the - completed ones the summary keeps, so a user or key that has only ever - failed is still selectable. Each list is capped at 500 values. Key labels - and last-4 suffixes are visible to every authenticated user here, as the - rest of the stats surface already is. -* `GET /stats/errors`: the errors dashboard in one call. `series` is the - gap-filled bucketed counts per outcome (`?bucket=` as in the series - endpoint); `breakdown` is one cell per `(provider, model, endpoint, - client, tags, outcome, error_kind, status_code)` with its request count, - average duration, last-seen timestamp and `bands`: counts of the cell's - requests by time-to-outcome (`<1s`, `1-5s`, `5-15s`, `15-30s`, `30-60s`, - `60-120s`, `>=120s`), so a cluster of cancellations at 120s reads as the - caller's timeout rather than a crash. Cells with outcome `ok` are included - so error rates per dimension have their denominator. - -```bash -curl -s "$P/stats/summary?since=2026-07-01&client=claude-cli" -H "authorization: Bearer $KEY" -``` - -```json -{ - "usage": [ - { - "principal": "alice", - "provider": "transparent:anthropic", - "model": "claude-opus-5", - "endpoint": "v1/messages", - "client": "claude-cli/2.0.13 (external, cli)", - "tags": "app:dataindex,context:search", - "requests": 3211, - "cancelled": 12, - "cost": 41.20734, - "units": {"input_tokens": 311140, "output_tokens": 378512, "cached_input_tokens": 1571200} - } - ] -} +curl -s -X DELETE $P/my/keys/8c1de2f4a6b84c33a1f09e7d5b246810 \ + -H "authorization: Bearer $KEY" +# {"deleted": "8c1de2f4a6b84c33a1f09e7d5b246810"} ``` -This is what the UI draws: the Usage tab (an Overview subtab with tiles, -per-period charts, the model share, the client breakdown and the by-user -table, filterable by user, provider and client; an Apps subtab breaking -the same window down by application tag; a Models subtab listing every -servable model with its capabilities, what it aliases and its usage over the -window, sortable by name or by any usage column; and an Errors subtab: failures over -time by class, the time-to-failure histogram, failure counts and rates by -provider, app and model, the failure-kind share and an error-signatures -table that deep-links into the explorer) and the Requests tab (the request -explorer: a window preset or a custom UTC date range, filters for user, key, -client, app, model, provider and outcome, eight columns with the rest of the -metadata in an expandable row, and numbered pages with a total). The older -`/my/usage*` and `/admin/v1/usage*` endpoints stay as documented above. - -## Metrics +Admins delete any key with `DELETE /admin/v1/keys/{id}`. -`GET /metrics` (unauthenticated) exposes Prometheus-text counters with -aggregate labels only; per-principal detail deliberately stays in the -database so metric cardinality stays bounded. +A deleted key authenticates like any unknown key: 401 `invalid_api_key`. Usage +history survives deletion, because usage events reference the key id, which is +never reused. -| Metric | Labels | Meaning | -|---|---|---| -| `llmproxy_requests_total` | `endpoint`, `provider`, `model`, `outcome` | Requests by outcome (`ok`, `upstream_error`, `unreachable`, `cancelled`) | -| `llmproxy_request_seconds_sum` | `endpoint`, `provider`, `model` | Total request duration | -| `llmproxy_request_seconds_count` | `endpoint`, `provider`, `model` | Request count (divide sum by count for the mean) | -| `llmproxy_usage_units_total` | `provider`, `model`, `unit` | Consumed units | -| `llmproxy_unpriced_units_total` | `model`, `unit` | Units that had no price entry; alert on growth here to catch pricing gaps | +`last_used_at` is refreshed at most once per minute per key, so treat it as +coarse. -`model` is always the public alias. Sample output: +## Where next -``` -llmproxy_requests_total{endpoint="chat",model="qwen-72b",outcome="ok",provider="vllm-1"} 412 -llmproxy_usage_units_total{model="qwen-72b",provider="vllm-1",unit="input_tokens"} 1.88234e+06 -llmproxy_unpriced_units_total{model="e5-embed",unit="input_tokens"} 4.4021833e+07 -``` +- [usage.md](usage.md) to see what each key consumed. +- [sso.md](sso.md) for group-derived roles and the offboarding sequence. +- [claude-code.md](claude-code.md) for relay tokens, which are not API keys. diff --git a/docs/guides/claude-code.md b/docs/guides/claude-code.md index ea88e56..a3d89dc 100644 --- a/docs/guides/claude-code.md +++ b/docs/guides/claude-code.md @@ -1,11 +1,9 @@ -# Transparent Anthropic relay +# Meter Claude Code -The transparent relay forwards Anthropic API traffic unchanged while -recording who consumed what. It exists for tools that already hold their own -Anthropic credentials, Claude Code first among them: the proxy substitutes -nothing, stores no upstream secrets, and rewrites no bytes. What it adds is -per-user usage accounting on the same tables, dashboards and metrics as the -rest of llmproxy. +The transparent relay forwards Anthropic API traffic unchanged while recording +who consumed what. It is for tools that already hold their own Anthropic +credentials, Claude Code first among them: the proxy substitutes nothing, +stores no upstream secrets and rewrites no bytes. ``` Claude Code ── x-api-key / OAuth bearer ──▶ llmproxy ──▶ api.anthropic.com @@ -14,69 +12,63 @@ Claude Code ── x-api-key / OAuth bearer ──▶ llmproxy ──▶ api.ant duration, status; no content) ``` -This is distinct from the `/v1` ingress: there the proxy owns the provider -credential and routes by model alias. Here the caller owns the credential and -the proxy is a bystander that meters. +This is not the `/v1` ingress. There the proxy owns the provider credential +and routes by model name. Here the caller owns the credential and the proxy +meters. -## Relay tokens +Examples assume: -Requests go to: - -``` -/transparent/anthropic// +```bash +P=https://proxy.example.com +KEY=lp_... # any key +ADMIN=lp_... # an admin key ``` -The relay token attributes traffic to a principal. It has the same lifecycle -mechanics as an API key: a random secret with an `lpt_` prefix, generated -server-side, stored only as a keyed hash plus a 4-character display suffix, -shown exactly once at creation, revoked by deletion. But it is deliberately -not an API key. A relay token cannot authenticate against `/v1`, `/my` or -`/admin`, and an API key is not accepted on the relay. Leaking one exposes -usage attribution, not proxy access; the Anthropic credential itself never -touches the proxy's storage at all. +## Set it up -Mint one in the UI (Keys page) or directly: +1. Mint a relay token, in the UI under Keys or directly: -```bash -curl -s $P/my/relay-tokens -H "authorization: Bearer $KEY" \ - -H 'content-type: application/json' -d '{"label": "claude-code laptop"}' -``` + ```bash + curl -s $P/my/relay-tokens -H "authorization: Bearer $KEY" \ + -H 'content-type: application/json' -d '{"label": "claude-code laptop"}' + ``` -```json -{ - "id": "5e9c1af2d4b04c33a1f09e7d5b246810", - "token_suffix": "Xk2m", - "label": "claude-code laptop", - "created_at": "2026-08-03T10:02:11.482235Z", - "last_used_at": null, - "token": "lpt_Vb0mJq..." -} -``` + ```json + { + "id": "5e9c1af2d4b04c33a1f09e7d5b246810", + "token_suffix": "Xk2m", + "label": "claude-code laptop", + "created_at": "2026-08-03T10:02:11.482235Z", + "last_used_at": null, + "token": "lpt_Vb0mJq..." + } + ``` -`GET /my/relay-tokens` lists metadata only, `DELETE /my/relay-tokens/{id}` -revokes immediately (the next relayed request gets 404 -`unknown_relay_token`). The token appears in the URL path, so the proxy's -access log masks that segment (`/transparent/anthropic/***Xk2m/v1/messages`), -and the prefix is stripped before forwarding, so nothing token-shaped reaches -Anthropic. + Copy `token` now. It is shown once. -## Claude Code setup +2. Point Claude Code at the relay: -Claude Code sends all API traffic to `ANTHROPIC_BASE_URL` and appends the -standard paths (`/v1/messages`, `/v1/messages/count_tokens`, ...). The UI -shows this exact command when a token is created: + ```bash + export ANTHROPIC_BASE_URL=https://proxy.example.com/transparent/anthropic/lpt_Vb0mJq... + claude + ``` -```bash -export ANTHROPIC_BASE_URL=https://proxy.example.com/transparent/anthropic/lpt_Vb0mJq... -claude -``` + The UI shows this exact command when the token is created. + +3. Run a prompt, then check the Usage tab. The request appears with provider + `transparent:anthropic` and client `claude-cli/`. + +Your Anthropic credential is unchanged by all of this: Claude Code keeps +sending its own, and the proxy forwards it untouched. + +## Make it permanent -To make it permanent, pick one: +Pick one: -* **Shell profile**: append the `export` line to `~/.zshrc` / `~/.bashrc`. - Everything on that machine that honours `ANTHROPIC_BASE_URL` goes through - the relay. -* **Claude Code settings**: scope it to Claude Code only, in +- **Shell profile.** Append the `export` line to `~/.zshrc` or `~/.bashrc`. + Everything on that machine honouring `ANTHROPIC_BASE_URL` goes through the + relay. +- **Claude Code settings**, to scope it to Claude Code only, in `~/.claude/settings.json`: ```json @@ -86,67 +78,58 @@ To make it permanent, pick one: } } ``` -* **Alias**: proxy Claude Code while other Anthropic tooling stays direct: + +- **Alias**, to relay Claude Code while other Anthropic tooling stays direct: ```bash alias claude='ANTHROPIC_BASE_URL=https://proxy.example.com/transparent/anthropic/lpt_Vb0mJq... claude' ``` Both authentication styles work, because credentials pass through -byte-for-byte: +byte-for-byte: an `x-api-key` is forwarded as received, and a Claude +subscription's `Authorization: Bearer` plus `anthropic-beta` OAuth header are +too. Token refresh happens against claude.ai directly and never crosses the +relay. -* **API key**: `x-api-key` forwarded as received. -* **Claude subscription (OAuth)**: `Authorization: Bearer` plus the - `anthropic-beta` OAuth header forwarded as received. Token refresh happens - against claude.ai directly and never crosses the relay. +Any other Anthropic SDK or tool honouring a base URL works the same way. -Any other Anthropic SDK or tool that honours a base URL works the same way. +## Relay tokens are not API keys -## What the relay does and does not do +A relay token has the same lifecycle mechanics as an API key: `lpt_` prefix, +generated server-side, stored as a keyed hash plus a 4-character display +suffix, shown once, revoked by deletion. -Forwarded verbatim: method, path suffix, query string, request body (streamed -unbuffered, no size cap, no field rewrites), response body (SSE flushed -chunk-by-chunk), and headers in both directions apart from hop-by-hop -headers, `Cookie` (proxy-local browser state), `Accept-Encoding` (dropped -so the response arrives unencoded and usage stays readable; clients that -asked for gzip still get a correct identity response) and `x-llmproxy-tags` -(addressed to the proxy: read for accounting, then stripped before -forwarding). Anthropic's -`request-id` and rate-limit headers come back untouched. Client disconnects -cancel the upstream request. - -Every method on every sub-path relays. Usage quantities are extracted where a -`usage` object exists (`/v1/messages`, unary and streamed); other endpoints -(`count_tokens`, `models`, future additions) still produce a metadata-only -usage event, so the request log stays complete. The exception is HEAD and -OPTIONS: they relay but are never recorded. They are connectivity probes -(Claude Code sends one per session start, which Anthropic's root answers -with a 404), carry nothing to meter, and would only fill the request log -with phantom failures. +It is deliberately a different credential. A relay token cannot authenticate +against `/v1`, `/my` or `/admin`, and an API key is not accepted on the relay. +Leaking one exposes usage attribution, not proxy access. -The proxy never buffers a whole streamed body and persists no request or -response content anywhere, same guarantee as the rest of llmproxy. - -## Accounting - -Each relayed request writes one `usage_event`: - -* `provider` is the sentinel `transparent:anthropic`; there is no provider - row and nothing to configure in the catalog. -* `model` is taken from the response (`message_start` on streams, the - top-level `model` field on unary bodies), never parsed out of the request. -* `endpoint` is the relayed path suffix (`v1/messages`, - `v1/messages/count_tokens`, ...). -* `client` is the caller's `User-Agent` header, truncated to 256 characters - (Claude Code sends `claude-cli/ ...`), so different tools behind - the same token stay distinguishable. Header metadata only, like everything - else here. -* `tags` is the caller's `x-llmproxy-tags` header, normalised the same way as - on `/v1`, so relay traffic shows up in the Usage tab's Apps subtab too. See - [keys-and-usage.md](keys-and-usage.md#application-tags). -* Outcome, status code, streamed/cancelled flags and duration behave exactly - as on `/v1`; a mid-stream client disconnect records the partial usage - reported so far, flagged `cancelled`. +```bash +curl -s $P/my/relay-tokens -H "authorization: Bearer $KEY" # metadata only +curl -s -X DELETE $P/my/relay-tokens/ -H "authorization: Bearer $KEY" # revoke +``` + +Revocation is immediate: the next relayed request gets 404 +`unknown_relay_token`. + +The token sits in the URL path, so the access log masks that segment +(`/transparent/anthropic/***Xk2m/v1/messages`), and the prefix is stripped +before forwarding, so nothing token-shaped reaches Anthropic. + +## What the relay records + +One `usage_event` per relayed request: + +| Field | Value | +|---|---| +| `provider` | The sentinel `transparent:anthropic`. There is no provider row and nothing to configure in the catalog | +| `model` | Taken from the response (`message_start` on streams, the top-level `model` on unary bodies), never parsed out of the request | +| `endpoint` | The relayed path suffix: `v1/messages`, `v1/messages/count_tokens`, ... | +| `client` | The caller's `User-Agent`, truncated to 256 characters, so different tools behind one token stay distinguishable | +| `tags` | The caller's `x-llmproxy-tags` header, normalised as on `/v1`. See [usage.md](usage.md#tag-requests-by-application) | + +Outcome, status code, streamed and cancelled flags and duration behave exactly +as on `/v1`. A mid-stream disconnect records the partial usage reported so +far, flagged `cancelled`. Units map from Anthropic's usage object: @@ -158,26 +141,65 @@ Units map from Anthropic's usage object: | `cache_creation_input_tokens` | `cache_creation_tokens` (recorded when non-zero) | Stored quantities are raw as Anthropic reported them, so each unit prices at -its own rate. The usage aggregates normalise `input_tokens` to the -non-cached input at read time: the relay's `cache_creation_tokens` (fresh -input billed at a premium rate) fold in, cache reads stay their own unit, -and the OpenAI shape's cached subset is subtracted out; see -[keys-and-usage.md](keys-and-usage.md#usage-model). - -Pricing uses the normal feed keyed on the model name as Anthropic reports it -(e.g. `claude-opus-5`); see [keys-and-usage.md](keys-and-usage.md#pricing). -Unpriced units are flagged, never counted as free. - -Relay traffic appears everywhere ordinary traffic does: `/my/usage`, -`/my/usage/series`, the admin summaries, the request log and `/metrics` +its own rate. + +Endpoints without a `usage` object (`count_tokens`, `models`) still produce a +metadata-only event, so the request log stays complete. HEAD and OPTIONS +relay but are never recorded: they are connectivity probes and would only fill +the log with phantom failures. + +Relay traffic appears everywhere ordinary traffic does: `/my/usage`, the +series endpoints, the admin summaries, the request log and `/metrics` (labelled `provider="transparent:anthropic"`). +## Price relayed models + +Use the normal pricing feed, keyed on the model name as Anthropic reports it: + +```bash +curl -s $P/admin/v1/pricing -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{ + "version": "2026-08", + "entries": [ + {"model": "claude-opus-5", "unit": "input_tokens", "price_per_million": 5.0}, + {"model": "claude-opus-5", "unit": "output_tokens", "price_per_million": 25.0} + ] + }' +``` + +Unpriced units are flagged, never counted as free. See +[pricing.md](pricing.md). + +## What passes through + +Forwarded verbatim: method, path suffix, query string, request body (streamed +unbuffered, no size cap, no field rewrites), response body (SSE flushed +chunk by chunk), and headers in both directions. Anthropic's `request-id` and +rate-limit headers come back untouched. Client disconnects cancel the upstream +request. + +Four headers are the exception: + +| Header | Why | +|---|---| +| Hop-by-hop headers | Not forwardable by definition | +| `Cookie` | Proxy-local browser state | +| `Accept-Encoding` | Dropped so the response arrives unencoded and usage stays readable. Clients that asked for gzip still get a correct identity response | +| `x-llmproxy-tags` | Addressed to the proxy: read for accounting, then stripped | + +The proxy never buffers a whole streamed body and persists no request or +response content, same guarantee as the rest of llmproxy. + ## Configuration | Variable | Default | Meaning | |---|---|---| -| `LLMPROXY_TRANSPARENT_ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Relay target. The relay only ever connects here, so it is not an open proxy. Empty disables the relay (404 `transparent_relay_disabled`). | +| `LLMPROXY_TRANSPARENT_ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Relay target. The relay only ever connects here, so it is not an open proxy. Empty disables the relay (404 `transparent_relay_disabled`) | + +The upstream connection allows up to 10 minutes before response headers and +never bounds a streaming response; the client hanging up is what ends a relay. + +## Where next -The upstream connection allows up to 10 minutes before response headers -(long non-streaming requests think for a while) and never bounds a streaming -response; the client hanging up is what ends a relay. +- [usage.md](usage.md) to read the resulting accounting. +- [pricing.md](pricing.md) to attach prices. diff --git a/docs/guides/models.md b/docs/guides/models.md new file mode 100644 index 0000000..468e41f --- /dev/null +++ b/docs/guides/models.md @@ -0,0 +1,238 @@ +# Manage models + +Three words used throughout: + +- A **provider** is an upstream endpoint. See + [providers.md](providers.md). +- A **model** binds a caller-facing name to one (provider, upstream model) + pair, with a capability set and prices. +- A model can instead be a **name for another model**, inheriting everything + from its target. + +Caller-facing names are globally unique across all providers, so resolution is +never ambiguous. Everything here is also in the built-in UI under Admin, +Models. + +Examples assume: + +```bash +P=http://127.0.0.1:4000 +ADMIN=lp_... # an admin key +``` + +## Discover what a provider serves + +```bash +curl -s $P/admin/v1/providers/vllm-1/discover -H "authorization: Bearer $ADMIN" +``` + +```json +{ + "provider": "vllm-1", + "discovered_at": "2026-07-29T09:15:40.582211Z", + "models": [ + {"upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", "bound_alias": null}, + {"upstream_name": "intfloat/e5-mistral-7b-instruct", "bound_alias": "e5-embed"} + ] +} +``` + +Discovery asks the provider's `/models` endpoint and annotates each name with +any existing binding. It is read-only and creates nothing. + +Failures: 502 `provider_unreachable` if the upstream cannot be reached, 502 +`discovery_failed` if it answers with an error status or invalid JSON. + +## Bind a model + +```bash +curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{ + "alias": "qwen-72b", + "provider": "vllm-1", + "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", + "capabilities": ["chat", "chat_stream"], + "pricing": {"input_tokens": 0.4, "output_tokens": 1.2} + }' +``` + +```json +{ + "alias": "qwen-72b", + "provider": "vllm-1", + "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", + "capabilities": ["chat", "chat_stream"], + "origin": "declared", + "created_at": "2026-07-29T09:16:12.007843Z", + "pricing": {"input_tokens": 0.4, "output_tokens": 1.2}, + "pricing_inherited": false +} +``` + +The binding serves callers as soon as it exists. + +| Field | Required | Meaning | +|---|---|---| +| `alias` | no | Matches `^[A-Za-z0-9][A-Za-z0-9._:/-]*$`, max 200 chars. Omit it and the model serves under `upstream_name`, which is what you want when exposing a provider's catalogue as-is | +| `provider` | yes | Registered provider name (unless `target` is given) | +| `upstream_name` | yes | Model name as the upstream knows it (unless `target` is given) | +| `capabilities` | no | Defaults to `["chat", "chat_stream"]` | +| `pricing` | no | Price per million units. See [pricing.md](pricing.md) | +| `target` | no | Another model to point at instead of a provider. See below | + +Binding an already-used name gets 409 `alias_exists`. + +### Capabilities + +Capabilities are declared, never probed: state what the deployment actually +does. A request to an endpoint outside the set fails at the proxy with 400 +`endpoint_not_supported` naming the supported set, instead of a confusing +upstream 404. + +| Capability | Required for | +|---|---| +| `chat` | `POST /v1/chat/completions` | +| `chat_stream` | The same with `"stream": true` | +| `completions` | `POST /v1/completions` | +| `embeddings` | `POST /v1/embeddings` | +| `transcription` | Reserved. The capability and the `audio_seconds` unit exist; `/v1/audio/transcriptions` is not served yet | +| `vision` | Nothing. Declarative only: it tells callers the model reads image parts | + +An embeddings model is `"capabilities": ["embeddings"]`. + +## Point a name at another model + +Give `target` instead of `provider` and `upstream_name`: + +```bash +curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' \ + -d '{"alias": "acme/smart", "target": "z-ai/glm-5.2"}' +``` + +```json +{ + "alias": "acme/smart", + "target": "z-ai/glm-5.2", + "provider": "tensorx", + "upstream_name": "z-ai/glm-5.2", + "capabilities": ["chat", "chat_stream"], + "pricing": {"input_tokens": 0.6, "output_tokens": 2.2}, + "pricing_inherited": true +} +``` + +The name inherits its target's provider, upstream model, capabilities and +prices. Reads always report the resolved values, and `target` says where they +came from (`null` for a model that routes to a provider itself). Repoint or +reprice the target and every name pointing at it follows. + +Three rules, enforced at write time: + +- **One hop.** A target must route to a provider itself. Pointing at another + name is 400 `invalid_target`, and the error names the right one to use. A + model cannot point at itself. +- **A name owns its name and its price, nothing else.** Sending + `capabilities`, `upstream_name` or `provider` for one is 400 + `invalid_target` rather than a silent no-op. Setting `pricing` overrides + what it would inherit, per unit. +- **A target cannot be deleted out from under its names:** 409 + `model_in_use`, listing them. Deleting the provider takes both with it. + +Usage is recorded against the name the caller passed, so two teams pointing at +one model stay distinguishable in the summaries. + +## Edit a model + +`PATCH /admin/v1/models/{alias}` takes any of `alias`, `provider`, +`capabilities`, `upstream_name`, `target`, `pricing` and `hidden`. Everything +is editable in place; nothing here needs deleting and recreating. + +```bash +# Retarget upstream and add a capability +curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' \ + -d '{"upstream_name": "Qwen/Qwen3-72B", "capabilities": ["chat", "chat_stream", "completions"]}' + +# Rename it, or move it to another provider +curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' \ + -d '{"alias": "acme/smart", "provider": "vllm-2"}' +``` + +`capabilities` and `pricing` each replace the whole set when present: a unit +left out of `pricing` has no price afterwards. + +A rename keeps the row, so the model stays one thing across the change. Its +prices follow the new name, names pointing at it keep pointing at it, and the +old name stops resolving at once (callers using it get `model_not_found`). +Renaming onto a name already bound is 409 `alias_exists`. Usage already +recorded keeps the name it was served under. + +To turn a name back into a binding of its own, send an empty `target` with a +provider and upstream name in the same call: + +```bash +curl -s -X PATCH $P/admin/v1/models/acme%2Fsmart -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' \ + -d '{"target": "", "provider": "vllm-1", "upstream_name": "Qwen/Qwen3-72B"}' +``` + +## Hide a model + +`hidden` takes a model off `GET /v1/models`, and so out of the Playground +picker, without changing anything else: + +```bash +curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{"hidden": true}' +``` + +This is decluttering, not access control. A hidden model still serves requests +under its name, and any key or session holder can list hidden models with +`GET /v1/models?include_hidden=1`, which is what the Usage tab's Models subtab +does behind its "Show hidden" toggle. + +The flag belongs to the row it is set on, so a name can be hidden while its +target stays listed. To stop a model serving, delete it or disable its +provider. + +## Check what a name resolves to + +`GET /admin/v1/resolve` answers "what would this request do" without calling +anything, using the same resolution logic and cache as the data plane: + +```bash +curl -s "$P/admin/v1/resolve?model=qwen-72b&endpoint=chat&stream=true" \ + -H "authorization: Bearer $ADMIN" +``` + +```json +{ + "alias": "qwen-72b", + "provider": "vllm-1", + "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", + "url": "http://10.0.0.5:8000/v1/chat/completions", + "capabilities": ["chat", "chat_stream", "completions"] +} +``` + +`endpoint` defaults to `chat`; `stream=true` additionally requires +`chat_stream`. `url` reflects any per-endpoint override. The failure modes are +the data-plane ones: 404 `model_not_found` (unknown name or disabled provider) +and 400 `endpoint_not_supported`. + +## Delete a model + +```bash +curl -s -X DELETE $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" +# {"deleted": "qwen-72b"} +``` + +Usage history survives deletion. + +## Where next + +- [pricing.md](pricing.md) to attach prices. +- [usage.md](usage.md) to see what the models cost. +- [providers.md](providers.md) for the upstream side. diff --git a/docs/guides/pricing.md b/docs/guides/pricing.md new file mode 100644 index 0000000..7bb2ce7 --- /dev/null +++ b/docs/guides/pricing.md @@ -0,0 +1,114 @@ +# Set prices + +Prices belong to the model, one price per unit, expressed per million units. +Without a price, usage is still recorded, but as unpriced rather than free. + +Examples assume: + +```bash +P=http://127.0.0.1:4000 +ADMIN=lp_... # an admin key +``` + +## Price one model + +```bash +curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' \ + -d '{"pricing": {"input_tokens": 0.4, "output_tokens": 1.2}}' +``` + +`pricing` is the model's complete price set: units left out are unpriced +afterwards, so the same call clears a price by omitting its unit. + +Every model read returns the prices in force: + +```json +{ + "alias": "qwen-72b", + "upstream_name": "Qwen/Qwen2.5-72B-Instruct", + "pricing": {"input_tokens": 0.4, "output_tokens": 1.2}, + "pricing_inherited": false +} +``` + +## The units + +| Unit | Metered from | +|---|---| +| `input_tokens` | Prompt tokens | +| `output_tokens` | Completion tokens | +| `cached_input_tokens` | Cache reads: OpenAI's `prompt_tokens_details.cached_tokens`, Anthropic's `cache_read_input_tokens` | +| `cache_creation_tokens` | Anthropic cache writes, through the [relay](claude-code.md) | +| `audio_seconds` | Reserved for transcription | + +Each unit prices at its own rate, which is the point of keeping cache reads +and writes separate from input. + +## How a price is found + +Prices resolve against the names a model is known by, most specific first: + +1. Its own name. +2. The model it points at, if it is [a name for another + model](models.md#point-a-name-at-another-model). +3. The upstream model name. + +So a name inherits a price without repeating it, and overrides it per unit by +setting its own. `pricing_inherited` is true when the price in force is not +keyed on this name. + +## Load a whole price list + +Prices are a versioned feed keyed on `(model, unit)`, so a full list loads in +one call, or at startup from `LLMPROXY_PRICING_FILE`: + +```bash +curl -s $P/admin/v1/pricing -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{ + "version": "2026-07", + "entries": [ + {"model": "qwen-72b", "unit": "input_tokens", "price_per_million": 0.4}, + {"model": "qwen-72b", "unit": "output_tokens", "price_per_million": 1.2}, + {"model": "e5-embed", "unit": "input_tokens", "price_per_unit": 0.00000002} + ] + }' +# {"version": "2026-07", "count": 3} +``` + +The format: + +| Field | Rule | +|---|---| +| `version` | Required string | +| `entries[].model` | The public name or the upstream model name; lookup tries the public name first | +| `entries[].unit` | One of the units above | +| `entries[].price_per_million` or `price_per_unit` | Exactly one of the two. Per-million is divided by 1,000,000 on load | + +Anything else is rejected with 400 `invalid_pricing_feed`. + +Loading a feed replaces the active one atomically and applies to new requests +immediately. Already-recorded events are not repriced. Editing one model's +prices publishes a new version too, carrying every other model's entries over +unchanged. + +Read the active feed with `GET /admin/v1/pricing`; it returns the version +(`null` if none), the entry count and the entries as per-million prices. + +`LLMPROXY_PRICING_FILE` loads a feed at startup, and stores it only when its +`version` differs from the active one. + +## Unpriced is not free + +A `(model, unit)` pair with no entry is recorded as unpriced, never as zero: + +- The quantity row gets `priced: false` and the event is flagged `unpriced`. +- Summaries show `cost: null` for fully unpriced groups. +- `/metrics` counts the volume under `llmproxy_unpriced_units_total`. + +Alert on growth in that metric to catch pricing gaps. + +## Where next + +- [usage.md](usage.md) to read the resulting costs. +- [models.md](models.md) for the model each price belongs to. diff --git a/docs/guides/providers.md b/docs/guides/providers.md index 793ec64..88d0d56 100644 --- a/docs/guides/providers.md +++ b/docs/guides/providers.md @@ -1,44 +1,16 @@ -# Providers and models +# Manage providers -Provider registration, upstream model discovery, model binding and editing, -pricing and deletion are all available in the built-in UI at `/` for admins -signed in via SSO or the admin password; the UI drives exactly the API below. -This document covers that admin JSON API directly. Examples assume: +A provider is an upstream HTTP endpoint speaking a wire format. Everything on +this page is also available in the built-in UI under Admin, Providers; the UI +drives exactly the API shown here. + +Examples assume: ```bash P=http://127.0.0.1:4000 ADMIN=lp_... # an admin key ``` -## The model - -A **provider** is an upstream HTTP endpoint speaking a wire format. Only the -`openai` wire format exists today. `base_url` is the full prefix including -`/v1` (for example `http://10.0.0.5:8000/v1`); the proxy appends -`/chat/completions`, `/completions`, `/embeddings`, `/audio/transcriptions` or -`/models` to it. The upstream credential, if any, is stored AES-256-GCM -encrypted and is never returned by any endpoint; responses only carry -`has_credential: true`. - -A **model binding** maps a caller-facing **alias** to exactly one -(provider, upstream model name) pair. Aliases are globally unique across all -providers, enforced by a unique constraint, so resolution can never be -ambiguous and there is no fallback to bare upstream names. Each binding -carries: - -- a **capability set**, admin-declared, a non-empty subset of `chat`, - `chat_stream`, `completions`, `embeddings`, `transcription`, `vision`. - Capabilities are not probed or inferred; you declare what the deployment - actually does. A chat request needs `chat` (plus `chat_stream` if - `stream: true`); a request to an endpoint outside the set fails at the proxy - with 400 `endpoint_not_supported` naming the supported set, instead of a - confusing upstream 404. `transcription` is reserved: the capability and the - `audio_seconds` billing unit exist, but `/v1/audio/transcriptions` is not - served yet. `vision` is declarative: it tells callers the model reads image - parts and gates nothing, since bodies pass through untouched. -- an **origin**, `declared` or `discovered`, recording where the binding came - from. Informational only. - ## Register a provider ```bash @@ -67,355 +39,121 @@ curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ } ``` -`name` must match `^[a-z0-9][a-z0-9._-]*$`. `api_key` is optional for -unauthenticated upstreams and is sent upstream as `Authorization: Bearer`. -Timeouts are seconds; `timeout_read` doubles as the deadline for unary -requests (streams are unbounded by design and end when the upstream or caller -does). `max_concurrency` caps connections per upstream host; omitted means -unlimited, and it only bounds request concurrency on HTTP/1.1 upstreams -(HTTP/2 multiplexes many requests per connection). A duplicate name gets 409 -`provider_exists`. - -`PATCH /admin/v1/providers/{name}` edits a provider in place: `enabled`, -`base_url`, `api_key`/`remove_credential`, `verify_tls`, `timeout_connect`, -`timeout_read` and `max_concurrency` (zero clears the cap). The same -settings are editable from the Providers admin page via the edit button. - -### TLS options +A duplicate name gets 409 `provider_exists`. -For upstreams with internal certificates: - -```bash -curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{ - "name": "internal-node", - "base_url": "https://gpu-1.internal:8443/v1", - "ca_pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n" - }' -``` +| Field | Required | Meaning | +|---|---|---| +| `name` | yes | Matches `^[a-z0-9][a-z0-9._-]*$` | +| `wire_format` | no | `openai` (the default and, today, the only value) | +| `base_url` | yes | Full prefix including `/v1`. The proxy appends `/chat/completions`, `/completions`, `/embeddings`, `/audio/transcriptions` or `/models` | +| `api_key` | no | Sent upstream as `Authorization: Bearer`. Omit for unauthenticated upstreams | +| `verify_tls` | no | Default `true` | +| `ca_pem` | no | PEM bundle replacing the system trust pool for this provider | +| `timeout_connect` | no | Seconds, default 10 | +| `timeout_read` | no | Seconds, default 300. Also the deadline for unary requests; streams are unbounded and end when the upstream or the caller does | +| `max_concurrency` | no | Connections per upstream host, unlimited when omitted. Only bounds request concurrency on HTTP/1.1 upstreams, since HTTP/2 multiplexes | +| `endpoints` | no | Per-endpoint URL overrides; see below | -`ca_pem` replaces the system trust pool with the given PEM bundle for this -provider. `"verify_tls": false` disables verification entirely; prefer -`ca_pem`. Responses report `has_custom_ca`, never the PEM itself. +The upstream credential is stored AES-256-GCM encrypted and is never returned +by any endpoint. Responses carry `has_credential` instead. -### Per-endpoint URL overrides +## Edit a provider -When one logical provider serves different endpoint families from different -hosts, override individual endpoints with absolute URLs at registration time: +`PATCH /admin/v1/providers/{name}` takes `enabled`, `base_url`, `api_key`, +`remove_credential`, `verify_tls`, `timeout_connect`, `timeout_read` and +`max_concurrency` (zero clears the cap). ```bash -curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{ - "name": "mixed", - "base_url": "http://chat-host:8000/v1", - "endpoints": { - "embeddings": "http://embed-host:8001/v1/embeddings" - } - }' +curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{"base_url": "http://10.0.0.6:8000/v1"}' ``` -Valid keys are `chat`, `completions`, `embeddings`, `transcription`; an -unknown key gets 400 `invalid_endpoint`, a non-http(s) value 400 -`invalid_override`. The override is the exact URL called, path included. -Overrides are set at creation and returned by `GET -/admin/v1/providers/{name}`; they are not editable via PATCH (recreate the -provider to change them). +`endpoints` is not editable; recreate the provider to change an override. -## Discover upstream models - -Discovery asks the provider's `/models` endpoint what it serves and annotates -each name with any existing binding. It is read-only and never exposes or -creates anything. +## Rotate the upstream credential ```bash -curl -s $P/admin/v1/providers/vllm-1/discover -H "authorization: Bearer $ADMIN" -``` +# Rotate +curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{"api_key": "new-upstream-secret"}' -```json -{ - "provider": "vllm-1", - "discovered_at": "2026-07-29T09:15:40.582211Z", - "models": [ - {"upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", "bound_alias": null}, - {"upstream_name": "intfloat/e5-mistral-7b-instruct", "bound_alias": "e5-embed"} - ] -} +# Remove (the upstream no longer requires auth) +curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{"remove_credential": true}' ``` -Failures: 502 `provider_unreachable` if the upstream cannot be reached, 502 -`discovery_failed` if it answers with an error status or invalid JSON. +Sending `"api_key": ""` is the same as `remove_credential: true`. Rotation +takes effect immediately: the catalog cache and the upstream connection pool +are flushed on every catalog mutation. -## Bind a model +## Use an internal certificate -A binding serves callers as soon as it exists; taking it out of service means -deleting it, or disabling its provider. +For upstreams with a private CA, pass the bundle at registration: ```bash -curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ +curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ -H 'content-type: application/json' -d '{ - "alias": "qwen-72b", - "provider": "vllm-1", - "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", - "capabilities": ["chat", "chat_stream"], - "pricing": {"input_tokens": 0.4, "output_tokens": 1.2} + "name": "internal-node", + "base_url": "https://gpu-1.internal:8443/v1", + "ca_pem": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n" }' ``` -```json -{ - "alias": "qwen-72b", - "provider": "vllm-1", - "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", - "capabilities": ["chat", "chat_stream"], - "origin": "declared", - "created_at": "2026-07-29T09:16:12.007843Z", - "pricing": {"input_tokens": 0.4, "output_tokens": 1.2}, - "pricing_inherited": false -} -``` - -`alias` must match `^[A-Za-z0-9][A-Za-z0-9._:/-]*$` (max 200 chars). Leave it -out and the model serves under `upstream_name`, which is what you want when -you are just exposing a provider's catalogue as-is; give one when callers -should see a name of your choosing. `capabilities` defaults to -`["chat", "chat_stream"]`. Binding an already-used alias gets 409 -`alias_exists`. An embeddings model would be `"capabilities": ["embeddings"]`. - -`pricing` is optional and holds the price per million units for each unit the -model meters (`input_tokens`, `output_tokens`, `cached_input_tokens`, -`audio_seconds`). A unit with no price records usage as unpriced, never as -free; see [keys-and-usage.md](keys-and-usage.md) for what that means downstream. - -### Point a name at another model - -A model can also be a name for a model already bound. Give `target` instead of -`provider` and `upstream_name`: - -```bash -curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"alias": "acme/smart", "target": "z-ai/glm-5.2"}' -``` - -```json -{ - "alias": "acme/smart", - "target": "z-ai/glm-5.2", - "provider": "tensorx", - "upstream_name": "z-ai/glm-5.2", - "capabilities": ["chat", "chat_stream"], - "pricing": {"input_tokens": 0.6, "output_tokens": 2.2}, - "pricing_inherited": true -} -``` - -The alias inherits its target's provider, upstream model, capabilities and -prices; `provider`, `upstream_name` and `capabilities` in any read are always -the resolved ones, and `target` says where they came from (`null` for a model -that routes to a provider itself). Repoint or reprice the target and every -name pointing at it follows, which is the point: `acme/smart` is a -promise to your callers, and what fulfils it is yours to change. - -Rules, all enforced at write time: - -- **One hop.** A target must route to a provider itself. Pointing at another - alias is 400 `invalid_target`, and names it so you can point at the right - one; a model cannot point at itself. -- **What an alias owns is its name and its price.** Sending `capabilities`, - `upstream_name` or `provider` for one is 400 `invalid_target` rather than a - silent no-op. Setting `pricing` on it overrides what it would inherit, per - unit, which is how an internal name charges differently from the model - behind it. -- **A target cannot be deleted out from under its aliases**: 409 - `model_in_use`, listing them. Deleting the provider takes both with it. +`ca_pem` replaces the system trust pool for this provider only. Responses +report `has_custom_ca`, never the PEM itself. -Usage is recorded against the name the caller passed, so two teams pointing at -one model stay distinguishable in the summaries. +`"verify_tls": false` disables verification entirely. Prefer `ca_pem`. -### Edit a binding +## Override individual endpoints -`PATCH /admin/v1/models/{alias}` takes any of `alias`, `provider`, -`capabilities`, `upstream_name`, `target`, `pricing`, `hidden`. Everything -about a binding is editable in place: nothing here needs deleting and -recreating. - -```bash -# Retarget the alias and add the completions capability -curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"upstream_name": "Qwen/Qwen3-72B", "capabilities": ["chat", "chat_stream", "completions"]}' - -# Reprice it -curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"pricing": {"input_tokens": 0.5, "output_tokens": 1.5}}' - -# Rename it, or move it to another provider -curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"alias": "acme/smart", "provider": "vllm-2"}' -``` - -`capabilities` and `pricing`, when present, each replace the whole set: a unit -left out of `pricing` has no price afterwards. The response is the updated -binding view. - -A rename keeps the row, so the model stays one thing across the change: its -prices follow the new name, names pointing at it keep pointing at it, and the -old name stops resolving at once (callers using it get `model_not_found`). -Renaming onto a name already bound is rejected with 409 `alias_exists`. Usage -already recorded keeps the name it was served under, so history stays truthful -about what callers asked for. - -`target` switches a model between the two kinds. Sending an empty `target` -turns an alias back into a binding of its own, which needs `provider` and -`upstream_name` in the same call: +When one logical provider serves different endpoint families from different +hosts, give absolute URLs at registration time: ```bash -curl -s -X PATCH $P/admin/v1/models/acme%2Fsmart -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' \ - -d '{"target": "", "provider": "vllm-1", "upstream_name": "Qwen/Qwen3-72B"}' +curl -s $P/admin/v1/providers -H "authorization: Bearer $ADMIN" \ + -H 'content-type: application/json' -d '{ + "name": "mixed", + "base_url": "http://chat-host:8000/v1", + "endpoints": { + "embeddings": "http://embed-host:8001/v1/embeddings" + } + }' ``` -### Hiding a model - -`hidden` takes a model off `GET /v1/models`, and so out of the Playground -picker, without changing anything else: +Valid keys are `chat`, `completions`, `embeddings` and `transcription`. The +override is the exact URL called, path included. An unknown key gets 400 +`invalid_endpoint`, a non-http(s) value 400 `invalid_override`. -```bash -curl -s -X PATCH $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{"hidden": true}' -``` +Overrides are returned by `GET /admin/v1/providers/{name}` and cannot be +changed with PATCH. -This is decluttering, not access control: a hidden model still serves requests -under its name, and anyone with a key or a session can list hidden models with -`GET /v1/models?include_hidden=1`, which is what the Usage tab's Models subtab -does behind its "Show hidden" toggle. The flag belongs to the row it is set on, -so hiding a model says nothing about the names pointing at it, and an alias can -be hidden while its target stays listed. To actually stop a model serving, -delete the binding or disable its provider. +## Take a provider out of service -Similarly for providers, `PATCH /admin/v1/providers/{name}` takes `enabled`, -`base_url`, `api_key` and `remove_credential`: +Disabling stops every one of its bindings resolving and deletes nothing: ```bash -# Drain a provider: all its bindings stop resolving, nothing is deleted curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ -H 'content-type: application/json' -d '{"enabled": false}' ``` -## Dry-run resolution - -`GET /admin/v1/resolve` answers "what would this request do" without calling -anything, using the same resolution logic and cache as the data plane: - -```bash -curl -s "$P/admin/v1/resolve?model=qwen-72b&endpoint=chat&stream=true" \ - -H "authorization: Bearer $ADMIN" -``` - -```json -{ - "alias": "qwen-72b", - "provider": "vllm-1", - "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct", - "url": "http://10.0.0.5:8000/v1/chat/completions", - "capabilities": ["chat", "chat_stream", "completions"] -} -``` - -`endpoint` defaults to `chat`; `stream=true` additionally requires the -`chat_stream` capability. Failure modes are exactly the data-plane ones: 404 -`model_not_found` (unknown alias or disabled provider) -and 400 `endpoint_not_supported` (message lists the missing and supported -capabilities). `url` reflects any per-endpoint override. - -## Credential rotation and removal - -```bash -# Rotate the upstream key -curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{"api_key": "new-upstream-secret"}' - -# Remove it (upstream requires no auth anymore) -curl -s -X PATCH $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" \ - -H 'content-type: application/json' -d '{"remove_credential": true}' -``` - -Sending `"api_key": ""` is equivalent to `remove_credential: true`. Rotation -takes effect immediately (the catalog cache and the upstream connection pool -are flushed on every catalog mutation). +Callers asking for one of its models get 404 `model_not_found`. Re-enable with +`{"enabled": true}`. -## Deleting +## Delete a provider ```bash -curl -s -X DELETE $P/admin/v1/models/qwen-72b -H "authorization: Bearer $ADMIN" -# {"deleted": "qwen-72b"} - curl -s -X DELETE $P/admin/v1/providers/vllm-1 -H "authorization: Bearer $ADMIN" # {"deleted": "vllm-1"} ``` -Deleting a provider cascades in one transaction: all its model bindings and -endpoint overrides go with it. Usage history is untouched (usage events store -copied identifiers, not foreign keys into the catalog). If you want the -option to come back, disable instead of delete. - -## Audit trail +Deletion cascades in one transaction: every model binding and endpoint +override goes with it. Usage history is untouched, because usage events store +copied identifiers rather than foreign keys into the catalog. -Every admin mutation writes a metadata-only event in the same transaction as -the change: `provider.create|update|delete`, `model.create|update|delete`, -`principal.create`, `key.create`, `key.delete`, `pricing.load`. +Disable instead of deleting if you might want the provider back. -```bash -curl -s "$P/admin/v1/events?limit=3" -H "authorization: Bearer $ADMIN" -``` - -```json -{ - "events": [ - { - "ts": "2026-07-29T09:20:01.334019Z", - "actor_principal_id": "b1f4c2d8e96a4f0b8d3a5c7e9f012345", - "action": "model.update", - "target_kind": "model", - "target_ref": "qwen-72b" - }, - { - "ts": "2026-07-29T09:16:12.008120Z", - "actor_principal_id": "b1f4c2d8e96a4f0b8d3a5c7e9f012345", - "action": "model.create", - "target_kind": "model", - "target_ref": "qwen-72b" - }, - { - "ts": "2026-07-29T09:14:02.114901Z", - "actor_principal_id": "b1f4c2d8e96a4f0b8d3a5c7e9f012345", - "action": "provider.create", - "target_kind": "provider", - "target_ref": "vllm-1" - } - ], - "limit": 3, - "offset": 0 -} -``` - -Events are newest first and carry identifiers only, never payloads. Direct -CLI operations (`llmproxy key create` etc.) bypass the HTTP layer and are not -audited. - -## Pagination - -All admin list endpoints (`providers`, `models`, `keys`, `principals`, -`events`) accept `limit` (1 to 500, default 100) and `offset` (default 0) and -echo both back in the response: - -```bash -curl -s "$P/admin/v1/models?provider=vllm-1&limit=50&offset=50" \ - -H "authorization: Bearer $ADMIN" -``` +## Where next -Out-of-range values silently fall back to the defaults. Ordering is by name -or alias (keys by creation time, events newest first), so pagination is -stable between requests. +- [models.md](models.md) to bind the provider's models. +- [../reference/api.md](../reference/api.md) for the full endpoint list, + pagination and the audit trail. diff --git a/docs/guides/sso.md b/docs/guides/sso.md index 756de34..d14fec4 100644 --- a/docs/guides/sso.md +++ b/docs/guides/sso.md @@ -1,108 +1,75 @@ -# SSO (OIDC) +# Set up SSO Setting `LLMPROXY_OIDC_ISSUER` switches the proxy from local single-admin mode -to SSO mode. `LLMPROXY_OIDC_CLIENT_ID`, `LLMPROXY_OIDC_CLIENT_SECRET` and -`LLMPROXY_OIDC_REDIRECT_URL` are then required; the server refuses to boot -without them. It works with any OIDC provider that exposes standard discovery -(`/.well-known/openid-configuration`) and a userinfo endpoint; Authentik is +to SSO mode. It works with any OIDC provider that exposes standard discovery +(`/.well-known/openid-configuration`) and a userinfo endpoint. Authentik is walked through below. -## How it works here - -The proxy is a minimal OIDC relying party using the authorization code flow -plus a userinfo lookup. `GET /auth/login` sets a short-lived signed state -cookie and redirects to the IdP. The callback (`GET /auth/callback`) verifies -the state, exchanges the code for an access token (confidential client: -`client_id` and `client_secret` in the token request), and then calls the -IdP's userinfo endpoint server-to-server with that token. Identity comes -entirely from userinfo; there is no local JWT validation, the trust chain is -TLS to the IdP plus client authentication. - -Principals are keyed on the IdP `sub` claim, which is the only stable join -key. On first login a principal is created with a name derived from -`preferred_username` (falling back to `nickname`, `name`, then email), -sanitized to the local naming rules and suffixed on collision. On every -subsequent login the email and the group-derived role are reconciled, so -group changes in the IdP take effect at the next sign-in; the principal name, -its keys and its usage history stay attached to the same `sub`. - -Group handling is configuration, not convention: - -- `LLMPROXY_OIDC_GROUPS_CLAIM` (default `groups`) names the userinfo claim - read as a string array. -- `LLMPROXY_OIDC_ADMIN_GROUP`: members get the `admin` role, everyone else is - a `member`. Unset means no SSO user is ever auto-promoted. -- `LLMPROXY_OIDC_REQUIRED_GROUP`: when set, users outside this group are - rejected at the callback with 403 `sso_group_required` and no principal is - created. - -### Sessions - -A successful login issues a session cookie (`llmproxy_session`). Sessions are -server-side, the standard model for first-party browser sessions: the cookie -holds an opaque random token, and the server stores only its keyed hash in -the `session` table (like API keys, a stolen database cannot forge one). -Lifetime is `LLMPROXY_SESSION_TTL` (default 12h). The cookie is `HttpOnly`, -marked `Secure` when the redirect URL is `https://` or the request itself -arrived over TLS (directly or via `X-Forwarded-Proto: https` from a -TLS-terminating reverse proxy), and `SameSite=Lax` rather than `Strict` -because it must survive the top-level redirect back from the IdP. To -compensate, every non-GET request authenticated by session is origin-checked: -an `Origin` header matching neither the request `Host` nor the host of -`LLMPROXY_OIDC_REDIRECT_URL` (the browser-facing name, for deployments where -the reverse proxy rewrites `Host`) is rejected with 403 -`cross_origin_rejected`. - -Two properties worth internalizing: - -- Sessions carry the principal's role. An admin's session can call the - `/admin/v1` API, which is how the built-in UI manages providers and models; - every session mutation is origin-checked. Automation and scripts use API - keys, which need no Origin header. -- Invalidation is deletion, immediately effective. `GET /auth/logout` deletes - the caller's session row, so even a stolen copy of the cookie dies with it. - `POST /admin/v1/principals/{id}/revoke-sessions` deletes every session of a - principal, while API keys are untouched. +SSO mode implies a TLS reverse proxy in front, because the redirect URL must +be externally reachable over HTTPS. + +## Configuration + +| Variable | Required | Meaning | +|---|---|---| +| `LLMPROXY_OIDC_ISSUER` | yes | Issuer URL. Setting it enables SSO mode | +| `LLMPROXY_OIDC_CLIENT_ID` | yes | OAuth client id | +| `LLMPROXY_OIDC_CLIENT_SECRET` | yes | OAuth client secret | +| `LLMPROXY_OIDC_REDIRECT_URL` | yes | Absolute callback URL, `https://your-proxy/auth/callback` | +| `LLMPROXY_OIDC_SCOPES` | no | Default `openid profile email` | +| `LLMPROXY_OIDC_GROUPS_CLAIM` | no | Userinfo claim read as a string array, default `groups` | +| `LLMPROXY_OIDC_ADMIN_GROUP` | no | Members get the `admin` role. Unset means no SSO user is ever auto-promoted | +| `LLMPROXY_OIDC_REQUIRED_GROUP` | no | When set, users outside it are rejected at the callback with 403 `sso_group_required`, and no principal is created | +| `LLMPROXY_SESSION_TTL` | no | Browser session lifetime, default `12h` | + +The server refuses to boot if the first four are not all set. It performs OIDC +discovery at boot and fails fast with a clear error if the issuer is wrong or +unreachable. -Signed stateless session cookies from releases before the session table are -still accepted until they expire (at most one `LLMPROXY_SESSION_TTL` after -the upgrade), so deploying the upgrade logs nobody out; revoke-sessions -covers them via a per-principal cutoff timestamp during that window. +The loopback guard does not apply in SSO mode, so `LLMPROXY_ALLOW_NONLOCAL` is +unnecessary. ## Authentik walkthrough -Assume the proxy is reachable at `https://llmproxy.example.com` (SSO mode -implies a TLS reverse proxy in front) and Authentik at -`https://auth.example.com`. +Assume the proxy is reachable at `https://llmproxy.example.com` and Authentik +at `https://auth.example.com`. ### 1. Create the provider -In the Authentik admin UI: Applications, Providers, Create, type -"OAuth2/OpenID Provider". +In the Authentik admin UI: Applications, Providers, Create, type "OAuth2/OpenID +Provider". -- Client type: **Confidential** (the proxy authenticates with a client - secret at the token endpoint). +- Client type: **Confidential**. - Redirect URI: `https://llmproxy.example.com/auth/callback` (exact match). - Scopes: the defaults (`openid`, `profile`, `email`) are enough. Authentik - includes the `groups` claim with the `profile` scope, so the proxy's default - `LLMPROXY_OIDC_SCOPES="openid profile email"` and - `LLMPROXY_OIDC_GROUPS_CLAIM=groups` work without a custom scope mapping. + includes the `groups` claim with the `profile` scope, so the proxy's + defaults work without a custom scope mapping. Note the generated client ID and client secret. ### 2. Create the application Applications, Create: name it (for example "LLM Proxy"), set the slug (for -example `llmproxy`), and select the provider from step 1. The slug determines -the issuer URL: `https://auth.example.com/application/o/llmproxy/`. You can -confirm it by fetching -`https://auth.example.com/application/o/llmproxy/.well-known/openid-configuration`. +example `llmproxy`), and select the provider from step 1. + +The slug determines the issuer URL: +`https://auth.example.com/application/o/llmproxy/`. Confirm it: + +```bash +curl -s https://auth.example.com/application/o/llmproxy/.well-known/openid-configuration | head -3 +``` -If you plan to use a required group, also bind that group to the application -in Authentik so users outside it do not even see it in their library. The -proxy enforces the gate itself either way. +### 3. Create the groups -### 3. Configure the proxy +Create `llm-admins` and `llm-users` in Authentik and add users. An admin who +is also a user belongs in both: `llm-admins` grants the role, `llm-users` +passes the gate. + +If you use a required group, bind it to the application in Authentik as well, +so users outside it do not see the application in their library. The proxy +enforces the gate either way. + +### 4. Configure the proxy ```bash LLMPROXY_OIDC_ISSUER=https://auth.example.com/application/o/llmproxy/ @@ -115,49 +82,81 @@ LLMPROXY_OIDC_ADMIN_GROUP=llm-admins # members become proxy admins LLMPROXY_OIDC_REQUIRED_GROUP=llm-users # everyone else is rejected ``` -Restart the proxy. It performs OIDC discovery at boot and fails fast with a -clear error if the issuer is wrong or unreachable. The loopback guard does not -apply in SSO mode, so no `LLMPROXY_ALLOW_NONLOCAL` is needed. - -Create the Authentik groups (`llm-admins`, `llm-users`) and add users. An -admin who is also a user belongs in both groups: `llm-admins` grants the role, -`llm-users` passes the gate. - -### 4. End-user flow - -A user visits `https://llmproxy.example.com/`, clicks "Sign in with SSO", and -lands back on the built-in UI after authenticating with Authentik. There they -mint an API key (shown exactly once, listed as `***xxxx` afterwards), delete -keys, and see their own usage and cost per model and endpoint. The key goes into whatever OpenAI SDK they use, pointed at -`https://llmproxy.example.com/v1`. - -An `llm-admins` member additionally sees usage by user, the request metadata -log, and the Admin tab of the UI: providers, models, users (sign someone out -everywhere), services (one key per service, created in one step) and all -keys. For scripted administration a key inherits the principal's role, so a -UI-minted key of an `llm-admins` member is an admin key. - -### Break-glass admin password - -Password login for the local admin principal stays available in SSO mode -unless `LLMPROXY_ADMIN_PASSWORD_DISABLED=1` is set, so an IdP outage cannot -lock you out of the proxy. See -[configuration.md](configuration.md#the-admin-password). - -### Role changes and offboarding - -Roles live on the principal and are reconciled only at login. Removing a user -from `llm-users` blocks their next login, and removing them from `llm-admins` -demotes them at their next login, but neither touches their live sessions or -existing API keys. To make an IdP change effective now instead of at the next -sign-in, force that sign-in: -`POST /admin/v1/principals/{id}/revoke-sessions` invalidates the user's -browser sessions, and their next UI access goes through a fresh login that -re-reconciles group membership. - -API keys keep working without any login, and they carry the principal's role -as last reconciled, so a removed admin's keys stay admin-capable until that -user signs in again. For offboarding, delete the keys explicitly: list them -with `GET /admin/v1/keys?principal=` and delete with -`DELETE /admin/v1/keys/{id}`. The full offboarding sequence is: remove the -user at the IdP, revoke their sessions, delete their keys. +Restart the proxy. + +### 5. Check it + +Visit `https://llmproxy.example.com/` and click "Sign in with SSO". You should +land back on the built-in UI, signed in. + +A member sees their own keys and usage. An `llm-admins` member additionally +sees usage by user, the request log, and the Admin tab: providers, models, +users, services and all keys. + +## What users do next + +From the UI, a user mints an API key (shown once, `***xxxx` afterwards) and +points any OpenAI SDK at `https://llmproxy.example.com/v1`. A key inherits the +principal's role, so a UI-minted key of an `llm-admins` member is an admin +key. + +## Keep the break-glass password + +Password login for the local admin stays available in SSO mode unless +`LLMPROXY_ADMIN_PASSWORD_DISABLED=1` is set, so an IdP outage cannot lock you +out of your own proxy. See +[the admin password](../reference/configuration.md#the-admin-password). + +## How identity is handled + +Four facts that matter operationally: + +- **Principals are keyed on the IdP `sub` claim.** On first login a principal + is created with a name derived from `preferred_username` (falling back to + `nickname`, `name`, then email), sanitised and suffixed on collision. The + name, its keys and its usage history stay attached to that `sub`. +- **Identity comes from userinfo**, fetched server-to-server after the code + exchange. There is no local JWT validation. +- **Roles are reconciled at login only.** A group change in the IdP takes + effect at the user's next sign-in. +- **Sessions are server-side rows and die on deletion.** The cookie holds an + opaque token; the server stores only its keyed hash. `GET /auth/logout` + deletes the caller's row, so even a stolen copy of the cookie stops working. + `POST /admin/v1/principals/{id}/revoke-sessions` deletes every session of a + principal and leaves API keys untouched. + +Upgrading does not log anyone out: stateless session cookies from releases +before the session table are accepted until they expire. + +## Offboard a user + +Removing a user at the IdP is not enough on its own. Roles are reconciled at +login, and API keys need no login at all, so a removed admin's keys stay +admin-capable until that user next signs in. + +The full sequence, with `P=https://llmproxy.example.com` and `ADMIN=lp_...`: + +1. Remove the user at the IdP. +2. Revoke their sessions, which forces a fresh login that re-reconciles group + membership: + + ```bash + curl -s -X POST $P/admin/v1/principals//revoke-sessions \ + -H "authorization: Bearer $ADMIN" + ``` + +3. Delete their keys: + + ```bash + curl -s "$P/admin/v1/keys?principal=alice" -H "authorization: Bearer $ADMIN" + curl -s -X DELETE $P/admin/v1/keys/ -H "authorization: Bearer $ADMIN" + ``` + +For a demotion rather than a departure, steps 1 and 2 are enough, as long as +the user's existing keys should keep working as a member. + +## Where next + +- [api-keys.md](api-keys.md) for the key lifecycle. +- [../reference/configuration.md](../reference/configuration.md) for every + variable. diff --git a/docs/guides/usage.md b/docs/guides/usage.md new file mode 100644 index 0000000..76137a2 --- /dev/null +++ b/docs/guides/usage.md @@ -0,0 +1,228 @@ +# Read usage and cost + +Every `/v1` request that reaches resolution, and every relayed Anthropic +request, records one usage event. The Usage and Requests tabs of the built-in +UI draw the endpoints below; use the API directly for exports and alerting. + +Examples assume: + +```bash +P=http://127.0.0.1:4000 +KEY=lp_... # any key +ADMIN=lp_... # an admin key +``` + +## What gets recorded + +Identifiers, flags and numbers. **No prompt, completion or embedding input is +ever persisted**, and there is no flag to turn that on; see +[../concepts/architecture.md](../concepts/architecture.md#the-schema-is-content-free-by-construction). + +Per event: principal, key, provider, model, upstream name, endpoint, client, +status code, outcome, error kind, cancelled and streamed flags, duration and +cost. Attached to it, one row per unit: `(unit, quantity, unit_price, +priced)`. + +| Field | Values | +|---|---| +| `outcome` | `ok`, `upstream_error`, `unreachable`, `cancelled` | +| `error_kind` | A short token classifying a failure: the transport class on `unreachable` (`timeout`, `connection_error`, `response_too_large`), the upstream's `error.type`/`error.code` on `upstream_error` (`rate_limit_error`, `overloaded_error`, ...). Provider error *messages* are never stored, because they can echo request content | +| `client` | The caller's `User-Agent`, truncated to 256 characters and stored verbatim (`claude-cli/2.0.13 (external, cli)`, `openai-python/1.51.0`). Grouping by product family happens at read time | + +Cancelled and errored requests are recorded too, with whatever partial usage +the upstream reported. Accounting runs detached from the request path, so a +client hanging up cannot lose the record. + +Relay traffic lands in the same tables, attributed by relay token instead of +API key, under the sentinel provider `transparent:anthropic`. + +**One number is normalised at read time.** Stored quantities stay exactly as +the upstream reported them, but every aggregate reports `input_tokens` as +input processed at the full input rate, so the number means the same thing +across OpenAI-shaped and Anthropic-shaped upstreams. `cached_input_tokens` is +always the cheap cache-read count on top of it. The mechanics are in +[../concepts/architecture.md](../concepts/architecture.md#usage-numbers-are-normalised-at-read-time). + +## Tag requests by application + +A key says who is calling. A tag says which of their applications. Callers +send comma-separated `key:value` pairs: + +``` +x-llmproxy-tags: app:dataindex,context:search +``` + +The header works on both ingresses, including the +[relay](claude-code.md), where the proxy consumes and strips it before +forwarding. + +Values are normalised at capture so grouping is stable: each pair is trimmed +and lowercased, must match `[a-z0-9][a-z0-9._-]*:[a-z0-9][a-z0-9._-]*`, keys +are unique (first occurrence wins), pairs are sorted by key, and the list is +capped at 8 pairs and 256 bytes. Anything malformed is dropped silently: +telemetry never fails a request. + +Events with no `app` tag are grouped as `untagged`, so no traffic drops out of +the breakdowns. + +## Read your own usage + +```bash +curl -s "$P/my/usage?since=2026-07-01&until=2026-08-01" -H "authorization: Bearer $KEY" +``` + +```json +{ + "usage": [ + { + "model": "qwen-72b", + "endpoint": "chat", + "requests": 412, + "cancelled": 3, + "cost": 1.20734, + "units": {"input_tokens": 1882340, "output_tokens": 378512} + } + ] +} +``` + +Grouped by (model, endpoint). `since` and `until` take ISO 8601 dates or +datetimes; `until` is exclusive. + +`GET /admin/v1/usage/summary` is the same aggregation across everyone, each +row carrying a `principal`, optionally filtered with `?principal=`. + +## Read usage over time + +```bash +curl -s "$P/my/usage/series?bucket=day&since=2026-07-01" -H "authorization: Bearer $KEY" +``` + +```json +{ + "bucket": "day", + "series": [ + { + "start": "2026-07-01T00:00:00Z", + "requests": 41, + "ok": 38, + "cancelled": 1, + "failed": 2, + "unpriced_requests": 3, + "cost": 0.10734, + "units": {"input_tokens": 188234, "output_tokens": 37851} + }, + { + "start": "2026-07-02T00:00:00Z", + "requests": 0, + "ok": 0, + "cancelled": 0, + "failed": 0, + "unpriced_requests": 0, + "cost": null, + "units": {} + } + ] +} +``` + +`bucket` is `hour | day | week | month`, default `day`. Buckets are UTC, weeks +start Monday, and the range is gap-filled: quiet periods come back as empty +buckets rather than holes, so a series plots directly. + +`ok`, `cancelled` and `failed` partition `requests`, where failed is any +non-ok outcome the caller did not cancel. `cost` is null when nothing in the +bucket had a price, and `unpriced_requests` counts requests that hit a missing +price. + +Without `since`, the series starts at the first recorded event. A window +needing more than 1000 buckets is refused with 400 `range_too_large`. + +`GET /admin/v1/usage/series` is the same across everyone. + +## Query team-wide statistics + +The `/stats` endpoints serve the same accounting to **every authenticated +user**. Usage is team-visible by design; the admin role gates configuration, +not visibility. + +| Endpoint | Returns | +|---|---| +| `GET /stats/series` | The bucketed series above, with the filters below | +| `GET /stats/summary` | The window aggregated over every dimension: one row per (principal, provider, model, endpoint, client, tags), so any roll-up is a client-side sum | +| `GET /stats/requests` | One page of the request metadata log, newest first, failures included. `limit` (max 500) and `offset` page it; the response carries `{requests, limit, offset, total}` | +| `GET /stats/requests/facets` | The distinct principals, keys, providers, models, clients and tags in the window, for filter menus. Each list caps at 500 values | +| `GET /stats/errors` | The errors dashboard in one call: a gap-filled series of counts per outcome, plus a breakdown per (provider, model, endpoint, client, tags, outcome, error_kind, status_code) with request count, average duration, last-seen and time-to-outcome bands | + +All of them take the same filters: + +| Filter | Matches | +|---|---| +| `principal=` | Exact | +| `key=` | Exact. API keys only; relay events never match one, so reach them through `provider=transparent:anthropic` | +| `provider=` | Exact, including the sentinel `transparent:anthropic`. A deleted provider's events resolve to `(deleted)` | +| `model=` | Exact | +| `client=` | Prefix on the stored User-Agent, so `client=claude-cli` covers every version | +| `tag=:` | An exact pair, repeatable up to four times, all of which must match | +| `outcome=` | `ok`, `upstream_error`, `unreachable`, `cancelled`, or the meta value `failed` for everything not ok. An unknown value is 400 `invalid_outcome` | +| `since=`, `until=` | The window, as above | + +```bash +curl -s "$P/stats/summary?since=2026-07-01&client=claude-cli" -H "authorization: Bearer $KEY" +``` + +```json +{ + "usage": [ + { + "principal": "alice", + "provider": "transparent:anthropic", + "model": "claude-opus-5", + "endpoint": "v1/messages", + "client": "claude-cli/2.0.13 (external, cli)", + "tags": "app:dataindex,context:search", + "requests": 3211, + "cancelled": 12, + "cost": 41.20734, + "units": {"input_tokens": 311140, "output_tokens": 378512, "cached_input_tokens": 1571200} + } + ] +} +``` + +Two things to know when reading `/stats/summary`: + +- It counts **completed requests with a model only**. A failure's model is + whatever the caller asked for, and model-less events (the relay's + `count_tokens` and `models` calls) have no place in a by-model breakdown. + Both stay visible in the series and the request log. +- `/stats/errors` includes cells with outcome `ok`, so error rates per + dimension have their denominator. + +## Scrape metrics + +`GET /metrics` is unauthenticated and exposes Prometheus text with aggregate +labels only. Per-principal detail stays in the database so metric cardinality +stays bounded. + +| Metric | Labels | Meaning | +|---|---|---| +| `llmproxy_requests_total` | `endpoint`, `provider`, `model`, `outcome` | Requests by outcome | +| `llmproxy_request_seconds_sum` | `endpoint`, `provider`, `model` | Total request duration | +| `llmproxy_request_seconds_count` | `endpoint`, `provider`, `model` | Request count (divide sum by count for the mean) | +| `llmproxy_usage_units_total` | `provider`, `model`, `unit` | Consumed units | +| `llmproxy_unpriced_units_total` | `model`, `unit` | Units with no price entry; alert on growth to catch pricing gaps | + +`model` is always the caller-facing name. + +``` +llmproxy_requests_total{endpoint="chat",model="qwen-72b",outcome="ok",provider="vllm-1"} 412 +llmproxy_usage_units_total{model="qwen-72b",provider="vllm-1",unit="input_tokens"} 1.88234e+06 +llmproxy_unpriced_units_total{model="e5-embed",unit="input_tokens"} 4.4021833e+07 +``` + +## Where next + +- [pricing.md](pricing.md) if costs come back `null`. +- [claude-code.md](claude-code.md) for relay accounting. +- [../reference/api.md](../reference/api.md) for the full endpoint list. From 16e3d11486fa1e6ec1228e8fbb0bc8ab5ca9d6f5 Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:14:42 -0600 Subject: [PATCH 4/6] docs: rewrite the reference pages and add the goal router reference/configuration.md keeps the full variable table and trims the secret and admin-password sections to their operational consequences; the cryptography narration is gone. reference/api.md gains the pagination and audit-trail sections that used to live in providers-and-models.md. concepts/architecture.md gains a design rule for read-time usage normalisation, the load-bearing why cut from the usage guide. docs/README.md is now a goal router (Start here / Deploy / Operate / Look up / Understand) instead of a flat file list. --- docs/README.md | 106 ++++++++++++++----------- docs/concepts/architecture.md | 27 ++++++- docs/reference/api.md | 55 +++++++++++-- docs/reference/configuration.md | 134 ++++++++++++++++---------------- 4 files changed, 203 insertions(+), 119 deletions(-) diff --git a/docs/README.md b/docs/README.md index 71cd178..26e480c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,47 +1,65 @@ # llmproxy documentation -The documentation follows the [Diátaxis](https://diataxis.fr/) structure: -a tutorial to learn by doing, how-to guides for specific goals, reference -material for looking things up, and explanation for understanding the design. - -## Tutorial - -- [getting-started.md](getting-started.md): from `git clone` to a first chat - completion, including provider registration, model binding and your first - API key. - -## How-to guides - -- [installation.md](installation.md): release binaries, build from source, - state layout, first key, systemd, upgrades. -- [docker.md](docker.md): the image, docker compose, switching to Postgres, deploying on Coolify. -- [sso.md](sso.md): OIDC login, sessions, group-based roles, an Authentik - walkthrough. -- [providers-and-models.md](providers-and-models.md): register providers, - discover and bind models, aliases, capabilities, dry-run resolution. -- [keys-and-usage.md](keys-and-usage.md): key lifecycle, usage accounting, - pricing feeds, team statistics, metrics. -- [transparent-relay.md](transparent-relay.md): meter Claude Code and other - Anthropic-native tools without touching their credentials. - -## Reference - -- [configuration.md](configuration.md): every environment variable, database - URLs, the secret, the admin password, CLI flags. -- [api.md](api.md): the full HTTP surface, endpoint by endpoint, with error - codes and headers. -- [openapi.yaml](openapi.yaml): the same surface as an OpenAPI 3.1 spec. -- [performance.md](performance.md): throughput and latency measurements and - how to reproduce them. - -## Explanation - -- [architecture.md](architecture.md): scope and non-goals, the three endpoint - families, and the design rules with their rationale, including the - no-content-persistence guarantee. - -## Contributing - -- [CONTRIBUTING.md](../CONTRIBUTING.md): development setup, the commit - convention, the review expectations and how a release is cut. +A self-hosted, OpenAI-compatible LLM proxy: one static Go binary, per-user API +keys, a curated model catalog and usage accounting. No request or response +content is ever persisted. + +## Start here + +| I want to | Go to | +|---|---| +| See it work, end to end | [Getting started](getting-started.md) | +| Deploy it on a server | [Install the binary](install/binary.md) | +| Deploy it with containers | [Install with Docker](install/docker.md) | +| Give my team access | [Set up SSO](guides/sso.md) | +| Meter Claude Code | [Meter Claude Code](guides/claude-code.md) | +| Look up a variable or endpoint | [Configuration](reference/configuration.md), [API](reference/api.md) | +| Understand why it is built this way | [Architecture](concepts/architecture.md) | + +## Deploy it + +- [install/binary.md](install/binary.md): release install, state layout, first + login, the loopback guard, a systemd unit. +- [install/docker.md](install/docker.md): the published image, compose, + Postgres, Coolify. +- [install/upgrades.md](install/upgrades.md): backup, upgrade, what migrations + do. + +## Operate it + +- [guides/providers.md](guides/providers.md): register upstreams, rotate + credentials, TLS, per-endpoint overrides. +- [guides/models.md](guides/models.md): discover, bind, alias, hide, resolve. +- [guides/api-keys.md](guides/api-keys.md): keys for people and services, and + how to revoke them. +- [guides/pricing.md](guides/pricing.md): per-model prices, the feed format, + what unpriced means. +- [guides/usage.md](guides/usage.md): what is recorded, application tags, + the usage and stats endpoints, Prometheus metrics. +- [guides/sso.md](guides/sso.md): OIDC login, group-derived roles, an + Authentik walkthrough, offboarding. +- [guides/claude-code.md](guides/claude-code.md): meter Anthropic-native tools + without touching their credentials. + +## Look it up + +- [reference/configuration.md](reference/configuration.md): every environment + variable, database URLs, the secret, CLI flags. +- [reference/api.md](reference/api.md): the HTTP surface endpoint by endpoint, + with error codes and headers. +- [reference/openapi.yaml](reference/openapi.yaml): the same surface as an + OpenAPI 3.1 spec. +- [reference/performance.md](reference/performance.md): throughput and latency + measurements, and how to reproduce them. + +## Understand it + +- [concepts/architecture.md](concepts/architecture.md): scope and non-goals, + the endpoint families, and every design rule with its rationale, including + the no-content-persistence guarantee. + +## Contribute + +- [CONTRIBUTING.md](../CONTRIBUTING.md): development setup, the writing rules + for these pages, the commit convention, how a release is cut. - [SECURITY.md](../SECURITY.md): how to report a vulnerability privately. diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index b6beea3..21cc6d8 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -142,6 +142,31 @@ quantity as unpriced and flags the event; summaries show `cost: null` and turns missing configuration into "free", which is exactly the failure a spend-accounting system exists to prevent. +### Usage numbers are normalised at read time + +The two supported wire formats disagree about what "input tokens" counts. The +OpenAI shape reports cached tokens as a subset of the prompt count +(`prompt_tokens_details.cached_tokens` ⊆ `prompt_tokens`), while Anthropic +reports cache reads and writes *outside* `input_tokens`. Summing either shape +naively produces a number that means something different per provider. + +Stored quantities stay raw as reported, because pricing multiplies each unit +by its own rate and cache reads and writes have their own rates. Every +aggregate (`/my/usage*`, `/admin/v1/usage*`, `/stats/*`) then normalises +`input_tokens` to the non-cached input: the OpenAI cached subset is subtracted +back out, and the relay's `cache_creation_tokens` (fresh input billed at a +premium) folds in while cache reads stay their own unit. + +So `input_tokens` always means "input processed at the full input rate" and is +comparable across providers, and `cached_input_tokens` is always the cheap +cache-read count on top of it. Normalising at write time instead would destroy +the raw numbers pricing needs; normalising in each caller would let two +dashboards disagree. + +An upstream that never reports `cached_tokens` shows no cached quantity. +Absence means "not reported", never "free", for the same reason unpriced is +not zero. + ### The schema is content-free by construction This is the hard privacy guarantee: **no prompt, completion, embedding input @@ -209,4 +234,4 @@ request starts; the relay holds no store handle, and usage is written after the fact, detached from the request path. Holding a pooled connection for the duration of a streamed response pins one connection per in-flight stream and collapses the pool at modest concurrency. This rule is what lets a small pool -serve thousands of concurrent streams; see [performance.md](performance.md). +serve thousands of concurrent streams; see [performance](../reference/performance.md). diff --git a/docs/reference/api.md b/docs/reference/api.md index e088aa0..3c8d683 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -17,7 +17,7 @@ The full machine-readable spec is in [openapi.yaml](openapi.yaml). | `POST /v1/audio/transcriptions` | Planned | Streamed multipart, no disk spill; `audio_seconds` unit reserved. | | `POST /v1/audio/translations` | Out of scope | Add on demand once transcription exists. | | `POST /v1/responses` | Out of scope for 1.0 | OpenAI Responses API; revisit with evidence. | -| `/v1/batches`, `/v1/files` | Out of scope | Deliberate non-goal; see [architecture.md](architecture.md#scope-and-non-goals). | +| `/v1/batches`, `/v1/files` | Out of scope | Deliberate non-goal; see [architecture](../concepts/architecture.md#scope-and-non-goals). | | `/v1/images/*`, `/v1/audio/speech`, `/v1/moderations`, fine-tuning, assistants | Out of scope | Non-goals. | | `POST /v1/messages` (Anthropic dialect) | Planned | Second ingress adapter over the same catalog. Until then, the transparent relay (below) covers Anthropic-native tooling that brings its own credentials. | @@ -43,11 +43,11 @@ Requests may carry `x-llmproxy-tags`, a comma-separated list of `key:value` pairs (`app:dataindex,context:search`) naming the calling application. The proxy stores the normalised list on the usage event and the Usage tab's Apps subtab breaks spend down by it; see -[keys-and-usage.md](keys-and-usage.md#application-tags). Malformed pairs are +[usage](../guides/usage.md#tag-requests-by-application). Malformed pairs are dropped rather than rejected, and the header never reaches an upstream. No request or response content is ever logged or persisted, and there is no -flag to turn that on; see [architecture.md](architecture.md). +flag to turn that on; see [architecture](../concepts/architecture.md). Proxy-generated error codes: `missing_api_key`, `invalid_api_key`, `admin_required`, `model_not_found` (404), `unknown_endpoint` (404, with a @@ -75,7 +75,7 @@ The proxy's usage is team-visible by design; the admin role gates configuration, not visibility. Every endpoint accepts the same filters: `principal`, `key` (an API key id), `provider`, `model`, `client`, `tag`, `outcome` and the `since`/`until` window; see -[keys-and-usage.md](keys-and-usage.md#team-statistics). `tag` takes one exact +[usage](../guides/usage.md#query-team-wide-statistics). `tag` takes one exact `key:value` pair, is repeatable up to four times, and several pairs narrow together; a pair nothing carries simply matches nothing. `outcome` takes `ok`, `upstream_error`, `unreachable`, `cancelled` or the meta value @@ -110,7 +110,7 @@ unary bodies) and recorded under the sentinel provider event, except HEAD/OPTIONS probes, which relay unrecorded. Proxy-generated errors: 404 `unknown_relay_token`, 404 `transparent_relay_disabled`, 502 `provider_unreachable`. Docs: -[transparent-relay.md](transparent-relay.md). +[claude-code](../guides/claude-code.md). ## Admin (`/admin/v1`, admin role required, lists paginated) @@ -129,6 +129,51 @@ Proxy-generated errors: 404 `unknown_relay_token`, 404 | `GET /requests?limit&offset` | The request metadata log with per-unit quantities (who, key, model, outcome, tokens; never content); same filters as `/stats/requests` | | `GET /events` | Metadata-only admin audit trail | +### Pagination + +Every admin list endpoint (`providers`, `models`, `keys`, `principals`, +`events`) accepts `limit` (1 to 500, default 100) and `offset` (default 0) and +echoes both back: + +```bash +curl -s "$P/admin/v1/models?provider=vllm-1&limit=50&offset=50" \ + -H "authorization: Bearer $ADMIN" +``` + +Out-of-range values fall back to the defaults. Ordering is by name or alias +(keys by creation time, events newest first), so pagination is stable between +requests. + +### Audit trail + +Every admin mutation writes a metadata-only event in the same transaction as +the change: `provider.create|update|delete`, `model.create|update|delete`, +`principal.create`, `key.create`, `key.delete`, `pricing.load`. + +```bash +curl -s "$P/admin/v1/events?limit=3" -H "authorization: Bearer $ADMIN" +``` + +```json +{ + "events": [ + { + "ts": "2026-07-29T09:20:01.334019Z", + "actor_principal_id": "b1f4c2d8e96a4f0b8d3a5c7e9f012345", + "action": "model.update", + "target_kind": "model", + "target_ref": "qwen-72b" + } + ], + "limit": 3, + "offset": 0 +} +``` + +Events are newest first and carry identifiers only, never payloads. Direct CLI +operations (`llmproxy key create` and friends) bypass the HTTP layer and are +not audited. + ## Browser auth (`/auth`) | Endpoint | Purpose | diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 2ec2080..11525d5 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -1,43 +1,44 @@ # Configuration All configuration is environment-driven with an `LLMPROXY_` prefix. There is -no config file. A malformed value (for example a non-numeric port) silently -falls back to the default, so double-check values when something looks like it -was ignored. +no config file. + +A malformed value (for example a non-numeric port) silently falls back to the +default, so check the value when a setting looks like it was ignored. ## Environment variables | Variable | Default | Meaning | |---|---|---| -| `LLMPROXY_DATABASE_URL` | `llmproxy.db` | SQLite file path, or a `postgres://` URL. See below. | -| `LLMPROXY_KEY_SECRET` | empty | Explicit HMAC / encryption secret. When empty, a random secret is generated and stored in the secret file. | -| `LLMPROXY_SECRET_FILE` | `.llmproxy/secret` | Where the generated secret lives (created mode 0600 on first run). Ignored when `LLMPROXY_KEY_SECRET` is set. | -| `LLMPROXY_HOST` | `127.0.0.1` | Bind address. | -| `LLMPROXY_PORT` | `4000` | Bind port. | -| `LLMPROXY_ALLOW_NONLOCAL` | `false` | Allow binding a non-loopback address in local (no-SSO) mode. Accepts Go bool syntax (`1`, `true`, ...). Has no effect in SSO mode. | -| `LLMPROXY_OIDC_ISSUER` | empty | OIDC issuer URL. Unset means local single-admin mode; set means SSO mode. | -| `LLMPROXY_OIDC_CLIENT_ID` | empty | OAuth client id. Required in SSO mode. | -| `LLMPROXY_OIDC_CLIENT_SECRET` | empty | OAuth client secret. Required in SSO mode. | -| `LLMPROXY_OIDC_REDIRECT_URL` | empty | Absolute callback URL, `https://your-proxy/auth/callback`. Required in SSO mode. An `https://` value also marks session cookies `Secure` (as does any request arriving over TLS or with `X-Forwarded-Proto: https`), and its host is accepted by the same-origin check when a reverse proxy rewrites `Host`. | -| `LLMPROXY_OIDC_SCOPES` | `openid profile email` | Space-separated scopes requested from the IdP. | -| `LLMPROXY_OIDC_GROUPS_CLAIM` | `groups` | Userinfo claim (a string array) read for group membership. | -| `LLMPROXY_OIDC_ADMIN_GROUP` | empty | Members of this group get the `admin` role, reconciled on every login. Empty means SSO users are never auto-promoted to admin. | -| `LLMPROXY_OIDC_REQUIRED_GROUP` | empty | When set, users not in this group cannot log in at all (403 `sso_group_required`). | -| `LLMPROXY_SESSION_TTL` | `12h` | Browser session lifetime (Go duration). Sessions are server-side rows; logout and `POST /admin/v1/principals/{id}/revoke-sessions` invalidate them immediately by deletion. | -| `LLMPROXY_LOCAL_ADMIN_NAME` | `local-admin` | Name of the admin principal bootstrapped in local mode. | -| `LLMPROXY_ADMIN_PASSWORD` | empty | Browser password for the local admin principal. When empty, a random one is generated at first boot and written to the password file. See below. | -| `LLMPROXY_ADMIN_PASSWORD_FILE` | `.llmproxy/admin-password` | Where the generated admin password lives (created mode 0600). Ignored when `LLMPROXY_ADMIN_PASSWORD` is set. | -| `LLMPROXY_ADMIN_PASSWORD_DISABLED` | `false` | Disable password login entirely, for SSO-only deployments. | -| `LLMPROXY_CATALOG_TTL` | `5s` | Alias-resolution cache TTL (Go duration). Catalog mutations through the admin API invalidate the cache immediately; this TTL is the staleness bound for changes made directly in the database or from another instance. | -| `LLMPROXY_MAX_BODY_BYTES` | `10485760` (10 MiB) | Request body ceiling on the `/v1` proxy endpoints. Oversize requests get 413 `request_too_large`. | -| `LLMPROXY_MAX_EMBEDDING_BATCH` | `2048` | Maximum items in an embeddings `input` array. `0` or negative disables the check. | -| `LLMPROXY_PRICING_FILE` | empty | Path to a pricing feed JSON loaded at startup. Stored (and activated) only when its `version` differs from the currently active feed. Format in [keys-and-usage.md](keys-and-usage.md#pricing). | -| `LLMPROXY_TRANSPARENT_ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Target of the [transparent Anthropic relay](transparent-relay.md). Empty disables the relay. | -| `LLMPROXY_LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`. At `info`, every request logs one line (method, path, status, duration); health checks and static assets only appear at `debug`. Client and server errors log at `warn`/`error`, so they stay visible at higher levels. Logs go to stderr and never contain request or response content. | +| `LLMPROXY_DATABASE_URL` | `llmproxy.db` | SQLite file path, or a `postgres://` URL. See [database URLs](#database-urls) | +| `LLMPROXY_KEY_SECRET` | empty | Explicit HMAC / encryption secret. When empty, a random secret is generated and stored in the secret file | +| `LLMPROXY_SECRET_FILE` | `.llmproxy/secret` | Where the generated secret lives (created mode 0600 on first run). Ignored when `LLMPROXY_KEY_SECRET` is set | +| `LLMPROXY_HOST` | `127.0.0.1` | Bind address | +| `LLMPROXY_PORT` | `4000` | Bind port | +| `LLMPROXY_ALLOW_NONLOCAL` | `false` | Allow binding a non-loopback address in local (no-SSO) mode. Accepts Go bool syntax. No effect in SSO mode. See [the loopback guard](../install/binary.md#the-loopback-guard) | +| `LLMPROXY_OIDC_ISSUER` | empty | OIDC issuer URL. Unset means local single-admin mode; set means SSO mode | +| `LLMPROXY_OIDC_CLIENT_ID` | empty | OAuth client id. Required in SSO mode | +| `LLMPROXY_OIDC_CLIENT_SECRET` | empty | OAuth client secret. Required in SSO mode | +| `LLMPROXY_OIDC_REDIRECT_URL` | empty | Absolute callback URL, `https://your-proxy/auth/callback`. Required in SSO mode. An `https://` value also marks session cookies `Secure` (as does any request arriving over TLS or with `X-Forwarded-Proto: https`), and its host is accepted by the same-origin check when a reverse proxy rewrites `Host` | +| `LLMPROXY_OIDC_SCOPES` | `openid profile email` | Space-separated scopes requested from the IdP | +| `LLMPROXY_OIDC_GROUPS_CLAIM` | `groups` | Userinfo claim (a string array) read for group membership | +| `LLMPROXY_OIDC_ADMIN_GROUP` | empty | Members get the `admin` role, reconciled on every login. Empty means SSO users are never auto-promoted | +| `LLMPROXY_OIDC_REQUIRED_GROUP` | empty | When set, users outside this group cannot log in at all (403 `sso_group_required`) | +| `LLMPROXY_SESSION_TTL` | `12h` | Browser session lifetime. Sessions are server-side rows; logout and `POST /admin/v1/principals/{id}/revoke-sessions` invalidate them immediately by deletion | +| `LLMPROXY_LOCAL_ADMIN_NAME` | `local-admin` | Name of the admin principal bootstrapped in local mode | +| `LLMPROXY_ADMIN_PASSWORD` | empty | Browser password for the local admin. When empty, a random one is generated at first boot into the password file. See [the admin password](#the-admin-password) | +| `LLMPROXY_ADMIN_PASSWORD_FILE` | `.llmproxy/admin-password` | Where the generated admin password lives (created mode 0600). Ignored when `LLMPROXY_ADMIN_PASSWORD` is set | +| `LLMPROXY_ADMIN_PASSWORD_DISABLED` | `false` | Disable password login entirely, for SSO-only deployments | +| `LLMPROXY_CATALOG_TTL` | `5s` | Alias-resolution cache TTL. Admin API mutations invalidate the cache immediately; this TTL bounds staleness for changes made directly in the database or from another instance | +| `LLMPROXY_MAX_BODY_BYTES` | `10485760` (10 MiB) | Request body ceiling on the `/v1` endpoints. Oversize requests get 413 `request_too_large` | +| `LLMPROXY_MAX_EMBEDDING_BATCH` | `2048` | Maximum items in an embeddings `input` array. `0` or negative disables the check | +| `LLMPROXY_PRICING_FILE` | empty | Path to a pricing feed JSON loaded at startup. Stored only when its `version` differs from the active feed. Format in [pricing](../guides/pricing.md#load-a-whole-price-list) | +| `LLMPROXY_TRANSPARENT_ANTHROPIC_BASE_URL` | `https://api.anthropic.com` | Target of the [transparent Anthropic relay](../guides/claude-code.md). Empty disables the relay | +| `LLMPROXY_LOG_LEVEL` | `info` | `debug`, `info`, `warn` or `error`. At `info`, every request logs one line (method, path, status, duration); health checks and static assets appear only at `debug`. Client and server errors log at `warn`/`error`. Logs go to stderr and never contain request or response content | Durations use Go syntax: `5s`, `90m`, `12h`, `1h30m`. -## DATABASE_URL forms +## Database URLs Anything starting with `postgres://` or `postgresql://` selects Postgres (via pgx, max 20 connections): @@ -46,9 +47,9 @@ pgx, max 20 connections): LLMPROXY_DATABASE_URL=postgres://llmproxy:secret@db.internal:5432/llmproxy ``` -Everything else is treated as a SQLite file path (an optional `sqlite://` -prefix is stripped). SQLite is opened with WAL journaling, a 30 s busy -timeout, `synchronous=NORMAL` and max 8 connections: +Everything else is a SQLite file path (an optional `sqlite://` prefix is +stripped). SQLite is opened with WAL journaling, a 30 s busy timeout, +`synchronous=NORMAL` and max 8 connections: ``` LLMPROXY_DATABASE_URL=/var/lib/llmproxy/llmproxy.db @@ -64,31 +65,24 @@ startup: `LLMPROXY_KEY_SECRET` verbatim if set, otherwise the contents of What it protects, via derived keys: -- API key verification. Caller keys are stored only as HMAC-SHA256 digests - under this secret. A stolen database cannot be used to recover or verify - keys without it. -- Provider credentials. Upstream API keys must be recoverable (they are - forwarded upstream), so they are encrypted with AES-256-GCM under a key - derived from the secret. -- Session cookies. The browser session cookie holds an opaque random token - stored server-side only as a keyed hash, like API keys. (The OAuth state - cookie and pre-upgrade session cookies are signed tokens under another - derived key.) - -Consequences: - -- Losing the secret: every API key stops validating and every stored provider - credential becomes undecryptable. Recovery means minting new keys (the CLI - still works, since it writes to the database directly) and re-entering each - provider's `api_key` via `PATCH /admin/v1/providers/{name}`. The database - contents (principals, models, usage history) survive. -- Rotating the secret has exactly the same effect as losing it; there is no - dual-key rotation mechanism. Treat rotation as a planned re-keying event. -- The CLI and the server must see the same secret (same env or same secret - file), or CLI-minted keys will not validate against the server. +| | How | +|---|---| +| API keys | Stored only as HMAC-SHA256 digests under this secret | +| Provider credentials | Encrypted with AES-256-GCM, since they must be recoverable to forward upstream | +| Sessions | The cookie holds an opaque token stored server-side as a keyed hash | -Back the secret up separately from the database. If both leak together, the -attacker can verify keys offline and decrypt provider credentials. +Four consequences: + +- **Losing the secret** stops every API key validating and makes every stored + provider credential undecryptable. Recovery means minting new keys (the CLI + still works, since it writes to the database directly) and re-entering each + provider's `api_key`. Principals, models and usage history survive. +- **Rotating the secret has exactly the same effect as losing it.** There is + no dual-key rotation. Treat it as a planned re-keying event. +- **The CLI and the server must see the same secret**, through the same + environment or the same secret file, or CLI-minted keys will not validate. +- **Back it up separately from the database.** If both leak together, the + attacker can verify keys offline and decrypt provider credentials. ## The admin password @@ -97,16 +91,18 @@ without SSO configured. In SSO mode it is the break-glass path: an IdP outage cannot lock you out of your own proxy. Resolution order: `LLMPROXY_ADMIN_PASSWORD` verbatim if set, otherwise the -contents of `LLMPROXY_ADMIN_PASSWORD_FILE`, generated on first boot. The -password is compared in constant time, never stored in the database, and a -failed attempt is delayed by 400 ms as a blunt brute-force damper (put real -rate limiting in your reverse proxy if the login page is internet-facing). -To rotate a generated password, delete the file and restart. To run SSO-only, -set `LLMPROXY_ADMIN_PASSWORD_DISABLED=1`. +contents of `LLMPROXY_ADMIN_PASSWORD_FILE`, generated on first boot. + +- **To rotate a generated password**, delete the file and restart. +- **To run SSO-only**, set `LLMPROXY_ADMIN_PASSWORD_DISABLED=1`. +- **The password is never stored in the database** and is compared in constant + time. A failed attempt is delayed by 400 ms as a blunt brute-force damper; + put real rate limiting in your reverse proxy if the login page is + internet-facing. -A successful login issues the same server-side session as SSO. Sessions -carry the principal's role: the local admin's session can use the full admin -API (mutations are origin-checked), which is what the built-in UI does. +A successful login issues the same server-side session as SSO, carrying the +principal's role. The local admin's session can use the full admin API, with +mutations origin-checked. ## CLI flags @@ -119,7 +115,7 @@ environment variables: | `-port ` | `LLMPROXY_PORT` | | `-allow-nonlocal` | `LLMPROXY_ALLOW_NONLOCAL` (sets it true; there is no flag to force it false) | -Everything else is environment-only. The `key` and `principal` subcommands -take no server flags; they read the same environment (`LLMPROXY_DATABASE_URL`, -`LLMPROXY_KEY_SECRET`, `LLMPROXY_SECRET_FILE`, `LLMPROXY_LOCAL_ADMIN_NAME`) -to find the database and secret. +Everything else is environment-only. The `key`, `relay-token` and `principal` +subcommands take no server flags; they read `LLMPROXY_DATABASE_URL`, +`LLMPROXY_KEY_SECRET`, `LLMPROXY_SECRET_FILE` and `LLMPROXY_LOCAL_ADMIN_NAME` +to find the database and the secret. From 9c1e3a6478ad53ab24d4f72e491dfab0c33a9f22 Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:16:04 -0600 Subject: [PATCH 5/6] docs: add a rendered site (MkDocs Material) with CI mkdocs.yml mirrors the docs tree in an explicit nav and maps every pre-reorganisation path to its new home with mkdocs-redirects, so links published against the old layout keep working on the site. .github/workflows/docs.yml runs 'mkdocs build --strict' on PRs touching docs/ and deploys to GitHub Pages on push to main. Versions are pinned, mkdocs included, because a 2.0 is announced that drops the plugin system. 'just docs' previews locally through uvx, so the repo gains no permanent Python dependency. --- .github/workflows/docs.yml | 53 +++++++++++++++++++++++ .gitignore | 1 + docs/README.md | 8 ++-- justfile | 4 ++ mkdocs.yml | 88 ++++++++++++++++++++++++++++++++++++++ 5 files changed, 151 insertions(+), 3 deletions(-) create mode 100644 .github/workflows/docs.yml create mode 100644 mkdocs.yml diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..6d83253 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,53 @@ +name: Docs + +on: + push: + branches: [main] + paths: + - 'docs/**' + - 'mkdocs.yml' + - '.github/workflows/docs.yml' + pull_request: + paths: + - 'docs/**' + - 'mkdocs.yml' + - '.github/workflows/docs.yml' + +permissions: + contents: read + +concurrency: + group: docs-${{ github.ref }} + cancel-in-progress: true + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Install MkDocs + # mkdocs is pinned too: a 2.0 is announced that drops the plugin system. + run: pip install mkdocs==1.6.1 mkdocs-material==9.5.49 mkdocs-redirects==1.2.2 + - name: Build (strict; fails on broken internal links) + run: mkdocs build --strict + - uses: actions/upload-pages-artifact@v3 + if: github.event_name == 'push' + with: + path: site + + deploy: + if: github.event_name == 'push' + needs: build + runs-on: ubuntu-latest + permissions: + pages: write + id-token: write + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index da3b097..c121104 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ /bin/ /dist/ +/site/ *.db *.db-wal *.db-shm diff --git a/docs/README.md b/docs/README.md index 26e480c..f433b06 100644 --- a/docs/README.md +++ b/docs/README.md @@ -60,6 +60,8 @@ content is ever persisted. ## Contribute -- [CONTRIBUTING.md](../CONTRIBUTING.md): development setup, the writing rules - for these pages, the commit convention, how a release is cut. -- [SECURITY.md](../SECURITY.md): how to report a vulnerability privately. +- [CONTRIBUTING.md](https://github.com/greyhavenhq/llmproxy/blob/main/CONTRIBUTING.md): + development setup, the writing rules for these pages, the commit convention, + how a release is cut. +- [SECURITY.md](https://github.com/greyhavenhq/llmproxy/blob/main/SECURITY.md): + how to report a vulnerability privately. diff --git a/justfile b/justfile index faf8814..61312e2 100644 --- a/justfile +++ b/justfile @@ -30,6 +30,10 @@ docker: lint: golangci-lint run +# Preview the documentation site at http://127.0.0.1:8000 (requires uv). +docs: + uvx --with mkdocs==1.6.1 --with mkdocs-material==9.5.49 --with mkdocs-redirects==1.2.2 mkdocs serve + # Remove build artifacts. clean: rm -rf bin diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..9070e85 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,88 @@ +site_name: llmproxy +site_description: >- + A self-hosted, OpenAI-compatible LLM proxy: one static Go binary, per-user + API keys, a curated model catalog and usage accounting. +site_url: https://greyhavenhq.github.io/llmproxy/ +repo_url: https://github.com/greyhavenhq/llmproxy +repo_name: greyhavenhq/llmproxy +edit_uri: edit/main/docs/ +copyright: Apache-2.0 + +docs_dir: docs + +theme: + name: material + features: + - navigation.sections + - navigation.top + - navigation.instant + - content.code.copy + - search.suggest + palette: + - media: "(prefers-color-scheme: light)" + scheme: default + primary: black + accent: indigo + toggle: + icon: material/brightness-7 + name: Switch to dark mode + - media: "(prefers-color-scheme: dark)" + scheme: slate + primary: black + accent: indigo + toggle: + icon: material/brightness-4 + name: Switch to light mode + +nav: + - Home: README.md + - Getting started: getting-started.md + - Deploy it: + - Install the binary: install/binary.md + - Install with Docker: install/docker.md + - Upgrades and backup: install/upgrades.md + - Operate it: + - Manage providers: guides/providers.md + - Manage models: guides/models.md + - Manage API keys: guides/api-keys.md + - Set prices: guides/pricing.md + - Read usage and cost: guides/usage.md + - Set up SSO: guides/sso.md + - Meter Claude Code: guides/claude-code.md + - Look it up: + - Configuration: reference/configuration.md + - API surface: reference/api.md + - Performance: reference/performance.md + - Understand it: + - Architecture: concepts/architecture.md + +plugins: + - search + # Keep links published before the docs were reorganised working. + - redirects: + redirect_maps: + installation.md: install/binary.md + docker.md: install/docker.md + upgrades.md: install/upgrades.md + providers-and-models.md: guides/providers.md + keys-and-usage.md: guides/api-keys.md + sso.md: guides/sso.md + transparent-relay.md: guides/claude-code.md + configuration.md: reference/configuration.md + api.md: reference/api.md + performance.md: reference/performance.md + architecture.md: concepts/architecture.md + +markdown_extensions: + - admonition + - tables + - toc: + permalink: true + - pymdownx.superfences + - pymdownx.highlight + - pymdownx.inlinehilite + +extra: + social: + - icon: fontawesome/brands/github + link: https://github.com/greyhavenhq/llmproxy From a23b6b5b0c930e678cb5b3c44c222ba1225543ed Mon Sep 17 00:00:00 2001 From: Mathieu Virbel Date: Fri, 28 Aug 2026 14:17:11 -0600 Subject: [PATCH 6/6] docs: point inbound references at the new doc paths README.md's Documentation section becomes a journey table and links the rendered site. CONTRIBUTING.md gains a 'Writing documentation' section holding the writing rules the rewrite applied. CLAUDE.md gains a Documentation section so the layout survives future edits. SECURITY.md, the justfile comment and every 'docs/.md' path are remapped. --- CLAUDE.md | 13 +++++++++---- CONTRIBUTING.md | 40 ++++++++++++++++++++++++++++++++++++---- README.md | 14 +++++++++++--- SECURITY.md | 4 ++-- justfile | 2 +- 5 files changed, 59 insertions(+), 14 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e98eb8e..fe12f0b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -10,8 +10,9 @@ just test # go test ./... just race # race detector, -count=1 just lint # golangci-lint run just ui # rebuild the embedded web UI (requires node) -just stress # load harness (cmd/stress), see docs/performance.md +just stress # load harness (cmd/stress), see docs/reference/performance.md just docker # docker build -t llmproxy . +just docs # preview the documentation site (requires uv) ``` Single test: `go test ./internal/server -run TestName`. Most server tests go through `testenv_test.go`, which boots a real server against a temp SQLite database. @@ -20,7 +21,7 @@ The web UI (`ui/`, React + Vite) builds into `internal/server/uidist/`, which is ## What this is -A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API keys, curated model catalog, usage accounting. Exactly two upstream wire formats (OpenAI-compatible and Anthropic native); the non-goals table in docs/architecture.md caps it there deliberately. Read docs/architecture.md before structural changes; it states every design rule with its rationale. +A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API keys, curated model catalog, usage accounting. Exactly two upstream wire formats (OpenAI-compatible and Anthropic native); the non-goals table in docs/concepts/architecture.md caps it there deliberately. Read docs/concepts/architecture.md before structural changes; it states every design rule with its rationale. ## Hard constraints @@ -33,7 +34,7 @@ A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API ke - `cmd/llmproxy`: CLI entry point; subcommands `serve`, `key`, `relay-token`, `principal`, `version`. - `cmd/stress`: self-contained load harness. -- `internal/config`: all configuration, environment-driven with `LLMPROXY_` prefix, defaults in `FromEnv()`. Document new variables in docs/configuration.md and expose them in `docker-compose.coolify.yml` as `${VAR:-default}`. +- `internal/config`: all configuration, environment-driven with `LLMPROXY_` prefix, defaults in `FromEnv()`. Document new variables in docs/reference/configuration.md and expose them in `docker-compose.coolify.yml` as `${VAR:-default}`. - `internal/server`: HTTP layer. `openai.go` is the OpenAI-compatible ingress, `transparent.go` the Anthropic relay, `admin.go` the admin API, `sso.go`/`password.go`/`session.go` auth, `ui.go` serves the embedded UI, `litellm.go` compatibility surface. - `internal/store`: all persistence; `schema.go` holds the DDL for both dialects. - `internal/catalog`: model alias -> provider binding resolution and capability checks (checked in exactly one place; a miss is a typed 400 `endpoint_not_supported`). @@ -49,6 +50,10 @@ A self-hosted OpenAI-compatible LLM proxy: one static Go binary, per-user API ke - API keys are stored as HMAC-SHA256 under a secret outside the database (`LLMPROXY_KEY_SECRET` or the generated secret file); plaintext shown once at mint. - Prometheus labels are aggregate-only (provider, model, endpoint, unit, outcome); never principals or keys. +## Documentation + +`docs/` is organised by reader intent: `getting-started.md` (tutorial), `install/` (deploy), `guides/` (operate), `reference/` (look up), `concepts/` (understand). Task pages are imperative and step-numbered with expected output; rationale is one sentence plus a link, and lives in `concepts/architecture.md`. The writing rules are in CONTRIBUTING.md. A new page needs an entry in `mkdocs.yml`'s nav and in the `docs/README.md` router. `mkdocs build --strict` fails on broken internal links and runs in CI on any PR touching `docs/`. + ## Deployment files -`docker-compose.yml` is the development setup (SQLite, loopback-only port). `docker-compose.coolify.yml` is the PaaS deployment (Postgres, no host port mapping, every `LLMPROXY_*` variable exposed as `${VAR:-default}` so Coolify lists them). Keep the two and docs/docker.md in sync when configuration changes. +`docker-compose.yml` is the development setup (SQLite, loopback-only port). `docker-compose.coolify.yml` is the PaaS deployment (Postgres, no host port mapping, every `LLMPROXY_*` variable exposed as `${VAR:-default}` so Coolify lists them). Keep the two and docs/install/docker.md in sync when configuration changes. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf39636..86c64c0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,7 +15,7 @@ just test # go test ./... just race # race detector, -count=1 just lint # golangci-lint just ui # rebuild the embedded web UI (requires node) -just stress # load harness, see docs/performance.md +just stress # load harness, see docs/reference/performance.md ``` `just` on its own lists every recipe. @@ -32,13 +32,45 @@ temporary SQLite database. `go test ./internal/server -run TestName` runs one. purpose so a plain `go build` embeds the current UI without a node toolchain; the `ui-drift` CI job fails if it is stale. - If you changed configuration, document the variable in - `docs/configuration.md` and expose it in `docker-compose.coolify.yml` as + `docs/reference/configuration.md` and expose it in `docker-compose.coolify.yml` as `${VAR:-default}`. - New behaviour comes with a test. +## Writing documentation + +`docs/` is organised by what the reader is trying to do: `getting-started.md` +to try it, `install/` to deploy it, `guides/` to operate it, `reference/` to +look something up, `concepts/` to understand why. Put a new page where its +reader would look for it, and add it to the nav in `mkdocs.yml` and the router +in `docs/README.md`. + +Six rules keep the task pages consistent: + +- **Second person, imperative, present tense.** "Create a key", not "a key can + be created". +- **One action per step.** Number the steps, and give each one its command and + the output that says it worked. +- **Happy path first.** Open with the goal and the prerequisites, then the + minimal path. Variations (TLS, Postgres, overrides) come after, under their + own headings, so they can be skipped. +- **Rationale is one sentence and a link.** The why belongs in + `concepts/architecture.md`, stated once. +- **Consequences stay, justifications go.** "Losing the secret invalidates + every key" survives the cut. "The alternative would buy microseconds" does + not, outside `concepts/`. +- **Reference pages are tables**, not narrative. The tutorial stays one tested + path. + +`concepts/architecture.md` and the README are the exception: they explain +rather than instruct, and their voice is deliberately different. + +Check a change with `just docs` (local preview) or `mkdocs build --strict`, +which fails on broken internal links. The `Docs` workflow runs the strict +build on every PR that touches `docs/`. + ## Design rules that are not negotiable -Read [docs/architecture.md](docs/architecture.md) before any structural change. +Read [docs/concepts/architecture.md](docs/concepts/architecture.md) before any structural change. It states every rule with its rationale. The ones that will get a patch rejected outright: @@ -58,7 +90,7 @@ rejected outright: - **Prometheus labels stay aggregate-only** (provider, model, endpoint, unit, outcome); never principals or keys. -The non-goals table in `docs/architecture.md` is deliberate. If a feature is +The non-goals table in `docs/concepts/architecture.md` is deliberate. If a feature is listed there, open an issue to discuss before writing code. ## Commit messages diff --git a/README.md b/README.md index aa915c9..66391bb 100644 --- a/README.md +++ b/README.md @@ -60,15 +60,23 @@ client.chat.completions.create(model="qwen-72b", messages=[{"role": "user", "content": "hello"}]) ``` -See [docs/docker.md](docs/docker.md) for compose, Postgres and the published image. +See [docs/install/docker.md](docs/install/docker.md) for compose, Postgres and the published image, and [docs/install/binary.md](docs/install/binary.md) for systemd. ## Documentation -Everything lives in [docs/](docs/README.md): a getting-started tutorial, how-to guides (installation, Docker, SSO, providers and models, keys and usage, the Anthropic relay), reference (configuration, the HTTP API, an OpenAPI 3.1 spec, performance) and an architecture explanation. +Rendered at **[greyhavenhq.github.io/llmproxy](https://greyhavenhq.github.io/llmproxy/)**, source in [docs/](docs/README.md). + +| | | +|---|---| +| Try it | [Getting started](docs/getting-started.md) | +| Deploy it | [binary](docs/install/binary.md), [Docker](docs/install/docker.md), [upgrades](docs/install/upgrades.md) | +| Operate it | [providers](docs/guides/providers.md), [models](docs/guides/models.md), [API keys](docs/guides/api-keys.md), [pricing](docs/guides/pricing.md), [usage](docs/guides/usage.md), [SSO](docs/guides/sso.md), [Claude Code](docs/guides/claude-code.md) | +| Look it up | [configuration](docs/reference/configuration.md), [API](docs/reference/api.md), [OpenAPI spec](docs/reference/openapi.yaml), [performance](docs/reference/performance.md) | +| Understand it | [architecture](docs/concepts/architecture.md) | ## Performance -The built-in stress harness sustains around 8,500 req/s at concurrency 200 on SQLite, every request accounted, heap under 50 MiB. Numbers, method and caveats in [docs/performance.md](docs/performance.md); reproduce with `just stress`. +The built-in stress harness sustains around 8,500 req/s at concurrency 200 on SQLite, every request accounted, heap under 50 MiB. Numbers, method and caveats in [docs/reference/performance.md](docs/reference/performance.md); reproduce with `just stress`. ## Development diff --git a/SECURITY.md b/SECURITY.md index 72b4a65..963701c 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -31,10 +31,10 @@ Things that are known and deliberate, not vulnerabilities: box grants access, which is why the server refuses to bind a non-loopback address without `-allow-nonlocal`. Running it exposed with that flag is a configuration choice, documented in - [docs/installation.md](docs/installation.md). + [docs/install/binary.md](docs/install/binary.md). - **The proxy does not inspect request bodies.** It forwards them byte-for-byte. Content filtering and prompt-level policy are out of scope by design; see the - non-goals table in [docs/architecture.md](docs/architecture.md). + non-goals table in [docs/concepts/architecture.md](docs/concepts/architecture.md). - **Losing the key secret invalidates keys, not confidentiality of past traffic.** The secret (`LLMPROXY_KEY_SECRET` or the generated secret file) is what makes stored key hashes and encrypted provider credentials usable. diff --git a/justfile b/justfile index 61312e2..e827f07 100644 --- a/justfile +++ b/justfile @@ -18,7 +18,7 @@ test: race: go test -race -count=1 ./... -# Run the self-contained stress harness (see docs/performance.md). +# Run the self-contained stress harness (see docs/reference/performance.md). stress: go run ./cmd/stress -requests 2000 -concurrency 100 -stream-ratio 0.5