feat: let an alias point at several models and fail over between them

docker-compose.coolify.yml

@@ -50,6 +50,7 @@ services:
       LLMPROXY_ADMIN_PASSWORD_FILE: ${LLMPROXY_ADMIN_PASSWORD_FILE:-/data/admin-password}
       LLMPROXY_ADMIN_PASSWORD_DISABLED: ${LLMPROXY_ADMIN_PASSWORD_DISABLED:-0}
       LLMPROXY_CATALOG_TTL: ${LLMPROXY_CATALOG_TTL:-5s}
+      LLMPROXY_FAILOVER_COOLDOWN: ${LLMPROXY_FAILOVER_COOLDOWN:-30s}
       LLMPROXY_MAX_BODY_BYTES: ${LLMPROXY_MAX_BODY_BYTES:-10485760}
       LLMPROXY_MAX_EMBEDDING_BATCH: ${LLMPROXY_MAX_EMBEDDING_BATCH:-2048}
       LLMPROXY_PRICING_FILE: ${LLMPROXY_PRICING_FILE:-}

docs/concepts/architecture.md

@@ -226,6 +226,15 @@ silent pick makes routing a mystery. A dry-run `resolve` endpoint answers
 another model, but only one hop, resolved in the same query, so lookup stays
 single-step and cycles are unrepresentable.
 
+### An alias with several targets fails over before the first byte
+
+An alias may list several targets with a declared strategy. The strategy is
+explicit, so the pick is never silent. The proxy tries the next target only
+before it writes a byte to the caller, because the body passes through
+unchanged and a partial response cannot be taken back. Rotation, in-flight
+counts and cooldowns are kept in memory per replica: they are hints about
+the near past, not state, and the database stays the source of truth.
+
 ### Admin mutations are audited in-transaction
 
 Every admin mutation writes a metadata-only `admin_event` row in the same

docs/guides/models.md

@@ -142,10 +142,72 @@ Three rules, enforced at write time:
 Usage is recorded against the name the caller passed, so two teams pointing at
 one model stay distinguishable in the summaries.
 
+## Point a name at several models
+
+Give `targets` and a `strategy` to spread one name over several models, for
+example the same model on two providers:
+
+```bash
+curl -s $P/admin/v1/models -H "authorization: Bearer $ADMIN" \
+  -H 'content-type: application/json' \
+  -d '{"alias": "acme/smart", "targets": ["glm-tensorx", "glm-vllm"], "strategy": "failover"}'
+```
+
+```json
+{
+  "alias": "acme/smart",
+  "target": null,
+  "targets": [
+    {"alias": "glm-tensorx", "weight": 1},
+    {"alias": "glm-vllm", "weight": 1}
+  ],
+  "strategy": "failover",
+  "capabilities": ["chat", "chat_stream"]
+}
+```
+
+The strategy picks the first target for each request:
+
+| `strategy` | First target |
+|---|---|
+| `failover` (default) | The first in the list |
+| `round_robin` | The next in the list, in turn |
+| `weighted` | A random pick, in proportion to `weight` |
+| `least_busy` | The one with the fewest requests in flight |
+
+A target entry is a model name, or `{"alias": "...", "weight": 3}`. A weight
+is used only by `weighted` and defaults to 1.
+
+With every strategy, the proxy tries the next target when an attempt fails
+before any byte reaches the caller:
+
+- The upstream is unreachable.
+- The upstream answers 429 or a 5xx status.
+
+After the first byte, the response is the caller's, and an error after that
+point is relayed as it is. A target that failed goes to the end of the order
+for `LLMPROXY_FAILOVER_COOLDOWN` (30 seconds by default). The response header
+`x-llmproxy-attempts` gives the number of targets tried. Each failed attempt
+is recorded as a usage row flagged `failed_over`, which request counts leave
+out.
+
+The rules of [a single target](#point-a-name-at-another-model) apply to each
+target. Two more apply to the list:
+
+- **Capabilities are the intersection.** The name serves only what every
+  target serves.
+- **Prices come from the target that answered,** unless the name has a price
+  of its own.
+
+A `strategy` with one target, or an unknown one, is 400 `invalid_strategy`. A
+name listed twice is 400 `invalid_target`.
+
 ## Edit a model
 
 `PATCH /admin/v1/models/{alias}` takes any of `alias`, `provider`,
-`capabilities`, `upstream_name`, `target`, `pricing` and `hidden`. Everything
+`capabilities`, `upstream_name`, `target`, `targets`, `strategy`, `pricing`
+and `hidden`. `targets` replaces the whole list, and `strategy` alone keeps
+the list. Everything
 is editable in place; nothing here needs deleting and recreating.
 
 ```bash
@@ -213,10 +275,26 @@ curl -s "$P/admin/v1/resolve?model=qwen-72b&endpoint=chat&stream=true" \
   "provider": "vllm-1",
   "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct",
   "url": "http://10.0.0.5:8000/v1/chat/completions",
-  "capabilities": ["chat", "chat_stream", "completions"]
+  "capabilities": ["chat", "chat_stream", "completions"],
+  "strategy": null,
+  "targets": [
+    {
+      "alias": "qwen-72b",
+      "provider": "vllm-1",
+      "upstream_name": "Qwen/Qwen2.5-VL-72B-Instruct",
+      "url": "http://10.0.0.5:8000/v1/chat/completions",
+      "weight": 1,
+      "cooling_down": false,
+      "cooldown_until": null
+    }
+  ]
 }
 ```
 
+For a name with several targets, `targets` lists them in the order the
+next request would try, and `cooling_down` marks a target that failed
+recently. The order is per replica.
+
 `endpoint` defaults to `chat`; `stream=true` additionally requires
 `chat_stream`. `url` reflects any per-endpoint override. The failure modes are
 the data-plane ones: 404 `model_not_found` (unknown name or disabled provider)

docs/reference/api.md

@@ -11,7 +11,7 @@ The full machine-readable spec is in [openapi.yaml](openapi.yaml).
 |---|---|---|
 | `POST /v1/chat/completions` | **Supported** | Unary JSON and SSE streaming. Vision content parts, tool/function calls, response_format, logprobs etc. pass through untouched. Capability `chat` (+ `chat_stream` for streaming). |
 | `POST /v1/completions` | **Supported** | Legacy text completions, unary and streamed. Capability `completions`. |
-| `GET /v1/models` | **Supported** | Aliases on enabled providers, minus the hidden ones; `?endpoint=chat\|embeddings\|...` filters by capability. `?include_hidden=1` adds hidden models and needs an API key or session (401 without one). `?include_pricing=1` adds `pricing` (per million units, the prices in force) and `pricing_inherited`, and needs an API key or session too. Each entry carries the OpenAI fields plus `capabilities` (resolved), `alias_of` (the model this name points at, or `null`) and `hidden`. Public, no API key required. |
+| `GET /v1/models` | **Supported** | Aliases on enabled providers, minus the hidden ones; `?endpoint=chat\|embeddings\|...` filters by capability. `?include_hidden=1` adds hidden models and needs an API key or session (401 without one). `?include_pricing=1` adds `pricing` (per million units, the prices in force) and `pricing_inherited`, and needs an API key or session too. Each entry carries the OpenAI fields plus `capabilities` (resolved), `alias_of` (the model this name points at, or `null`), `targets` and `strategy` (for an alias with several targets, else `null`) and `hidden`. Public, no API key required. |
 | `GET /v1/models/{id}` | Not yet | Trivial to add; nothing has needed it. |
 | `POST /v1/embeddings` | **Supported** | Unary passthrough. Capability `embeddings`. Array `input` is capped at `LLMPROXY_MAX_EMBEDDING_BATCH` items (default 2048); larger batches get 400 `embedding_batch_too_large`. |
 | `POST /v1/audio/transcriptions` | Planned | Streamed multipart, no disk spill; `audio_seconds` unit reserved. |
@@ -29,16 +29,19 @@ only exposes curated aliases and provider names. Its `include_hidden=1` form
 is not, since it names models an admin took off the list.
 
 Model names callers see are curated aliases, globally unique across all
-providers; there is no bare-upstream-name fallback and no multi-provider
-resolution. Calling an endpoint outside a model's capability set fails at the
+providers; there is no bare-upstream-name fallback. An alias can point at
+several models, and the proxy then picks one per request by the alias's
+strategy (see [models](../guides/models.md#point-a-name-at-several-models)).
+Calling an endpoint outside a model's capability set fails at the
 proxy with a 400 naming the supported capabilities, never with a confusing
 upstream 404.
 
 Every response carries `x-llmproxy-provider` and `x-llmproxy-model`. When the
 provider has rate-limit tracking configured, responses also carry
 `x-llmproxy-ratelimit-requests-remaining`, `x-llmproxy-ratelimit-requests-limit`,
 `x-llmproxy-ratelimit-tokens-remaining` and `x-llmproxy-ratelimit-tokens-limit`
-(only the observed values are set). Errors are
+(only the observed values are set). A request to an alias with several
+targets also carries `x-llmproxy-attempts`, the number of targets tried. Errors are
 OpenAI-shaped with an added `llmproxy.source` field plus
 `x-llmproxy-error-source: proxy|upstream`; upstream error bodies and status
 codes pass through intact.
@@ -89,7 +92,7 @@ together; a pair nothing carries simply matches nothing. `outcome` takes
 |---|---|
 | `GET /stats/series?bucket&since&until` | Bucketed usage across everyone |
 | `GET /stats/summary?since&until` | Usage aggregated per (principal, provider, model, endpoint, client, tags) |
-| `GET /stats/requests?limit&offset` | One page of the filtered request metadata log (never content), newest first; returns `{requests, limit, offset, total}` |
+| `GET /stats/requests?limit&offset` | One page of the filtered request metadata log (never content), newest first, with `failed_over` set on attempts another target replaced; returns `{requests, limit, offset, total}` |
 | `GET /stats/requests/facets?since&until` | Distinct principals, keys, providers, models, clients and tags in the window, for the explorer's filter options |
 | `GET /stats/errors?bucket&since&until` | The errors dashboard in one call: a gap-filled series of counts per outcome, plus a breakdown per (provider, model, endpoint, client, tags, outcome, error_kind, status_code) with request count, average duration, last-seen and time-to-outcome bands (<1s, 1-5s, 5-15s, 15-30s, 30-60s, 60-120s, >=120s). Rows with outcome `ok` are included so error rates have their denominator. |
 | `GET /stats/performance?bucket&since&until` | The performance view in one call: a summary, a gap-filled series and a per-model breakdown. Each holds `requests`, `measured`, and `duration_ms`, `ttft_ms` and `tokens_per_second` as `{mean, p50, p95}`, null when no request qualifies. The summary and the series also hold `concurrency` as `{average, peak}`. A range with more than 500,000 requests is a 400 `range_too_large`. |
@@ -125,15 +128,15 @@ Proxy-generated errors: 404 `unknown_relay_token`, 404
 |---|---|
 | `POST/GET/PATCH/DELETE /providers[/{name}]` | Register, inspect, edit, unregister providers (upstream key encrypted at rest, never returned) |
 | `GET /providers/{name}/discover` | Upstream model listing; read-only, never auto-binds |
-| `POST/GET/PATCH/DELETE /models[/{alias}]` | Bind upstream models to globally unique aliases with capability sets and per-unit prices; a name can instead `target` another model and inherit its provider, capabilities and prices (one hop). Everything, including the name, is editable in place; a binding serves as soon as it exists (disable the provider to take it offline) |
+| `POST/GET/PATCH/DELETE /models[/{alias}]` | Bind upstream models to globally unique aliases with capability sets and per-unit prices; a name can instead `target` another model and inherit its provider, capabilities and prices (one hop), or list several `targets` with a `strategy`. Everything, including the name, is editable in place; a binding serves as soon as it exists (disable the provider to take it offline) |
 | `GET /resolve?model&endpoint&stream` | Dry-run alias resolution |
 | `POST/GET /principals` | Users and service principals |
 | `POST /principals/{id}/revoke-sessions` | Delete every browser session of a principal (API keys untouched) |
 | `POST/GET/DELETE /keys[/{id}]` | Keys for any principal |
 | `POST/GET /pricing` | Load/inspect the versioned pricing feed (bulk; per-model prices go through `/models`) |
 | `GET /usage/summary?since&until&principal` | Usage and cost by principal/model/endpoint/unit |
 | `GET /usage/series?bucket&since&until&principal` | Usage bucketed by hour/day/week/month across everyone |
-| `GET /requests?limit&offset` | The request metadata log with per-unit quantities (who, key, model, outcome, tokens; never content); same filters as `/stats/requests` |
+| `GET /requests?limit&offset` | The request metadata log with per-unit quantities (who, key, model, outcome, `failed_over`, tokens; never content); same filters as `/stats/requests` |
 | `GET /events` | Metadata-only admin audit trail |
 
 ### Pagination

docs/reference/configuration.md

@@ -30,6 +30,7 @@ default, so check the value when a setting looks like it was ignored.
 | `LLMPROXY_ADMIN_PASSWORD_FILE` | `.llmproxy/admin-password` | Where the generated admin password lives (created mode 0600). Ignored when `LLMPROXY_ADMIN_PASSWORD` is set |
 | `LLMPROXY_ADMIN_PASSWORD_DISABLED` | `false` | Disable password login entirely, for SSO-only deployments |
 | `LLMPROXY_CATALOG_TTL` | `5s` | Alias-resolution cache TTL. Admin API mutations invalidate the cache immediately; this TTL bounds staleness for changes made directly in the database or from another instance |
+| `LLMPROXY_FAILOVER_COOLDOWN` | `30s` | How long an alias target that failed goes to the end of the order. `0` disables the cooldown. Kept per replica |
 | `LLMPROXY_MAX_BODY_BYTES` | `10485760` (10 MiB) | Request body ceiling on the `/v1` endpoints. Oversize requests get 413 `request_too_large` |
 | `LLMPROXY_MAX_EMBEDDING_BATCH` | `2048` | Maximum items in an embeddings `input` array. `0` or negative disables the check |
 | `LLMPROXY_PRICING_FILE` | empty | Path to a pricing feed JSON loaded at startup. Stored only when its `version` differs from the active feed. Format in [pricing](../guides/pricing.md#load-a-whole-price-list) |