An open-source platform for self-hosting open models: an OpenAI-compatible gateway with metering, and Kubernetes-native model deployment on GPU.
Repository: https://github.com/zlogic-labs/fleet
Status: early development. The design is settled and documented; the implementation is being built module by module against it.
Self-hosting an open model is not a GPU problem, it is an operations problem. The GPU is the easy half. What teams actually lack is a way to give the model an OpenAI-compatible endpoint, decide who may call it, charge them for what they used, and place it on hardware that is not fully utilised today.
Fleet targets that gap, and it targets the part of it nobody has built: the economics of a fleet whose cost is fixed rather than metered per token. See docs/architecture.md §6.
These are the rules the code is written against. Changing one requires an ADR.
| P1 | The OpenAI protocol is the only contract with an inference engine. No vendor SDK is ever imported. |
| P2 | Scale replicas horizontally, not cards vertically. |
| P3 | The gateway owns endpoint selection, because it is the only layer holding token counts and quota state. |
| P4 | Fleet owns its CRDs. Upstream operators are rendered to, never adopted as the API. |
| P5 | Rate limits are reserved up front and settled after, never deducted after the fact. |
| P6 | Nobody's token claim is trusted. The engine's usage is used when there is one; otherwise the gateway counts the text it forwarded — and GET /api/v1/usage-agreement checks those two measurements against each other. |
| P7 | The gateway does not depend on Kubernetes. Enforced by the module graph, not by review. |
| P8 | Billing allocates a fixed cost pool by GPU-hour; it is not a prepaid balance. |
This repository holds the gateway and everything about money: tenants, keys,
quotas, the ledger and the cost pool. The Kubernetes controller is a separate
repository, fleet-serving, so
that the module graph makes P7 a compile error rather than a convention —
nothing reachable from a gateway process can pull in client-go.
core/ gateway, control plane, billing, storage no k8s dependency
web/ operator console (React + antd) builds into core
deploy/ k3s bootstrap script and the kustomize manifests
docs/ architecture and ADRs
The two repositories talk over one versioned contract, PUT /api/v1/inventory
(core/pkg/inventory). That is the whole surface between them: fleet-serving
reads Kubernetes because only it can, and pushes what it finds; it never calls
back into the gateway.
Neither core nor the console reads Kubernetes. Node and GPU inventory
arrives from the operator running inside the cluster, which reports it to the
control plane (--report-to=http://127.0.0.1:8081); the gateway then learns
its endpoints from that same inventory (FLEET_CONTROL_PLANE_URL), so
scaling a deployment reaches the router without restarting anything. A cluster
page that is empty on a fresh checkout is the correct state rather than a
missing feature — nothing has reported yet.
One command. It builds the console if it is not built, starts the gateway and the control plane, and needs no GPU, no S3, and no network:
./scripts/dev.shThen open http://127.0.0.1:8080. Four pages: a chat playground, live fleet state, the model registry, and cluster inventory. To fill them with data:
./scripts/seed.shdev.sh --real does the same but pulls from huggingface.co instead of a
synthetic repository. Everything lands in .dev/, which is gitignored.
seed.sh fills the pages; it does not check anything. To check:
./scripts/smoke.sh # in a second terminal, with dev.sh running128 assertions, and it exits non-zero on the first failure. It is safe to re-run. What it covers, and why each check exists:
| Area | Checks |
|---|---|
| OpenAI compatibility | a real completion, a 54-frame stream, a usage object on both, [DONE], error envelopes instead of a router's 404 page |
| Console | the app shell, a deep link, and that each /assets/*.js is served as JavaScript and is not the app shell — a 200 that is really index.html is the failure that hides |
| CORS | a loopback origin is allowed and a foreign one is not, because the console is on :8080 and the control plane answers on :8081 |
| Engine profiles | vLLM demands compute 7.5+ and llama.cpp demands none; vLLM maps cache occupancy to a real series and publishes KV capacity, llama-cpp publishes queue signals but no occupancy signal and no capacity |
| Weight formats | a safetensors and a GGUF repository side by side, each passing its own engine and each refused by the other's, with the reason in the message |
| Operator inventory | counts recomputed from the node list, and Scheduling with a reason kept distinct from Pending |
| Authentication | health without a key, a valid key served, a missing and an unknown key both 401, and the 401 advertising WWW-Authenticate |
| Rate limiting | a per-minute request ceiling, a token ceiling that bites before the request ceiling, Retry-After on the refusal, one tenant's keys sharing a bucket, and another tenant staying unaffected |
| Metrics | /metrics needs a credential while /healthz does not, the exposition is the versioned text format, a served request is attributed to the tenant that spent it, tokens are split into the three sides that are billed at three rates, an embedding spends no completion tokens, histogram buckets never decrease, and no series is labelled with a key |
The last two rows run against a second gateway the script starts itself, on port 8099 and 8098. The main gateway on :8080 has authentication off — a developer's laptop must work with no setup — so it cannot check this surface, and leaving it unchecked is how a gateway ships that serves anyone's requests.
That last row is the one to read twice. A Ready model that cannot be loaded
is a claim the platform cannot back up, and the pull is where that claim is
made.
make check runs what CI runs and needs no server. It also compiles
fleet-serving when that repository sits beside this one: each repository's
own checks pass while the pair does not compile, which is how a deleted core
interface left the controller broken with both suites green. smoke.sh is
separate because it needs both processes up, so it cannot be part of
check.
Off by default, because a developer should be able to run the console with no setup. On in production, and the gateway refuses to start rather than serving everyone when the two halves of the configuration disagree:
export FLEET_AUTH_REQUIRED=true
export FLEET_API_KEYS='acme/research/team-a;acme/batch/worker'
export FLEET_RATE_TENANTS='acme|rpm=1200,tpm=400000'
export FLEET_RATE_PROJECTS='acme/research|rpm=600,tpm=200000;acme/batch|rpm=60,tpm=20000'
export FLEET_DEFAULT_MAX_TOKENS=4096 # what an unbounded request reserves againstA key is tenant/project/keyid, and that whole string is also the credential
the client sends as its bearer token. Keys carry no limits — a key rotates
and leaks, so a budget attached to one would have to be reissued along with it,
and lowering a budget would invalidate whatever the caller was using when they
were cut off.
Limits live on two levels, and both are checked on every request:
| Declared in | Bounds | |
|---|---|---|
| Envelope | FLEET_RATE_TENANTS |
everything the tenant spends, across all its projects |
| Partition | FLEET_RATE_PROJECTS |
one project, within its tenant's envelope |
They are two counters rather than one merged figure because a project limit on its own is arithmetic the tenant performs: ten projects at 600 rpm is 6000, and no per-project number would ever have bound. The envelope is what makes the partitions add up. A project limit above its tenant's envelope is refused at startup rather than clamped — it could never apply, and an operator who wrote one believes they have divided a budget they have not divided.
FLEET_RATE_RPM and FLEET_RATE_TPM are the server defaults for a tenant
nobody declared. A limit that says only rpm stays unlimited on the token
dimension — for a partition that means "no limit of its own here", which is
not the same as inheriting the tenant default.
Policies are read once per scope and cached, so changing a limit needs a gateway
restart. Three configurations are refused at startup — required with no keys,
keys without required, and a key spec that does not parse — because each of
them looks like a working gateway from the outside while serving nobody. A key
spec still carrying |rpm=… is also refused, with a message saying where limits
moved, rather than being silently accepted with the numbers dropped.
/healthz and /health stay unauthenticated, because a Kubernetes probe
cannot hold a credential and a 401 there reports a healthy pod as dead.
The env-var configuration above is for a gateway with no database. Connect one and tenants, keys and budgets become rows, created through the API:
export FLEET_DATABASE_URL='postgres://user:pass@host:5432/fleet?sslmode=disable'
./scripts/dev.sh # the control plane picks it up; the gateway is left aloneA=http://127.0.0.1:8081/api/v1
curl -X POST $A/tenants -H 'content-type: application/json' -d '{"id":"acme","name":"Acme","requestLimit":1200,"tokenLimit":400000}'
curl -X POST $A/projects -H 'content-type: application/json' -d '{"tenantId":"acme","name":"research","requestLimit":600,"tokenLimit":200000}'
# The secret is in this response and nowhere else: only its hash is stored.
curl -X POST $A/keys -H 'content-type: application/json' -d '{"projectId":"acme/research","label":"team-a"}'
# A budget is a dimension over a window, not one number. Several rules apply
# at once: 5M tokens per 5 hours AND 200 units per 30 days.
curl -X POST $A/budget-rules -H 'content-type: application/json' -d '{"scopeKind":"tenant","scopeId":"acme","dimension":"tokens_total","limit":5000000,"window":"5h"}'
curl -X POST $A/budget-rules -H 'content-type: application/json' -d '{"scopeKind":"tenant","scopeId":"acme","dimension":"units","limit":200,"window":"1mo"}'Windows roll: 1mo is 30 days, because a window that is sometimes 28 days is
not a budget anyone can reason about. The calendar month still exists, for the
cost pool in docs/architecture.md §6 — a different window, not a switch on
this one.
Dimensions are tokens_total, tokens_input, tokens_output, tokens_cached,
tokens_fresh and units. tokens_fresh is the one that matters on a
deployment with a warm prefix cache: total prompt tokens can be orders of
magnitude above what was actually charged, so a budget on the total stops a
tenant who has barely spent anything.
The whole API is plural, top-level and slash-tolerant — see
docs/architecture.md §11.6 for the three rules and why each one is there.
A tenant or project that has billed requests cannot be deleted; set
active: false instead, because usage refers to it.
Budgets are denominated in quota units because P8 says cost is fixed: an idle GPU burns money just as fast as a busy one, so what a request will cost cannot be known before the month closes. That leaves the question the platform exists to answer, which is what the period cost pool answers:
A=http://127.0.0.1:8081/api/v1
# What one GPU-hour costs you. Fleet cannot know this — a cloud bill, a
# colocation contract and a depreciated on-prem fleet have nothing in common.
curl -X POST $A/cost-rates -H 'content-type: application/json' -d '{"cluster":"k3s-dev","gpuHourMicro":3200000,"currency":"USD"}'
# Close the month. 3.20 USD per GPU-hour.
curl -X PUT "$A/cost-periods/2026-09"The report reads as a sentence:
pool |
what the fleet cost for the month |
busy |
the part some request occupied |
idle |
the difference — the number P8 exists for |
allocated |
the pool, split across tenants by what they consumed |
busy + idle = pool and allocated = pool, and both are true: idle capacity is
a fixed cost, so it has to land on somebody's invoice. Allocating only the busy
part would leave it unbilled and the operator would quietly absorb the exact
problem the platform is supposed to surface.
A closed period keeps its numbers. Closing it again recomputes it, and the difference is booked to the month after rather than rewritten into the month that was already sent — a correction discovered late still has to be collected somewhere, and that somewhere cannot be the invoice already issued. Once the following month is closed the earlier one is frozen and answers 409 naming the period that closed the door, because after that there is nowhere left for a correction to land.
That is the whole of Fleet's cross-period rule. There is no balance to roll forward and no debt to carry: a period apportions a pool that has already been paid for, and a request is reserved and accounted for together (P5), so nothing can be spent in one month and paid in another.
Fleet refuses to close a month it did not watch. The capacity time series comes
from the operator's own inventory reports, so on a fresh deployment a month is
0% observed and the close answers 400 with the coverage it actually computed
from. Publishing a pool of zero instead would be indistinguishable from a fleet
that cost nothing. ?minCoverage=0 forces it, loudly.
| Page | Backed by | Needs |
|---|---|---|
| Playground | an in-process stub engine (--demo) |
nothing |
| Fleet | the gateway's own request log | nothing |
| Models | fleet-apiserver + a directory as the object store |
dev.sh |
| Cluster | an operator inventory report | seed.sh, or a real operator |
The stub engine's replies are canned. The point is to exercise streaming,
usage accounting and routing, all of which behave the same against a real
engine. The stub honours stream_options.include_usage exactly as vLLM does,
including omitting the final usage frame when the client did not ask for
one, so the metering path is genuinely covered.
The synthetic hub serves a few kilobytes of plausible filenames for three repositories — including one that fails mid-pull and one slow enough to cancel. That is enough to exercise the whole pull path, progress reporting and cancellation, without anyone downloading 15 GiB to test a progress bar.
fleet-gateway --config fleet.yamllisten: ":8080"
upstreams:
- id: llama-7b
model: Qwen/Qwen2.5-7B-Instruct
base_url: http://127.0.0.1:8001 # llama-server, or a vLLM Service
replicas: 1
api_key: "" # the engine's key, not a tenant'sThe control plane uses a directory when FLEET_S3_ENDPOINT is unset, and any
S3-compatible endpoint when it is set — Ceph RGW, R2, Spaces, AWS S3, and
anything else that answers the S3 API:
export FLEET_S3_ENDPOINT=127.0.0.1:9000
export FLEET_S3_BUCKET=fleet-weights
export FLEET_S3_ACCESS_KEY=fleetadmin
export FLEET_S3_SECRET_KEY=fleet-secret-key-01
export FLEET_S3_USE_SSL=false
fleet-apiserver --listen :8081To try it locally, ./scripts/install-objectstore.sh stands up SeaweedFS in
WSL and ./scripts/s3-smoke.sh drives the whole path against it: a real pull
off huggingface.co, the objects counted back out, and the cancel path.
The FLEET_S3_ prefix deliberately shadows the AWS_ names: a process that
also talks to AWS must not pick up Fleet's bucket by accident.
make check # gofmt + go vet + no-cgo check + tests + console types + controller
make smoke # end-to-end assertions; needs ./scripts/dev.sh running
make build # host binaries into ./bin
make web # build the console and stage it for embedding
make dev # the same as ./scripts/dev.sh
make seed # fill the dev control plane with data
make web-dev # console dev server on :5173, proxying to a local gateway
make build-release # linux/amd64, linux/arm64, darwin/arm64
make help # all targetsRequires Go 1.26+ and Node 20+.
scripts/dev.sh, scripts/seed.sh and scripts/smoke.sh do not need make —
the Makefile targets are aliases so the sequences are discoverable.
make e2e is the one target that needs both repositories, because it is the
only test that goes from a CRD to an inference through the real gateway. Clone
the two side by side and the default works:
git clone https://github.com/zlogic-labs/fleet
git clone https://github.com/zlogic-labs/fleet-servingOtherwise point FLEET_SERVING_DIR at the fleet-serving checkout.
The console bundle is built by web/ and embedded into the gateway binary; it
is not committed. A fresh clone still compiles — the binary serves a
placeholder telling you to run make web — and build-release depends on it
so a release never ships a binary with the placeholder in it.
Fleet is pure Go. There is no cgo anywhere in the dependency graph and no
architecture-specific code, so every release binary is statically linked and
cross-compiles from any host. make check-cgo and the CI build matrix both
enforce this — a dependency that only builds on one architecture fails CI
rather than a customer's cluster.
| Target | Why |
|---|---|
linux/amd64 |
NVIDIA, and the mainstream domestic GPU cards |
linux/arm64 |
Ascend 910B on Kunpeng, Apple Silicon hosts |
darwin/arm64 |
local development |
If your
goreportsgo1.x windows/386, the toolchain is 32-bit. That is a workstation problem, not a Fleet one — install a 64-bit Go.go test -raceadditionally needs a C toolchain, because the race detector is cgo-based.
k3s under WSL2, for control-plane work only — see docs/architecture.md §9 for the WSL2-specific limitations (no GPU Operator, no MIG). GPU-specific behaviour is verified in CI on bare Linux.
Apache 2.0 — see LICENSE. Third-party dependency licenses are recorded in NOTICE.
Copyright 2026 Zlogic Labs.
Not open for outside contributions at this stage. Issues and PRs from early adopters are welcome once the first release is cut.