Repository navigation
feat(compute): opt-in shared bearer token for the worker - #70
Merged
Merged
Conversation
FUSIONLAB_WORKER_TOKEN set on both ends locks the worker's whole /v1
surface: one FastAPI middleware demands Authorization: Bearer (constant-
time hmac.compare_digest) and answers 401 + WWW-Authenticate before any
body parsing or queue interaction, while /health stays open so the
tunnel probe and autostart keep working. Unset — the default — the
worker is byte-for-byte today's behavior.
Provider side: the token rides all three httpx clients in compute/ssh.py
(long-lived client, SSE stream client, health probe — 9 call sites), and
any 401 maps through one shared helper to RemoteComputeError("auth", …)
whose message the fallback banner renders: the two ends disagree about
the token, not "unreachable". The handshake advertises the auth mode.
ssh_options() gains a loud guard: a credential-shaped key in
fusionlab.toml is a ValueError naming FUSIONLAB_WORKER_TOKEN — the
no-secrets-in-config rule is enforced, not just documented. The worker
docstring gains its missing 8th route line (GET /v1/jobs/{id}).
Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
- unset-token no-op pin: /v1 answers unauthenticated requests exactly as before the token existed - 401 matrix over all seven /v1 routes (missing and wrong Bearer), each asserting 401 + WWW-Authenticate arrives before routing and validation — ghost-id and valid-body rows would answer 404/201/422 if the gate ever leaked past the middleware - a rejected submit leaves the depth-1 queue empty; /health stays open - loopback end-to-end with the token set: handshake, weight upload, submit, SSE stream, result and cooperative cancel all green through the real worker; a mismatched env re-point asserts the actionable "auth" error and its banner wording - config guard: a token/worker_token key in fusionlab.toml (either spelling, base or host profile) is a loud ValueError, and FUSIONLAB_WORKER_TOKEN never rides ssh_options() Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
.env.example scopes its no-secrets comment to SSH credentials and gains the commented FUSIONLAB_WORKER_TOKEN line (same home as ANTHROPIC_API_KEY); fusionlab.toml.example notes the token is env-only and a credential key there is a loud error. docs/COMPUTE.md covers the token in the security list and drops auth from "Not in this version" (multi-user accounts stay out — one shared secret is not an account system); README's SSH section gains the two-line when-to-set-it note. Co-authored-by: Tapan Ganatma Nakkina <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Today the remote worker trusts every byte that reaches it through the SSH tunnel: all 8 of its
routes are unauthenticated, by a documented design that assumes the worker is loopback-only. Once a
worker host is shared, reachable from a lab network, or simply misconfigured, the only thing
between a stranger and free GPU compute is the tunnel's existence. The backlog spec (BACKLOG-2,
issue #45) adds an opt-in shared token that changes nothing until an operator sets one — and when
the two ends disagree, says so plainly instead of reporting a generic "unreachable".
Fixes #45
What
Opt-in by absence. With
FUSIONLAB_WORKER_TOKENunset (the default), behavior is byte-for-bytetoday's: no middleware effect, loopback-first, every existing test untouched, pinned by a new
no-op test. With it set on both ends:
fusionlab/worker.py): one@app.middleware("http")gate — every/v1/*routedemands
Authorization: Bearer <token>, compared withhmac.compare_digest(constant time),answered
401+WWW-Authenticate: Bearerbefore any body parsing or queue interaction.GET /healthstays outside the check: a token-gated probe would be misread as "worker absent"and trigger a doomed autostart. All 8 route decorators untouched.
GET /v1/capabilitiesadvertises"auth": "token" | "open"so both sides can seea disagreement coming.
fusionlab/compute/ssh.py): the header rides all three httpx clients — thelong-lived client, the SSE stream client, the health-probe client (9 call sites). Any 401 maps
through one shared helper (
_raise_for_auth) toRemoteComputeError("auth", …)whose messagethe status layer already renders verbatim as the fallback banner: "worker rejected the token —
FUSIONLAB_WORKER_TOKEN differs between this machine and the worker host (or is set on one side
only)". No router changes; the local fallback still answers.
ssh_options()gains a loud guard — a credential-shaped key(
token/worker_token/password/secret/api_key) infusionlab.tomlraises aValueErrornaming
FUSIONLAB_WORKER_TOKEN. This enforces, rather than re-states, the twice-documentedno-secrets-in-config rule.
.env.example(the token's documented home, with the no-secrets comment scoped to SSHcredentials),
fusionlab.toml.example,docs/COMPUTE.md(token covered in the security list;"auth" removed from "Not in this version" — per-user accounts stay out), README's SSH section,
and the worker module docstring, which gains its missing 8th route line (
GET /v1/jobs/{id}).Rejected alternative: a token field in
fusionlab.toml— it would contradict the standingno-secrets rule (
test_example_config_parses_and_holds_no_credentials); per-host tokens, if a realuser appears, use an env-var indirection key, not a secret in the file.
How to Review
worker.py(right after the appobject) and
_worker_auth_headers/_raise_for_authincompute/ssh.py./healthexemption is deliberate and tested — see the middleware docstring for theautostart-probe reasoning.
stream()builds a freshAsyncClientper call, so its 401 test injects the mock transport atthat construction site (same patch the probe test uses).
Test Evidence
uv run ruff checkanduv run mypy fusionlabclean; pre-push hook re-ran the suite on push./v1routes asserting 401 precedes routing/validation (ghost-id and valid-body rows would answer
404/201/422 if the gate leaked); rejected submit leaves the depth-1 queue empty;
/healthopenwith the gate on; loopback end-to-end with the token (handshake, upload, submit, stream, result,
cancel) through the real worker; mismatched-env test asserting the
autherror and bannerwording; config guard tests for both toml spellings and for
FUSIONLAB_WORKER_TOKENneverriding
ssh_options().the fallback banner and is asserted in tests, so visual dogfood evidence does not apply.
Human author: Tapan Ganatma Nakkina ([email protected])
🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact