serving: pay only what this validator measured (compute red-team round 1) - #1730
Merged
Conversation
added 8 commits
August 28, 2026 16:05
…othing was verified Two ways a validator paid for something it had not established. Pricing failed open. `serving_share` returned the whole 3.5% cap whenever pricing was unusable, so a failed alpha/TAO read or a missing `pricing.tao_usd` in the loadout handed one verified card every emission in the pool. It now pays nothing and recycles, and a network that has no price to read at all — testnet — opts back in through SERVING_PAY_CAP_WITHOUT_PRICING. So that a transient read does not move pay either, `serving_pricing` reuses the last usable reading for up to SERVING_PRICING_MAX_AGE_S. `round_rows` takes the same flag: what the DB reports is what will be paid. Credit assumed a perfect round. A miner whose requests were none of them verified — none drawn by the audit sample, or our own reference could not answer — took `credit = 1.0`, which pays full speed credit for a round this validator observed nothing in. It now freezes at the last credit measured for that hotkey, and only a hotkey never measured starts at 1.0.
Three ways a miner wrote its own "neutral" — a verdict that touches neither the window nor the speed credit, so a hotkey that had passed ten audits once was never judged again and kept paying out. The "budget spent" refusal was matched by substring on the status string the miner itself sends. Any miner could answer the two recognisable baseline prompts a round, refuse every user with that string, and be paid full credit while users got 502s. The validator now keeps its own ledger of the max_tokens it asked each miner for; a budget refusal is neutral only when that ledger is near the miner's per-tempo allowance, and never when this validator clears the stake floor (a staked caller has no budget to be refused on). Malformed miner data raised before the reference was called — a byte value of 256 in any logprob entry, a token id past any vocabulary — and the exception was filed under "reference hiccup". Those are now soft misses, as is a completion longer than the max_tokens asked for, and a reference that answers HTTP 4xx to the miner's own ids. Only a reference that cannot be reached, or fails on its side, is neutral. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
…ken by naming it The attestation judged one number it had computed itself — the digest — and took every other number from the miner's own report: how many cards, each card's wall, how much VRAM it filled, whether the model was resident. One 5090 could answer with N device entries, one real digest and N invented UUIDs, and be paid as N cards on every validator; the capacity probe that used to bound this was removed in #1721. A malformed field in one of those entries (`filled_bytes: "x"`) raised out of the judge and through the round, which zeroed every hotkey and blanked the gateway for the interval. A hotkey that named a competitor's GPU UUID failed both of them, every round. A reference sidecar keyed as the validator docs instruct returned 401 to a client that never sent a bearer, and that "neutral" kept paying each hotkey's last verdict indefinitely. Now every device index answers its own seed (seed + index) and gt_attest runs the cards in parallel, so the reference recomputes one digest per claimed index and the validator holds the whole reply to one card's budget plus a network slack: N real cards take one card's wall, one card faking N runs the chain N times. At most SERVING_ATTEST_MAX_CARDS are judged. A GPU UUID belongs to the first hotkey that passed with it; a later claimant loses the card and the holder keeps it, while two new claimants in one round still both fail (the sharing case the overlap exists for). A verdict the reference has not renewed within the memory window pays nothing. Casts are guarded, a fault anywhere in attestation is neutral for the cohort rather than fatal for the round, the reference sidecar gets its bearer (`attest.reference_api_key` / SERVING_ATTEST_REFERENCE_API_KEY), and the round counter and UUID ownership survive a restart. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
…t a time The attest hook charged nothing to a validator's per-tempo budget, so any hotkey holding a permit — on a small subnet that is nearly every UID — could send a miner an unlimited stream of challenges. Each one makes the sidecar allocate all of the card's free VRAM and run the chain under a global lock, so a stream of them stalls the runtime behind it and queues every real validator's challenge, which then reads as "too slow". The sidecar's own bearer was never sent by the miner either, so keying it broke the miner. An attestation now requires validator_trust > 0 (or the stake floor a gateway validator clears) and at most one challenge in flight per caller; the miner sends its sidecar's bearer (`attest.api_key` in the loadout, or SERVING_ATTEST_API_KEY). Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
…s the text that was verified Four ways a miner could serve less than the model's answer under a perfect transcript. It could stop after one correct token. The verifier forced only the ids the miner reported and compared only those positions; nothing asked whether the model would have stopped there, and a one-token answer also fell under the decode-pricing floor, so it earned full speed credit for a fiftieth of the work while the user got a fragment. Now an answer short of max_tokens must end with an end-of-turn token, which is forced and judged like any other position: the reference's argmax there must be the end-of-turn. It could make any wrong answer a miss instead of a strike. A wrong first token returned "diverged at first token" with hard=False, and omitting token ids sent the text through re-tokenization, where a length mismatch is always a miss. A first-token divergence on aligned positions is now a wrong answer, and a release on a real runtime must report ids. It could stream one text to the user and another through the logprob entries. With bytes present the verifier compared bytes to bytes and never to the completion, and without them it skipped the text compare whenever a replacement character appeared on either side. The decoded bytes must now read as the completion, where a run of replacement characters stands for one or more non-ASCII characters (a multibyte character split across chunks) and for nothing else. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
…pts, and a slow reader is not a slow card Three things a user with a gateway key could do to any miner's pay, plus one leak. The gateway bounded a prompt's shape but not its size. A prompt past the runtime's context made the miner's backend answer 400, the stream ended without [DONE], and the request went into the miner's window as a miss — always sampled, never checked against anything. Ten such prompts took a miner off READY; least-in-flight routing spread them over the whole pool. Prompts are now capped at SERVING_MAX_PROMPT_CHARS (413), and a request the runtime still rejects is put to the validator's own reference first: if the reference refuses it too, nobody is blamed. A request's latency was stamped when the client finished reading the stream, in the streaming response's finally. A client that drained slowly — deliberately or on a bad link — drove the miner's decode rate to zero and its speed credit with it. The latency is now the end of the miner's stream, taken where the stream is consumed. A streamed event the assembler could not fold (a JSON list where a dict was expected) raised past the point where the miner's in-flight slot was released, and the count carried across rounds: less traffic to that miner and a higher recorded in-flight on the traffic it did get, which reads as a lower expected decode rate. The slot is now released however the stream ends. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
… to agree with the reference State. Any exception in the audit round published an empty round: every settled hotkey recorded a zero and the READY set was cleared, so a metagraph read that failed or one malformed reply cost the whole fleet a twelfth of an hour and blanked the gateway. A failed round now settles nothing; the READY set stands until the TTL runs out. The last measured speed credit (what a round with nothing verified freezes at) now survives a restart, and SERVING_TAO_USD is gone: the TAO/USD rate is the loadout's, so validators price a card the same way. Strikes. Every band failure on aligned positions was a strike, so a validator whose reference drifted — a driver, an image — quarantined every honest miner for an hour and again each time the quarantine lifted. A strike now needs the reference to agree with most of the fleet that round: with two or more hotkeys judged and fewer than half passing anything, the reference is the odd one out and the failures are misses. And each further strike on the same (hotkey, release) quarantines four times longer than the last, up to 64 hours, so one wrong answer costs an hour and a pattern costs days. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
…design before it Stale claims mislead the next audit. The round-score formula in the audit loop's docstring still read "x attested (0/1)" (it is x attested cards), the cohort was "a random half" (least recently challenged first since #1725), the window was persisted to serving_audits.json (SQLite since #1718), the latency comment said same-region proxying was "caught by the concurrent capacity probe" (removed in #1721), the baseline module promised prompts a miner "cannot pick out of real traffic" and a mixing-in of user prompts that was never built, the miner neuron said there was "nothing to gain by editing it", and /v1/models described the runtime pin and model digest as what miners are "verified against" when nothing checks either string. probe_history was persisted, restored and never written since the capacity probe went. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
anderdc
changed the title
…; size the prompt cap from the measured prefill Measured on a rented 5090 (2026-08-28): sparkinfer 7498736 lists only the content tokens in logprobs on a natural stop — "Hi there!" comes back as three entries ending in "!", with no end-of-turn entry. The early-stop rule from the previous commit would have missed every honest short answer. /v1/score does score an appended token id and returns the argmax and bytes at that position, so the validator now appends the release's end_of_turn_token_id (151645 for Qwen3.6, in the loadout) to the miner's ids itself and requires the reference's argmax there to be end-of-turn; when the model would have continued, that is a wrong answer. A release without the id, or a miner without ids, still misses on an early stop. Prefill on the same card: TTFT 403 ms at 8.9k prompt tokens, 500 ms at ~11.5k, 1453 ms at 35.6k of a 36864 context. The gateway's prompt cap is 40k characters (~10k tokens), inside the full-credit TTFT band. Hostile completion_token_ids (1e9, -1, a 100k-long list, empty) all answer HTTP 400 in under two seconds with the server alive: a soft miss, as judged. Claude-Session: https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes for the compute sub-subnet red-team round of 2026-08-28, one commit per class of failure. Each commit body describes the failure mode in plain language; the audit with the full ranked list is in the private repo docs (
redteam/SECURITY_AUDIT_compute_round1.md).The shape they share: the validator recomputed exactly one number it did not receive from the miner (the attest digest), and every other number that became pay was the miner's — or, when malformed, turned into a neutral verdict, and a round with nothing judged paid full credit. A miner could earn one passing window honestly, make every later audit neutral, claim N cards, and be paid N× on every validator while users got 502s.
By commit
1. Pricing fails closed; credit freezes when nothing was verified.
serving_share()returned the whole 3.5 % cap whenever pricing was unusable — a transientsubtensor.subnet()failure or a missingpricing.tao_usdhanded one verified card the entire pool. Unpriced now pays nothing and recycles; testnet opts in withSERVING_PAY_CAP_WITHOUT_PRICING=true; the last usable reading carries forSERVING_PRICING_MAX_AGE_S(1 h);round_rows()reports the same. Andcredit = … if round_speeds else 1.0gave full speed credit to a miner none of whose requests were verified; it now freezes at the last measured credit (ServingState.last_credit), only a never-measured hotkey starts at 1.0.2. A neutral verdict needs a cause this validator can see. A "budget spent" refusal was matched by substring on the miner's status string, and malformed bytes/ids raised before the reference was asked and were filed as reference hiccups — neutral touches neither the window nor the speed credit. Now the validator keeps its own ledger of the
max_tokensit sent each miner; a budget refusal is neutral only when that ledger is near the allowance and never for a staked caller. Malformed miner data, over-length completions, and reference HTTP 4xx on the miner's own ids are soft misses.3. A card count the validator measures; a GPU that cannot be taken by naming it. Per-card attestation judged the digest against the reference and took
uuid/wall_ms/filled_bytes/residency from the miner, with one seed for every card: one 5090 could answer with N entries and be paid as N cards. Now device i answersseed + i, the reference recomputes one digest per claimed index,gt_attestruns cards in parallel, and the whole reply must arrive within one card's budget plusSERVING_ATTEST_RTT_SLACK_MS— N real cards take one card's wall, one card faking N takes N; at mostSERVING_ATTEST_MAX_CARDS. A malformed entry (filled_bytes: "x") used to raise through the round and zero everyone; casts are guarded and any attest fault is neutral for the cohort. A UUID belongs to the first hotkey that passed with it — a newcomer naming it loses the card, the holder keeps it; two new claimants in one round both fail (the sharing case). A verdict the reference has not renewed within the memory window pays nothing. The reference sidecar gets a bearer (attest.reference_api_key/SERVING_ATTEST_REFERENCE_API_KEY);attest_roundand UUID ownership persist.4. Attestation is for validating hotkeys, one challenge at a time. Any permit holder could send unlimited budget-free challenges that fill a competitor's VRAM under the sidecar's global lock, stalling its runtime and queueing every real validator's challenge into "too slow". Now
validator_trust > 0(or the stake floor), one in flight per caller; the miner sends its sidecar's bearer (attest.api_key/SERVING_ATTEST_API_KEY).5. An early stop is the model's own; the user's text is the verified text. One correct token per request was a full-credit answer (and fell under the decode-pricing floor); a wrong first token or omitted ids was always a soft miss, never a strike; the completion the user received was never compared with the verified bytes (and skipped on any U+FFFD). Now an answer short of
max_tokensmust end in an end-of-turn token that is forced and judged (the reference's argmax there must be end-of-turn), first-token divergence on aligned positions is a wrong answer, a real runtime must report ids, and the decoded bytes must read as the completion — replacement-character runs stand only for split non-ASCII characters.6. Gateway hygiene. No prompt-size bound, so a key holder could take any miner off READY with over-context prompts that were blamed without asking the reference; latency was stamped when the client finished reading, so a slow reader was a slow card; an SSE event the assembler could not fold leaked the miner's in-flight slot across rounds. Now
SERVING_MAX_PROMPT_CHARS(413), a runtime rejection is checked against the reference first, latency is the end of the miner's stream, and the slot is released however the stream ends.7. A failed round settles nothing; strikes need the fleet to agree. Any exception in the round zeroed every settled hotkey and blanked the gateway; a drifted reference struck every honest miner each time quarantine lifted. A failed round now publishes nothing (READY stands until the TTL),
last_creditpersists,SERVING_TAO_USDis gone (the rate is the loadout's), band failures are misses when fewer than half the judged hotkeys passed anything, and quarantine escalates 1 h → 4 → 16 → 64.8. Rot. Docstrings describing the pre-#1718/#1721/#1725 design and the dead
probe_history.9. Early stop verified by forcing the release's end-of-turn id; prompt cap sized from measured prefill. Measured on a rented 5090: sparkinfer 7498736 lists only the content tokens on a natural stop (no
<|im_end|>entry), so commit 5's rule would have missed every honest short answer./v1/scoredoes score an appended id, so the validator appendsend_of_turn_token_id(151645, in the loadout) itself and requires the reference's argmax there to be end-of-turn — hard when the model would have continued. Prompt cap is 40k chars (~10k tokens; TTFT 500 ms at ~11.5k of a 36864 context).Measured on a rented 5090 + RTX 6000 Ada (2026-08-28, ~$0.75)
completion_token_ids(1e9, −1, 100k-long, empty): HTTP 400 each, server alive — a soft miss under commit 2, never a reference DoS./v1/scorereturnsbytesandtoken_ids; scoring an appended end-of-turn id returns its argmax and bytes.vram_model_reserved_bytesat blessing under load.GET /infoanswers with no auth on the documented Lium layout: setATTEST_API_KEY(both sides send bearers now); docs to follow.entrius/gt-attestimage (per-device seeds, threads) on a two-card box. Until miners run it, every miner reports index 0 only, which still passes.Not in this PR (decisions, not code)
CI: ruff, format, vulture, 757 tests green; pyright's 40 errors are the pre-existing
bittensorimport resolution intests/, identical ontest.https://claude.ai/code/session_01WVxb9UNM7sK7HY5BYXUVrf