serving: spread gateway traffic instead of sending it all to the top-scored miner - #1733
Merged
Conversation
…scored miner Ties on in-flight were broken by score, and at one request per 20 s against sub-second completions nothing overlaps, so in-flight is always 0 and every request went to the same miner. Measured in soak 7 with two healthy 5090s: UID 16 took 12 requests to UID 27's 3, and the loading pushed its own TTFT to 1030 ms against 461 ms, cutting its credit to 0.68 while UID 27 sat at 0.945 on almost no traffic. The miner doing the work was paid less for doing it, and the idle one was barely measured. Among the miners with the fewest in flight the choice is now random, weighted by score: a better miner is still sent more, and neither is starved. The existing test asserted the deterministic pick and passed only because the default RNG happened to agree.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Measured in soak 7, on two healthy 5090s.
ServingState.acquirepickedmin(inflight, -score). At the soak's rate — one request per 20 s against sub-second completions — requests never overlap, so in-flight is always 0 and the tie-break decides every single request. All of it went to one miner:Both cards are healthy and identical. The concentration is what raised UID 16's TTFT, and the TTFT band is what cut its pay — the miner doing the work was paid less for doing it, while the idle miner was barely measured at all. Left alone it oscillates: the loaded miner's score decays until it drops below the other, and then all the traffic swaps over.
Among the miners with the fewest in flight, the choice is now random weighted by score — a better miner is still sent more, neither is starved, and load no longer piles onto whoever is currently winning.
_rngis a field onServingStateso tests are deterministic.test_state_least_inflight_dispatchasserted the deterministic pick; it passed only because the default RNG happened to agree with it, so it was flaky as written. It now pins the part that is actually contractual (fewest in-flight wins outright) plus two distribution tests: weighted when scores differ, uniform when they do not.This is the "weighted-random routing tie-break" item from
14-handoff§5, with the soak numbers behind it. CI green (ruff, format, pyright, vulture, 760 tests).