Extract per-device state out of RequestHandler (#11)#74
Open
ff225 wants to merge 2 commits into
Open
Conversation
The committed lock listed `google-cloud-firestore` in the `analysis` extra
without a version/source, while the package is forked across two versions by
platform marker. uv 0.11+ refuses to parse it:
Dependency `google-cloud-firestore` has missing `source` field
but has more than one matching package
Every uv command failed on this, so the environment could not be synced at all.
Regenerated with `uv lock` (224 packages resolved).
RequestHandler kept every piece of runtime state in class-level dicts mutated from both the ASGI thread and the background I/O thread, with no locking. Introduce DeviceStateManager, which owns the device profiles, the EMA tables, the variance detectors, the adaptive-risk state and the layer-sizes cache behind a single RLock. RequestHandler.device_profiles stays as a live view of the manager's mapping, since http_server and the dashboard read it directly. The variance detector is now scoped per device: it used to be a single class-level instance, so heterogeneous devices reporting the same layer indices polluted each other's variance statistics and skewed offloading decisions. Also drop a redundant _load_stats() call in the inference-cycle telemetry path that recomputed layer sizes already available in scope, behind a hasattr() guard that was always true. First step of #45; unblocked now that #9 has landed.
This was referenced Jul 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11. First of three steps toward #45; unblocked now that #9 has landed.
Why
RequestHandlerkept every piece of runtime state in class-level dictionaries —device_profiles,variance_detector,offloading_states,_layer_sizes_cache— mutated from both the ASGI thread and the background I/O thread with no locking.What changed
New
src/server/communication/device_state.py.DeviceStateManagerowns the device profiles, the EMA tables, the variance detectors, the adaptive-risk state and the layer-sizes cache, all behind oneRLock.Per-device variance detectors. The detector used to be a single class-level instance, so heterogeneous devices reporting the same layer indices polluted each other's variance statistics and skewed offloading decisions. Each device now has its own.
Thread-safety. The debug-JSON snapshot is now taken under the lock before being handed to the background writer; it previously serialized a dict that the ASGI thread could be mutating.
Backwards compatibility.
RequestHandler.device_profilesremains a live view of the manager's mapping, sincehttp_server.py:233and the dashboard read it directly as a plain dict. No caller outsiderequest_handler.pychanges.Drive-by cleanup. The inference-cycle telemetry path recomputed
layer_sizesvia_load_stats()— data already in scope — behind ahasattr()guard that was always true.uv.lock
The first commit regenerates
uv.lock, which was unparsable by uv 0.11+ and blocked everyuvcommand, including CI's:The package is forked across two versions by platform marker; the
analysisextra listed it without a version/source to disambiguate. Regenerated withuv lock(224 packages resolved). Included here because nothing else could be verified without it.Testing
244 passed, 1 skipped (fast suite + HTTP end-to-end).
device_state.pyat 100% coverage; newtests/unit/test_device_state.pycovers the EMA maths, per-device isolation, snapshot copying, and an 8-thread concurrency test. ruff clean on all touched files.Notes for the reviewer
_load_statsis gone rather than kept as a deprecated shim: it is private and had no remaining callers.VarianceDetector). This PR preserves both behaviours exactly; they are fixed in the follow-up.