refactor: non-blocking remote job dispatch (HARN-4)

[obvious-autobuild]

Human author: Tapan Ganatma Nakkina ([email protected])

Why

A remote compute job used to pin one of FastAPI's shared request-threadpool tokens for its entire life. SshProvider.result() polls every 0.25 s with time.sleep for up to 1800 s (ssh.py:50,323-348), and every compute router was a sync def — so a few concurrent traces could hold the ~40-token pool anyio gives all sync endpoints and starve /health (and /db/search, /simulate, ...) until the traces finished. HARN-4's fix: async routes that await compute on a dedicated bounded pool, leaving the shared pool to the endpoints that only need it for milliseconds.

What

• status.run_compute_async(fn) + status.run_task_async(task, inputs) — the dispatch seam. Bodies run on a plain loop-agnostic ThreadPoolExecutor (COMPUTE_SLOTS = 8, module-level in status.py) via loop.run_in_executor. Chosen over an anyio CapacityLimiter because anyio limiters bind to the first event loop that touches them — unsafe across uvicorn/TestClient-portal loops — while threads and queues need no loop. Extra requests queue on the pool's own queue; they never consume shared tokens.
• Routes converted by extraction, not rewrite: each route keeps its response contract; the body moved to a named sync function (_replay_psi_slice, _replay_fieldlines_slice, _replay_bundles, _virtual_shot_body) and the route is now a two-line async shell awaiting it on the pool. Converted: /replay/{id}, /replay/{id}/psi/{i}, /replay/{id}/fieldlines/{i}, /eq_surrogate, /surrogate, /export/{id}, /export, POST /compute/benchmark, POST /virtual/{id}.
• Deliberately still sync (bounded, never long-waiting): /compute (TTL-cached, single-flight 10 s probe cap), /compute/cancel (worker POST), POST /replay/{id}/usd (process-lifetime single-flight; sub-second work), /health, /db*.
• Preserved exactly: provider selection, the depth-1 queue, local-fallback semantics (RemoteComputeError → local re-run → fallback banner state), provenance labels, and HARN-1's progress SSE + HARN-2's cancel routes — they observe the same job records run_task registers, from whichever thread it runs on. A cancelled HTTP request cannot stop a running body (a thread can't be interrupted); it completes and its caches/provenance land — the same semantics the abandoned sync-router thread had.
• Blast radius handled: agent_tools.py called replay_shot/virtual_shot directly as sync code; it now imports the sync bodies (_replay_bundle, _virtual_shot_body). ssh.py's module docstring no longer claims sync routers.
• Trade-off / rejected alternative: converting the providers to async (asyncio.to_thread inside run_task) was rejected — it rewrites the protocol contract HARN-1/HARN-2 just stabilized and risks the depth-1 + fallback invariants this PR must preserve exactly. run_task itself is untouched; the dispatch wraps it.

How to Review

• Start at fusionlab/compute/status.py (the dispatch section) — everything else is a shell + extraction.
• The concurrency test (tests/test_nonblocking_compute.py::test_health_stays_responsive_while_replays_wait) is discriminating: with the shared limiter shrunk to 2 tokens it fails against the old sync dispatch (verified by stashing the fix) and passes here.
• The new file also closes the audit's coverage gap 1: router-level remote-provider tests, including the fallback labels a real route serves (learned_computed_on is None + recorded reason + local provenance).
• tests/test_api.py::test_concurrent_requests_share_one_fetch updated consciously: it called the route function directly; it now calls the body (_replay_bundle) the async route shells to — same three-threads-one-fetch assertion.
• No frontend changes: web/replay.js is untouched (its polling and SSE consumption are unchanged), so substring tripwires hold.

Test Evidence

• Full suite on the rebased head: 435 passed, 1 skipped (~74 s); uv run ruff check and uv run mypy fusionlab clean.
• 8 new tests in tests/test_nonblocking_compute.py: run_task_async remote-success + fallback semantics; router-level remote coverage (remote answer served with ssh provenance; replay bundle labels remote host; dying remote → local bundle with honest labels); the /health concurrency property; HARN-2 cancel and HARN-1 progress SSE through the awaited dispatch.
• Backend-only refactor — visual evidence does not apply: no frontend or rendered output changes (zero lines in web/).

Closes #42

🔗 Obvious Project · 🧵 Obvious Thread · 🔀 Obvious PR Artifact