Health monitoring: detect and report asyncio event-loop starvation

[majkelx]

Motivation

The 2026-08-05 ofp incident (#18, see forensic comment) took real investigation to diagnose: JetStream publish "timeouts" that were actually a client-side frozen event loop — both messages were delivered, only the acks were processed too late (5 s timeout fired after 11 s and 54 s).

A loop-lag detector in serverish would have turned that investigation into a single log line.

Proposal

A lightweight monitor task (opt-in or on by default at low cost) that:

• schedules a heartbeat coroutine every interval (e.g. 1 s) and measures scheduling delay;
• when the delay exceeds a threshold (e.g. 1 s), logs a warning with the measured stall duration: event loop stalled for 17.3 s — timeout-based operations (JetStream acks, RPC) may misfire;
• exposes the max/recent lag via the existing HasStatuses health-check machinery (Messenger status check, e.g. loop_lag), so services and dashboards can see chronic starvation;
• optionally annotates MessengerPublishAckTimeout with the recent lag measurement ("loop was stalled 17 s during this publish"), directly pointing at the root cause.

Notes

• Publisher-side mitigations (Nats-Msg-Id dedup + ack retries + MessengerPublishAckTimeout) are being added on fix/fetch-available-eager-finish; this issue covers the diagnosis layer only.
• Related milestone: "Subscription Reliability & Testing" — client-side freezes are an explicit scenario there.

🤖 Generated with Claude Code