Keep startup polling failures isolated

[dmulcahey]

One unexpected startup device-refresh failure can end the supervising gather while sibling refreshes keep running, and skip re-enabling normal polling. Wait for every device outcome, report failures/cancellations per device, and restore allow_polling when the startup operation exits. Cancellation of the overall operation still cancels and awaits its children.

Following the review suggestion, remove ZHA's whole-device concurrency limit and express startup work as LOW-priority radio requests. Keep the ambient priority context and forward explicit LOW through aggregated attribute reads, so packet-priority consumers receive it too. Existing initialization callers retain their default behavior.

Tests cover sibling supervision, failure recovery, cancellation, eligible-device selection, and priority propagation. This is not a measured startup-speed improvement. Radio backends own request scheduling; backends that do not use zigpy's limiter no longer have a separate ZHA whole-device cap.

Validation against dev at 66603431339afe37fa0048b70ff31d77dceb8f95: Python 3.12 full suite, 1385 passed, coverage above the 95% project gate; full pre-commit (codespell, Ruff, formatting, mypy, lock check) passed. Relevant regression checks fail on the unchanged base. GitHub CI for Python 3.12/3.13/3.14 is reported separately on the PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.19%. Comparing base (6660343) to head (5e3fb6d).

Additional details and impacted files

@@           Coverage Diff           @@
##              dev     #841   +/-   ##
=======================================
  Coverage   97.19%   97.19%           
=======================================
  Files          57       57           
  Lines       10560    10568    +8     
=======================================
+ Hits        10264    10272    +8     
  Misses        296      296           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[puddly]

An alternative approach here would be to disable concurrency limiting entirely and make sure that device state polling is done with low priority:

async with self.request_priority(t.PacketPriority.LOW):
    # code to poll a device

Maybe we can add this as a kwarg to async_initialize or something?

Zigpy has a mechanism to control concurrency if a request is properly tagged with the above context manager.

[dmulcahey]

@puddly Thanks — I updated the PR to take this approach. ZHA no longer limits complete device-initialization pipelines during startup. It starts every eligible initializer as LOW-priority work and delegates actual radio-request scheduling to zigpy and the radio backend.

The revised call site is here. In abbreviated form, it is now:

startup_polling_priority = t.PacketPriority.LOW

async with self.request_priority(startup_polling_priority):
    initialization_results = await asyncio.gather(
        *(
            device.async_initialize(
                from_cache=False,
                request_priority=startup_polling_priority,
            )
            for device in online_devices
        ),
        return_exceptions=True,
    )

Why removing the ZHA-level limiter is the better boundary

The previous implementation inspected zigpy's private _concurrent_requests_semaphore.max_concurrency, subtracted four, and used that result to limit entire Device.async_initialize() calls. That mixed two different kinds of concurrency:

• device initialization also includes local discovery, cache processing, and entity work; and
• radio request concurrency is the scarce resource that actually needs scheduling.

Zigpy now owns the latter concern. Its request-priority context supplies the ambient priority, and its global request limiter resolves that context when a backend submits a request without an explicit priority. The limiter itself is a priority-aware cascading limiter, rather than a single undifferentiated semaphore.

With zigpy's default concurrency of 8 and 25% LOW / 50% NORMAL / 25% HIGH capacity fractions, the cumulative admission thresholds are 2 for LOW, 6 for LOW plus NORMAL, and 8 when HIGH is included. These are cascading priority thresholds, not a strict total-concurrency ceiling for every possible arrival ordering, but the important distinction is that startup traffic now enters the LOW tier instead of consuming an arbitrary number of whole-device slots.

For common configured values, the structural difference looks like this:

Configured concurrency	Old ZHA initializer cap (max(1, C - 4))	Zigpy LOW request tier (25%)
4	1 complete initializer	1 radio request
8	4 complete initializers	2 radio requests
12	8 complete initializers	3 radio requests

Those columns are intentionally not performance equivalents: the old limit covered the entire initialization pipeline, while the new tier governs individual radio requests. The latter is the abstraction we actually want. The old ZHA reservation heuristic came from ZHA #510; zigpy subsequently gained its priority-aware request limiting in zigpy #1635, so the scheduling responsibility can now live at the lower layer.

Important — zigpy-ziggurat needs explicit priority propagation, not only the context manager.

There is a subtle distinction between the priority used to acquire a local limiter and the priority stored on the outgoing packet:

1. The LOW ContextVar is inherited by the tasks created inside the async with block. Zigpy defines its default request context here.
2. If a cluster call leaves priority=None, zigpy.device.Device.request() resolves the ambient value for its per-device limiter, but still forwards the original None to ControllerApplication.request().
3. ControllerApplication.request() builds the packet with that value, so ambient LOW alone can still produce ZigbeePacket.priority is None.
4. The zigpy-ziggurat 1.0.1 send path does not re-resolve zigpy's ambient context. It serializes packet.priority, and deliberately converts None to 0 (NORMAL). The relevant SendAps priority field therefore receives NORMAL unless the packet was tagged explicitly.

That would make a context-only version look correct inside ZHA while Ziggurat still received the startup reads as normal interactive traffic. The revised implementation closes that gap:

• Device.async_initialize() now has an optional keyword-only request_priority.
• initialize_cluster_configs() adds the priority to both cache-permitted and fresh attribute reads when one is supplied. This matters because a cache-permitted read can still fall through to the radio when the value is absent.
• Existing callers pass no priority, and the helper omits the keyword entirely when it is None, preserving their current behavior.
• The ambient LOW context remains in place for nested request paths and backends that resolve it themselves.

PacketPriority.LOW is -1, while NORMAL is 0. On the server side, the current Ziggurat source carries the signed wire priority through its bridge conversion, maps -1 to UserLow and 0 to UserNormal in the Zigbee stack, and schedules transmissions through its priority heap when APS work is enqueued. I am describing the current server source separately here; the pinned zigpy-ziggurat client commit above does not imply that it pins this server revision.

Backend behavior

Radio library	How startup priority is handled after this change
bellows	Its send path uses zigpy's global priority-aware limiter. Ambient LOW is available, and the revised reads also carry explicit LOW.
zigpy-znp	Same global-limiter behavior.
zigpy-deconz	Same global-limiter behavior.
zigpy-ziggurat	The explicit argument is essential: it puts -1 on the packet so the server receives UserLow, rather than converting a missing value to NORMAL.
zigpy-zigate / zigpy-xbee	These send paths do not currently use zigpy's global priority limiter. We are intentionally not retaining a duplicate ZHA whole-device limiter solely for these backends.

The explicit propagation currently covers the radio-producing work in Device.async_initialize(): the aggregated attribute reads. The current entity-level async_initialize_cluster() override is local/cache-only. If a future entity hook adds network I/O, that new path should either accept the explicit priority as well or zigpy-ziggurat should resolve the ambient priority before serializing a packet. That is a future-proofing consideration, not a reason to broaden this focused change.

Concrete behavior on a larger network

Consider 50 recently seen mains-powered devices during startup:

• ZHA now starts all 50 initialization pipelines, so local discovery and cache work are not artificially serialized behind a radio-derived number.
• On bellows, ZNP, or deCONZ, each actual startup request enters zigpy's LOW tier. Normal and high-priority work retain their priority-aware capacity instead of waiting behind an undifferentiated ZHA batch.
• On Ziggurat, the startup attribute requests carry priority=-1; the server classifies them as UserLow, so normal interactive work can outrank queued startup transmissions.
• Starting all 50 initializers therefore does not mean that a priority-aware backend transmits 50 frames concurrently. Scheduling happens where the individual requests are visible.

F4 failure and cancellation semantics remain intact

Removing the outer concurrency helper does not undo the original corrective behavior:

• asyncio.gather(..., return_exceptions=True) retains one ordered outcome per device and waits for every sibling initializer, so one quirk or device failure cannot orphan the rest of the batch.
• A child that cancels independently is returned as that device's result and logged separately from an ordinary exception.
• Cancellation of the overall startup operation still propagates through gather, cancels unfinished child tasks, and is re-raised by the wrapper.
• allow_polling is restored in finally, so an unexpected device or orchestration failure cannot leave polling globally disabled for the rest of the gateway lifetime.

So I agree with the suggested direction and have changed the implementation accordingly: ZHA expresses startup polling as LOW-priority work, zigpy/backends own radio scheduling, and Ziggurat receives the priority explicitly on the packet rather than accidentally treating startup reads as NORMAL.

[zigpy-review-bot]

Approve. Nicely reasoned and thoroughly tested — verified the load-bearing pieces locally against zigpy 2.0.1 and CI is green across 3.12/3.13/3.14.

What I checked:

• request_priority is genuinely wired through, not dead code. Traced initialize_cluster_configs → cluster.read_attributes(priority=…) → read_attributes_raw → _read_attributes/general_command(**kwargs) → Cluster.request(priority=…) → the packet. When no priority is passed the packet's priority stays None (the ambient contextvar only feeds zigpy's per-request limiter fallback — it's never written onto the packet), and on-packet-priority backends like zigpy-ziggurat treat that None as NORMAL, so the explicit propagation is what actually puts LOW on the wire. test_initialize_request_priority locks this in.
• Dropping the radio_concurrency - 4 reservation is reasonable. For backends that route through zigpy's global request limiter (bellows/ZNP/deCONZ), RequestLimiter caps the LOW tier structurally — with the default 0.25/0.50/0.25 fractions and concurrency 8 the cumulative caps are LOW=2, LOW+NORMAL=6, +HIGH=8, so startup reads occupy at most 2 slots and interactive NORMAL/HIGH work retains reserved capacity (more than the old -4 left at default concurrency). Caveat: those fractions are user-configurable, and backends that don't call zigpy's limiter (zigpy-xbee/zigate) lose ZHA-level startup limiting entirely — which the PR calls out as an intentional trade-off rather than reinstating a duplicate whole-device limiter.
• Failure isolation and cancellation semantics are correct. gather(return_exceptions=True) awaits every sibling before returning, so one device's failure can no longer orphan the rest or skip the polling re-enable (the real pre-existing bug). zip(strict=True) over order-preserving results is sound, the isinstance(CancelledError)-before-Exception ordering is right (CancelledError is BaseException), and outer cancellation still propagates through the gather and re-raises — with finally: allow_polling = True now guaranteeing polling is restored on any exit. The three new gateway tests exercise exactly these paths.
• gather_with_limited_concurrency is still used in helpers.py; only the now-unused radio_concurrency property is removed, with no remaining references in zha (or ha-core's ZHA component).
• mypy clean on the changed files.