Investigate intermittent TCP stalls with 5,000 active loopback connections on macOS #61

Status

Deferred by maintainer decision. Stop further connection-load experiments, packet captures and dependency comparisons for now; this investigation must not block the remaining framework work.

Problem

Some connections intermittently stop completing requests in a 5,000-active-connection macOS loopback workload. The native Asio control also reproduces the symptom. This does not establish a cause in the workload, Asio, arknet or the OS TCP stack.

Reproduction context

• macOS 27.0.1 (26A434), arm64, Darwin 27.0.0; AppleClang 21, C++20 Release.
• Boost 1.90.0 / Boost.Asio 1.38.0; TLS off.
• Tool: benchmarks/asio_connections.cpp, source 66289bbfb34cc1c68ea438fd572fbe3cb675acac (diagnostic change in fix(benchmarks)!: preserve TCP kernel sample onset #60).
• One IO context / one IO worker per process; 5,000 loopback TCP connections.
• 1,024-byte application payload, four outstanding requests per connection; 4,096-byte initial/max receive buffer.
• Connection rate 1,000/s, at most 32 concurrent connects; traffic hold duration 3 s; request/connect timeout 5 s, workload deadline 40 s, cleanup allowance 5 s.
• Limits: 20,000 in-flight requests, 64 MiB in-flight bytes, 512 MiB buffer budget. Operation trace, socket-operation trace and TCP kernel samples were all enabled.
• Process descriptor soft limit was 1,048,575. All 5,000 connections were established/accepted in each run; no system-limit tuning was performed.

Completed observations

A fixed three-run campaign retained all outcomes: 5 unfinished connections, all connections completed, then 3 unfinished connections. These observations are not a failure-rate or capacity estimate.

Eight failing peer pairs matched by identity and reversed socket endpoints. Schema 2 retained 21 server queries before the corresponding last client query, addressing the earlier loss of pre-cancellation server history.

• Retained client queries: established connection, loss-recovery flag, congestion window 0, reported retransmission timeout 0, 4,364 bytes in the TCP send buffer, positive peer receive window.
• Corresponding server queries before cancellation: established connection, empty TCP send buffer, positive congestion window.
• Completed client write bytes minus completed server read bytes: 4,364 bytes, equal to four 1,091-byte framed requests.
• Both processes completed cleanup. These counters do not establish where bytes were lost/stalled or why progress stopped.

Earlier schema 1 tail-only evidence and the new schema 2 first-four/latest-four evidence remain distinct. Raw JSON, packet captures and logs remain local and are not release assets or public downloads.

Resume later

• Review existing local evidence and select a concrete new discriminator before running further workloads.
• Use same-connection, pre-cancellation observations; distinguish workload, reactor and kernel progress without assuming a root cause.
• Preserve fixed run counts, source/binary identities and failures; do not retry to green or increase timeouts to hide the problem.
• Obtain cross-host/Linux acceptance when an appropriate test environment is available.

Related diagnostic changes: #58, #59, #60. No fix or 5,000-active-connection production-capacity claim is made.