Skip to content

Fix cluster service port allocation races - #155

Merged
huangminghuang merged 2 commits into
masterfrom
fix/e2e-pubsub-port-race
Oct 27, 2026
Merged

huangminghuang merged 2 commits into
masterfrom
fix/e2e-pubsub-port-race

Conversation

@huangminghuang

@huangminghuang huangminghuang commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Allocate automatic service ports outside the host's actual client-port ranges, including complete Solana RPC/PubSub pairs and dynamic windows.
  • Probe explicit pinned ranges to their exact inclusive end and skip automatic ranges before applying the dynamic-window search limit.
  • Capture socket states and owners promptly on validator bind errors, retaining the snapshot in startup diagnostics.
  • Preserve the shared Node/Python bind registry and availability-checked persisted relaunches without requiring automatic-range observations for pinned bindings.

Why

This PR was motivated by failed E2E run 40113142471. Both attempts finished 13/14: attempt 1 failed emissions-soak when PubSub could not bind 0.0.0.0:34502; attempt 2 failed batch-operator-slashing on 0.0.0.0:34662. Each reported Address already in use while the validator continued producing slots until its 480-second readiness timeout.

The cross-language bind registry coordinates planned ports among cooperating processes, but get-port closes its probe sockets before returning. An unrelated outbound connection can then receive a planned service port as its automatic source port, causing the later daemon bind to fail with EADDRINUSE. The registry support now on master still permits this kernel-selected fallback; reserving PubSub as RPC + 1 alone cannot prevent it.

The historical runs did not capture the conflicting socket owner at bind time, so their particular holders are not proven. A separate local reproduction demonstrated kernel-assigned source-port reuse of a registry-reserved PubSub port and the resulting EADDRINUSE. This PR prevents that reproduced collision class and captures prompt bind-time diagnostics for future failures.

Automatic allocation now uses explicit candidates outside observed Linux, macOS, and Windows client-port ranges, with randomized fallback starting points. Unreadable host settings and exhausted safe pools fail automatic allocation. Fully pinned plans retain their availability checks without requiring host-range observations; operators must keep host settings stable while a cluster runs. Validator readiness continues to require both slot production and an acknowledged subscription.

Validation

  • 354 tests passed across 13 focused allocator, configuration, Python-registry interoperability, shared-schema, cluster-manager, network-probe, validator lifecycle/readiness/diagnostic, and process-orchestration suites on the merged candidate.
  • Persisted-plan regression confirms reclaim succeeds without reading automatic ranges and writes the shared registration shape.
  • Built-code smoke check: four concurrent plans, 324 distinct ports outside the host's observed automatic range; shared registration and persisted reclaim also passed.
  • Cluster-tool build and referenced workspace/test compilation passed; generated JSON Schemas match their sources; scoped ESLint, Prettier, and diff checks passed.
  • Host parsing is covered with injected Linux/macOS/Windows observations; native macOS/Windows execution has not run.
  • Historical E2E validation passed all 14 runnable flows exactly once, Release, concurrency 4, with no runnable flows excluded, on pre-merge Tools fd576232, Sysio b537bb50, Ethereum a76e7a0b, and Solana 2c7ba391. Resolved revisions, build artifacts and sibling SDK consumption were verified. This is not exact-head validation of the merge-resolution candidate.
  • Current-head E2E validation passed all 14 runnable flows exactly once in attempt 1, Release, concurrency 4, with no runnable flows excluded. Build/artifact checks and resolved Tools 737f4fe8, Sysio e34ccf9a, Ethereum a76e7a0b, Solana 2c7ba391 and Libraries 254badb6 were verified, including sibling SDK consumption and complete flow coverage.

@huangminghuang
huangminghuang marked this pull request as ready for review October 26, 2026 17:11
@heifner
heifner requested a review from valthon October 26, 2026 17:17

@valthon valthon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at 737f4fe. No correctness findings.

Coverage: read every changed line in all 10 files and traced allocation, registry locking, persisted reclaim, and validator readiness. Compared all seven of my other open PRs and the relevant recently merged registry, locking, and readiness changes. The shared Node/Python registration protocol and persisted relaunch guarantees remain intact.

Merge simulations: #122 has one additive export conflict in packages/cluster-tool/src/utils/index.ts; retain both assetUtils.js and portRangeUtils.js. #79 already conflicts with the base in 146 files, with no additional conflicting paths introduced here. #157, #152, #149, #136, and #80 merge cleanly. No opposing behavioral intent found.

Validation: 168/168 focused tests passed locally, including Python interoperability and WebSocket readiness. The initial eight readiness failures reproduced on base 58facaa with identical failing test names; the reused installation lacked the declared ws dependency, and supplying ws 8 in the temporary checkout resolved them. Exact-head CI passed 466 suites / 4,692 Jest tests plus package unit lanes. No live E2E or native macOS/Windows execution performed locally.

@huangminghuang
huangminghuang merged commit 5259e9e into master Oct 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants