Skip to content

fix(runtime-host): escalate reconnect delay while a Host never stabilizes - #3462

Open
me2seeks wants to merge 1 commit into
apache:mainfrom
me2seeks:fix/3458-runtime-host-flap-oom
Open

fix(runtime-host): escalate reconnect delay while a Host never stabilizes#3462
me2seeks wants to merge 1 commit into
apache:mainfrom
me2seeks:fix/3458-runtime-host-flap-oom

Conversation

@me2seeks

@me2seeks me2seeks commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Fixes #3458

Problem

When a Runtime Host dies immediately after spawn (transport read_eof on every attempt), the reconnect lifecycle retried roughly every maxMs (5s by default) forever. Each retry re-ran the full desktop registration stack — new candidate process, IPC handlers, subscriptions, capability publisher bind. ~12 hours of this churn produced hundreds of candidate generations and ended in an Electron main-process OOM (heap pinned at ~3.6 GB), with the log dominated by repeated MCP capability alignment failed errors.

Change

RuntimeHostReconnectBackoff gains an optional unstableMaxMs ceiling (default 60s, validated >= maxMs). While the failure streak persists (no connection stays installed for stableConnectionMs, which already resets the streak), the jittered delay now keeps doubling past maxMs up to that ceiling instead of saturating at maxMs. As soon as a connection stabilizes, the ladder restarts from minMs.

  • Worst-case regeneration cadence during a persistent startup-death loop drops from ~every 5s to ~every 60s (~12x less churn of processes and desktop-side registration).
  • Callers that need the old flat ceiling can set unstableMaxMs equal to maxMs; callers with minMs: 0 are unaffected.
  • No behavior change for short transient outages: escalation only engages after the exponential ladder would have saturated anyway.

Testing

  • New tests in reconnecting-connection.test.ts: delay escalates past maxMs and saturates at unstableMaxMs; a stabilized connection restarts the ladder; unstableMaxMs < maxMs is rejected.
  • npm run test --workspace @maka/runtime-host: 1065/1065 pass; typecheck + biome lint clean.

Note: the full desktop workspace build could not run locally because this environment's node_modules is missing declared deps (@slack/*) — unrelated to this change; CI will cover it.

Follow-ups (not in this PR)

  • The retention root behind the OOM still needs a heap snapshot under reproduction; this change bounds the churn amplifying it.
  • Surface diagnostics when candidates die at startup so the flapping root cause (read_eof origin) is observable.

AI use

  • Generative tooling made a substantive contribution
  • No generative tool made a substantive contribution

Tool(s) and scope: Maka (AI coding agent) authored the implementation and tests; the diff was human-reviewed before push. Generated-by: Maka trailers are present on the branch commits.

@jackwener jackwener left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review of exact head 6e323981ddd625ca6eab0ee3aa70ad3be83ad677 against main@f1f4e71a9540a4da23159052c924fee72202e989.

Two actionable blockers:

  1. Existing valid maxMs configurations now throw. requireDelay allows maxMs through 120000, and before this change a caller could set (for example) { minMs: 100, maxMs: 90000 }. Lines 116–123 default unstableMaxMs to 60000 and then require it to be at least maxMs, so that previously valid call now rejects during construction even though the caller never opted into the new field. When unstableMaxMs is omitted, derive it as Math.max(DEFAULT_UNSTABLE_MAX_MS, this.#maxMs) (or otherwise preserve the old valid range), and add the missing compatibility test.

  2. This does not fix the issue it closes. #3458 requires that prolonged flapping not exhaust the main-process heap and calls for backoff escalation / circuit-breaking instead of unbounded regeneration. This patch only changes the unbounded loop from roughly once per 5 seconds to once per 60 seconds; if the observed cross-generation retention remains, it delays the same OOM rather than bounding it. The PR itself says the retention root still needs investigation. Either re-scope this as a mitigation (Refs #3458, keep #3458 open) or add a real bounded/circuit-breaking ownership rule or fix the retained root before using Fixes #3458.

Required conclusions:

  1. Optimal for the actual problem: no for the OOM claim; it is a useful churn mitigation, not the root fix.
  2. Production code to delete: none identified.
  3. Tests to delete/replace: none; add the omitted-unstableMaxMs / large-maxMs compatibility case.
  4. Deeper refactor: not required if the PR is honestly narrowed to mitigation; the retention/circuit-break owner still needs a separate completed fix.
  5. Ready to merge: no on this head; both points above need resolution, required test has not run, and the PR also omits the CONTRIBUTING-required AI-use declaration (state explicitly “none” if none was used).
  6. Residual risks: the flapping root and retained heap owner remain unknown; the new cadence still retries forever.

This changes exported reconnect behavior and user-visible recovery timing. Independent human review is required; this automated review is not approval.

…izes

A Host that dies immediately after spawn previously forced a regeneration
attempt roughly every maxMs (5s by default) forever. Hours of this churn
re-ran the full desktop registration stack thousands of times and ended in
a main-process OOM (apache#3458).

Once the failure streak persists - no connection stayed installed for
stableConnectionMs - the jittered delay now keeps doubling past maxMs up to
an absolute ceiling (unstableMaxMs, default 60s, must be >= maxMs), and the
ladder restarts as soon as a connection stabilizes again. Callers that need
the old flat ceiling can set unstableMaxMs equal to maxMs.

Generated-by: Maka
@me2seeks
me2seeks force-pushed the fix/3458-runtime-host-flap-oom branch from 6e32398 to 082085c Compare August 22, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Desktop main process OOM after prolonged Runtime Host connection flapping (repeated MCP capability alignment failures)

2 participants