Skip to content

fix(runtime-host): let operators raise the election deadline and clarify timeout copy - #3480

Merged
Astro-Han merged 1 commit into
apache:mainfrom
me2seeks:fix/3474-startup-deadline-and-orphan-candidates
Aug 22, 2026
Merged

fix(runtime-host): let operators raise the election deadline and clarify timeout copy#3480
Astro-Han merged 1 commit into
apache:mainfrom
me2seeks:fix/3474-startup-deadline-and-orphan-candidates

Conversation

@me2seeks

Copy link
Copy Markdown
Contributor

Closes #3474

Problem

On a large workspace (~788 MB runtime.sqlite), the first Runtime Host start after an upgrade emitted runtime_host_ready after ~75s — 30s after the client's fixed 45s election window had already elapsed. The user sees:

Error: Runtime Host stopped responding during startup

which is misleading: the host was alive and making progress the whole time. The candidates spawned during the election keep starting detached, so they eventually win and hold the root while every later attempt starts from a cold election again.

Change

  • MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS now overrides the default election deadline at its single resolution point in connectOrSpawnRuntimeHostWithDependencies, so CLI, desktop, and owned launches all honor it. The window stays bounded (1–120000 ms). Invalid values fail closed with a `RangeError` naming the variable — a silently ignored typo would leave the operator believing they widened the window when they did not.
  • The two deadline-elapsed failure copies (startup_timeout, host_unresponsive) no longer claim things neither reason can know ('stopped responding'). They describe what was observed, name slow first starts on large workspaces as the likely cause, and point at the override.
  • No candidate lifecycle change: ephemeral candidates already self-terminate via their initial connection deadline once startup settles; the lingering processes in Post-upgrade cold start can exceed the 45s election deadline, and abandoned candidates then hold the root lock #3474 were startup work starving the timer, not missing exit logic.

Testing

  • New election-deadline-env.test.ts: blank/unset handling, valid parsing, fail-closed on invalid values, and an ordering test proving an invalid override fails before storage is touched.
  • startup-error.test.ts updated for the new copy plus a knob-mention assertion.
  • Desktop manager assertions updated to the new copy.
  • packages/runtime-host: full suite 1067/1067 green; desktop manager suite 22/22; tsc --noEmit and biome clean.

Generated-by: Maka

…ify timeout copy (apache#3474)

A workspace whose first post-upgrade start takes longer than the fixed
45s election window fails the client with 'Runtime Host stopped
responding during startup', and every candidate spawned along the way
keeps starting in the background. Observed on a ~788MB runtime.sqlite:
the host emitted runtime_host_ready after ~75s, 30s after the client had
already given up.

The election deadline was already an interface parameter, but nothing
let an operator widen it. MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS now
overrides the default at the single resolution point, so CLI, desktop,
and owned launches all honor it. Invalid values fail closed instead of
being silently ignored - a typo would otherwise leave the operator
believing they widened the window.

The two deadline-elapsed reasons also claimed things neither could know
('stopped responding'); both now describe what was observed, name slow
first starts on large workspaces as the likely cause, and point at the
override. Candidates already self-terminate through their initial
connection deadline once startup settles, so no lifecycle change beyond
copy is needed there.

Fixes apache#3474

Generated-by: Maka

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the bounded range and the fail-closed validation are the right shape for an operational override, and keeping it as one more input to the existing deadline (rather than a second timer) means there's still a single election authority.

What this solves / how: the fixed 45s election window is too short for real installs — #3474's 788MB workspace takes ~75s to reach ready after an upgrade, through cold start, WAL and one-time migration work. The client can't estimate that before root authority initialises, so it can't self-tune. MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS resolves at the single existing resolution point, defaults unchanged at 45s, and is clamped to 1–120000ms.

On whether this belongs as a knob at all, since that's the question worth asking of any new tunable: yes, for the case you're targeting. Startup cost genuinely varies with workspace size, disk cache state and pending upgrade work, and none of that is knowable client-side before the host comes up. A bounded escape hatch is the honest answer, and the closed interval keeps it from becoming "wait forever."

Two findings inline, both about diagnosis rather than the mechanism.


*AI-assisted review. Reviewed independently by two reviewers; the severity of the first finding was revised downward after checking it against a second real-world failure sample.

const parsed = Number(rawValue);
if (!Number.isSafeInteger(parsed) || parsed <= 0 || parsed > 120_000) {
throw new RangeError(
`${ELECTION_DEADLINE_MS_ENV_VAR} must be an integer between 1 and 120000 milliseconds`,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] The error text points every startup failure at this variable, including the class it can't fix.

There's a second failure shape this knob doesn't address. From a real CI run on #3454, which now emits election diagnostics:

{"deadlineMs":45000,"elapsedMs":45174,"candidateLaunches":69,
 "observations":{"notRegistered":73,"connectFailed":0,"handshakeFailed":0,
                 "connected":1,"readyWaitFailed":1}}

69 candidate launches in 45 seconds. Zero connect failures, zero handshake failures — it connected once, then ready-wait failed, and candidates kept being launched roughly every 0.65s until the deadline. It missed by 174ms, so raising the deadline would have made that particular run pass, but the run isn't slow — it isn't converging. At 90s it launches ~138 candidates instead of ~69 and passing becomes a matter of luck.

That's different from #3474's genuinely-slow start, where waiting longer is exactly right. The problem is that both produce the same message, so an operator hitting the retry-storm shape is told to raise the timeout, raises it, fails again, and raises it further.

Not blocking, and possibly not yours to fix here — but worth asking: with candidateLaunches high and connectFailed/handshakeFailed at zero, should the message distinguish "this start is slow, wait longer" from "candidates aren't converging, the timeout isn't your problem"? You have the #3474 case in hand and probably know which shape operators hit more often.

const deadlineMs = input.electionDeadlineMs ?? DEFAULT_ELECTION_DEADLINE_MS;
const deadlineMs =
input.electionDeadlineMs ??
electionDeadlineMsFromEnvironment(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] On the owned-launch path an invalid value loses the variable name and surfaces as host_unresponsive.

electionDeadlineMsFromEnvironment() throws a RangeError naming the variable, which is exactly right for direct connectOrSpawnRuntimeHost. But connectOwnedRuntimeHostWithDependencies() wraps the call in a broad catch that converts anything into { kind: 'failed', reason: 'host_unresponsive' }. So MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS=abc reaches the user as Runtime Host did not start: host_unresponsive — they're told the host is unresponsive when in fact they made a typo, and the message gives them nothing to act on.

This is the same shape as the finding above: the mechanism is fine, the diagnosis points the wrong way. Config errors specifically are worth letting through that catch (or resolving before it, as a typed configuration failure), since they're the one class the user can fix immediately once they know the variable name. Worth a test on both the owned and hosted paths.

@Astro-Han
Astro-Han merged commit ce8b07f into apache:main Aug 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Post-upgrade cold start can exceed the 45s election deadline, and abandoned candidates then hold the root lock

2 participants