fix(runtime-host): let operators raise the election deadline and clarify timeout copy - #3480
Conversation
…ify timeout copy (apache#3474) A workspace whose first post-upgrade start takes longer than the fixed 45s election window fails the client with 'Runtime Host stopped responding during startup', and every candidate spawned along the way keeps starting in the background. Observed on a ~788MB runtime.sqlite: the host emitted runtime_host_ready after ~75s, 30s after the client had already given up. The election deadline was already an interface parameter, but nothing let an operator widen it. MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS now overrides the default at the single resolution point, so CLI, desktop, and owned launches all honor it. Invalid values fail closed instead of being silently ignored - a typo would otherwise leave the operator believing they widened the window. The two deadline-elapsed reasons also claimed things neither could know ('stopped responding'); both now describe what was observed, name slow first starts on large workspaces as the likely cause, and point at the override. Candidates already self-terminate through their initial connection deadline once startup settles, so no lifecycle change beyond copy is needed there. Fixes apache#3474 Generated-by: Maka
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks — the bounded range and the fail-closed validation are the right shape for an operational override, and keeping it as one more input to the existing deadline (rather than a second timer) means there's still a single election authority.
What this solves / how: the fixed 45s election window is too short for real installs — #3474's 788MB workspace takes ~75s to reach ready after an upgrade, through cold start, WAL and one-time migration work. The client can't estimate that before root authority initialises, so it can't self-tune. MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS resolves at the single existing resolution point, defaults unchanged at 45s, and is clamped to 1–120000ms.
On whether this belongs as a knob at all, since that's the question worth asking of any new tunable: yes, for the case you're targeting. Startup cost genuinely varies with workspace size, disk cache state and pending upgrade work, and none of that is knowable client-side before the host comes up. A bounded escape hatch is the honest answer, and the closed interval keeps it from becoming "wait forever."
Two findings inline, both about diagnosis rather than the mechanism.
*AI-assisted review. Reviewed independently by two reviewers; the severity of the first finding was revised downward after checking it against a second real-world failure sample.
| const parsed = Number(rawValue); | ||
| if (!Number.isSafeInteger(parsed) || parsed <= 0 || parsed > 120_000) { | ||
| throw new RangeError( | ||
| `${ELECTION_DEADLINE_MS_ENV_VAR} must be an integer between 1 and 120000 milliseconds`, |
There was a problem hiding this comment.
[P3] The error text points every startup failure at this variable, including the class it can't fix.
There's a second failure shape this knob doesn't address. From a real CI run on #3454, which now emits election diagnostics:
{"deadlineMs":45000,"elapsedMs":45174,"candidateLaunches":69,
"observations":{"notRegistered":73,"connectFailed":0,"handshakeFailed":0,
"connected":1,"readyWaitFailed":1}}69 candidate launches in 45 seconds. Zero connect failures, zero handshake failures — it connected once, then ready-wait failed, and candidates kept being launched roughly every 0.65s until the deadline. It missed by 174ms, so raising the deadline would have made that particular run pass, but the run isn't slow — it isn't converging. At 90s it launches ~138 candidates instead of ~69 and passing becomes a matter of luck.
That's different from #3474's genuinely-slow start, where waiting longer is exactly right. The problem is that both produce the same message, so an operator hitting the retry-storm shape is told to raise the timeout, raises it, fails again, and raises it further.
Not blocking, and possibly not yours to fix here — but worth asking: with candidateLaunches high and connectFailed/handshakeFailed at zero, should the message distinguish "this start is slow, wait longer" from "candidates aren't converging, the timeout isn't your problem"? You have the #3474 case in hand and probably know which shape operators hit more often.
| const deadlineMs = input.electionDeadlineMs ?? DEFAULT_ELECTION_DEADLINE_MS; | ||
| const deadlineMs = | ||
| input.electionDeadlineMs ?? | ||
| electionDeadlineMsFromEnvironment( |
There was a problem hiding this comment.
[P3] On the owned-launch path an invalid value loses the variable name and surfaces as host_unresponsive.
electionDeadlineMsFromEnvironment() throws a RangeError naming the variable, which is exactly right for direct connectOrSpawnRuntimeHost. But connectOwnedRuntimeHostWithDependencies() wraps the call in a broad catch that converts anything into { kind: 'failed', reason: 'host_unresponsive' }. So MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MS=abc reaches the user as Runtime Host did not start: host_unresponsive — they're told the host is unresponsive when in fact they made a typo, and the message gives them nothing to act on.
This is the same shape as the finding above: the mechanism is fine, the diagnosis points the wrong way. Config errors specifically are worth letting through that catch (or resolving before it, as a typed configuration failure), since they're the one class the user can fix immediately once they know the variable name. Worth a test on both the owned and hosted paths.
Closes #3474
Problem
On a large workspace (~788 MB
runtime.sqlite), the first Runtime Host start after an upgrade emittedruntime_host_readyafter ~75s — 30s after the client's fixed 45s election window had already elapsed. The user sees:which is misleading: the host was alive and making progress the whole time. The candidates spawned during the election keep starting detached, so they eventually win and hold the root while every later attempt starts from a cold election again.
Change
MAKA_RUNTIME_HOST_ELECTION_DEADLINE_MSnow overrides the default election deadline at its single resolution point inconnectOrSpawnRuntimeHostWithDependencies, so CLI, desktop, and owned launches all honor it. The window stays bounded (1–120000 ms). Invalid values fail closed with a `RangeError` naming the variable — a silently ignored typo would leave the operator believing they widened the window when they did not.startup_timeout,host_unresponsive) no longer claim things neither reason can know ('stopped responding'). They describe what was observed, name slow first starts on large workspaces as the likely cause, and point at the override.Testing
election-deadline-env.test.ts: blank/unset handling, valid parsing, fail-closed on invalid values, and an ordering test proving an invalid override fails before storage is touched.startup-error.test.tsupdated for the new copy plus a knob-mention assertion.packages/runtime-host: full suite 1067/1067 green; desktop manager suite 22/22;tsc --noEmitand biome clean.Generated-by: Maka