Skip to content

Spool remote ensure payloads over stdin so big-memory loops reattach - #548

Merged
scgopi merged 1 commit into
mainfrom
fix/remote-reattach-random
Oct 3, 2026
Merged

scgopi merged 1 commit into
mainfrom
fix/remote-reattach-random

Conversation

@scgopi

@scgopi scgopi commented Oct 1, 2026

Copy link
Copy Markdown
Owner

Problem

After a codespace restart, some loops never reattach: the host's dials.log shows only reboot wait-daemon for them, never an ensure line, while sibling loops on the same codespace recover.

The daemon's remote ensure put the CLI shim (~45 KB), the briefing (twice for Copilot) and that loop's wake digest (up to 40 × 512 B memos + an 8 KB playbook) inline in the ssh remote command, base64'd twice. sshd runs that command as the single -c argument of the login shell, and Linux refuses any single argument over 128 KiB (MAX_ARG_STRLEN) with E2BIG before a byte of it runs. So:

  • loops with a large memory failed every ensure, silently, with no line on the host's dial log;
  • after a reboot, their panes waited on graphcoded forever, because only the daemon restores unattended loops;
  • loops with small memories came back. Which loops fail depends on each loop's memory, so the subset looks random. Long-lived time loops collect the most memory.

The pane's own dial carries no wake digest, so it stayed under the limit.

Measured on main, worst-case memory: 160,051 bytes (Claude Code) and 174,681 bytes (Copilot) against 131,072.

Fix

  • The ensure reads every delivered file from the dial's stdin (RemotePayloadSpool). The command line carries only a byte count and a SHA-256 per payload, so it no longer grows with the loop's memory. A short or garbled read leaves the file empty, which shows up as delivery install failed in the host's dial log. Stdin is then closed, and a 60 s alarm stops a dial whose input never arrives.
  • The payload travels as base64 wrapped at 76 columns, because the daemon's PTY is in canonical mode, which drops lines past MAX_CANON. It is written off the calling task, since a PTY write blocks until ssh reads it.
  • The pane's inline delivery (shim + briefing, no memory) is deflated for margin. Python inflates it with zlib.decompress(…, -15).
  • A failed ensure is now logged locally: graphcoded.log gets event=remote-ensure-failed node=… argvBytes=… stdinBytes=… output=…. The host's dial log can never record a dial that failed before its script ran.

Evidence

RED: main sources + anEnsureFitsOneLinuxArgumentWithAFullMemory via swift test (SwiftPM target over GraphcodeKit) -> FAILED, remote command is 160051 bytes (claudeCode) and 174681 bytes (copilotCLI), limit 131072
GREEN: swift test --filter RemoteEnsureArgumentLimitTests and RemoteSessionLaunchTests on this branch -> 26 tests in 2 suites passed, including a 150 KB+ spooled delivery landing byte-exact through runRemoteRetryingCollecting's PTY
REGRESSION: swift build (GraphcodeKit, graphcode-cli, graphcoded) -> exit 0; swift test RemoteSessionLaunchTests + RemoteSessionResumeTests -> 56 tests in 3 suites passed; swift format lint --strict on touched files -> exit 0

Caveats:

  • The tests ran through a temporary SwiftPM test target, not committed. The xcodebuild app-scheme test could not finish here: a cold build ran out of a 2-hour window on a host at load average 50–100. CI's macos job is the full app-scheme run.
  • E2BIG has not been executed on a real Linux sshd. The rig is macOS sshd, which has no per-argument limit, and no codespace scope is available here. The size test pins the bound.
  • An earlier run of the spool test hung once, at the 60 s timeout, under that load. It was not reproduced in 6 direct PTY runs or 3 later test runs.

Not in this PR: CodespaceDialBreaker can strand every loop on a codespace whose daemon reads fail for more than 240 s. Only a human marker or a successful daemon dial clears it, and a pane's successful redial does neither. That needs a separate change.

@scgopi
scgopi force-pushed the fix/remote-reattach-random branch from 0dc7af9 to e7241af Compare October 2, 2026 15:24
The daemon's remote ensure carried the CLI shim, the briefing (twice for
Copilot) and the loop's wake digest inline in the ssh remote command, which
the host runs as a single `zsh -c` argument. Linux refuses one argument over
128 KiB (MAX_ARG_STRLEN) with E2BIG before any of it runs, so loops with a
large memory failed every ensure silently: no dial-log line on the host,
and after a reboot their panes sat on "reboot wait-daemon" while siblings
with smaller memories came back.

The ensure now spools every delivered file from the dial's stdin (length
and SHA-256 checked, 60s alarm), so its command line no longer grows with
the loop's memory. The pane's inline delivery is deflated for margin. A
failed ensure is recorded in graphcoded.log with the node, argv and stdin
sizes and ssh's output.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: scgopi <[email protected]>
@scgopi
scgopi force-pushed the fix/remote-reattach-random branch from e7241af to e8c060b Compare October 3, 2026 01:21
@scgopi
scgopi merged commit 9a693ae into main Oct 3, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant