Run the real binary, and the real CLIs: a two-tier e2e harness#29
Merged
Conversation
Every test so far asserted on the launch PLAN — the in-memory object planLaunch
and buildEnvPlan produce. Nothing ran the real bin/swisscode.js and observed
what it launched, so the execve handoff — the single most consequential line in
the tool, which ci.yml itself admitted nothing covered — was untested. This adds
a two-tier end-to-end harness that closes it, with no production code change: it
rides the SWISSCODE_*_BIN override seam that already exists.
TIER A — hermetic (test/e2e/*.e2e.ts, `npm run test:e2e`, every PR). Point the
three overrides at a recorder that writes down its own argv/env/cwd and exits.
Run the real binary; read the capture. The whole pipeline runs — parse, load,
resolve, lower, execve — with no network, no credential, full determinism. 29
cases: every provider through Claude Code, the three agents, the --cc-* surface,
passthrough, session mode, and the error/exit-code paths.
The assertion a plan test cannot make: for a third-party provider, the stale
ANTHROPIC_API_KEY from the polluted ambient env is ABSENT FROM THE LAUNCHED
CHILD. Present-in-plan.unset is not the same as gone-from-the-process; only a
real launch shows it. golden's claim, verified one layer further out — and it
now also covers the execve dispatch of replace(), which taking that branch in a
unit test could never do (it would replace the test runner; here it replaces the
child).
Two things the build itself taught, fixed rather than worked around:
- The recorder must be COPIED into /tmp, not symlinked from the repo: the
recursion guard realpaths every candidate and correctly rejects anything
under swisscode's own directory, which a symlink resolves back into. The
recursion-guard e2e depends on this being real.
- The Dockerfile needs `npm ci --ignore-scripts`: the repo's `prepare` is
`node build.js`, and at the deps layer the source it needs is not copied
yet. Caught by actually building the image, not by reading.
TIER B — real CLIs (test/e2e/*.real.ts, `npm run test:e2e:real`, manual only). A
Docker image installs @anthropic-ai/claude-code, @kilocode/cli and opencode-ai
at pinned versions and runs swisscode against them. Non-billable: it forwards
only --version, which every CLI answers before auth. It cannot be deterministic
(it depends on upstream), so e2e-real.yml is workflow_dispatch only and never
gates a PR — it catches an upstream flag rename when someone runs it. Verified
for real: the image builds and all three CLIs resolve and hand off cleanly.
The three test sets are disjoint by extension — *.test.ts (unit), *.e2e.ts
(hermetic), *.real.ts (Docker) — so `npm test` stays hermetic and fast and a PR
that has not installed the real CLIs stays green. Nothing ships: `files` is
bin/dist/README, so the packed size is unchanged.
783 unit tests, 29 hermetic e2e, 4 real (in Docker). ci.yml runs Tier A on every
PR across Node 22 and 24.
Signed-off-by: jellologic <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every test today asserts on the launch plan (an in-memory object). Nothing ran the real
bin/swisscode.jsand observed what it launched — so theexecvehandoff, whichci.ymlitself admits nothing covers, was untested. This closes it with no production code change, riding theSWISSCODE_*_BINoverride seam that already exists.Tier A — hermetic (every PR). Point the overrides at a recorder that captures its own argv/env/cwd and exits; run the real binary; read the capture. 29 cases: every provider, the three agents, the
--cc-*surface, session mode, error/exit paths. The assertion a plan test cannot make — for a third-party provider, the staleANTHROPIC_API_KEYis absent from the launched child, the billing guard verified at the OS level.Tier B — real CLIs (manual). A Docker image installs the three actual CLIs at pinned versions and hands off via
--version— non-billable, no secrets. Verified: the image builds and all three resolve and hand off cleanly.Two bugs the building caught: the recorder had to be copied (a symlink resolves back into the repo and the recursion guard correctly refuses it), and the Dockerfile needed
--ignore-scripts(the repo'sprepareisnode build.js, absent at the deps layer). One honest correction: Node 22.23 backportedexecve, so the e2e covers the execve dispatch (the real gap); spawn stays unit-tested — the docs say so plainly.783 unit, 29 hermetic e2e, 4 real (Docker). Nothing ships — packed size unchanged.