Skip to content

ci: bind only stable host paths in Docker-based local proofs, so each run stops leaking a mount #2376

Open

Description

@schlessera

Context

On WSL2 with Docker Desktop, the daemon stages every distinct host source path used in a bind into the distro as a permanent mount under /mnt/wsl/docker-desktop-bind-mounts/<distro>/<sha256-of-realpath>. Nothing removes it until Docker Desktop restarts. This is upstream behaviour, reported without a fix in docker/for-win#9823. Measured 2026-10-18, one container run per row, mount delta in /proc/self/mountinfo:

form new source path same path again
-v src:dst:ro +1 (3/3) +0
--mount type=bind,… +1 (3/3) +0
single-file -v +1 not measured
-v src:dst:ro,rslave +1 not measured
--mount type=tmpfs +0 —
docker cp into a created container +0 —
named volume filled by tar on stdin +0 —

Deleting the source path afterwards does not release its mount. bind-propagation=slave|rslave|shared|rshared via --mount is rejected outright ("mounted on / but it is not a shared mount"), so propagation is not a way out.

Several local tools bind a freshly created path on every run, so the host accumulates mounts with every proof: 7,892 such mounts had built up on one development host in under two days, and leaked mounts make every bubblewrap 0.9 sandbox slower (#2370, #2375). Docker callers at starting main 82a64a13a08c44f2daef425d1a9df2c10840ea8e:

  • (const artifact, scripts/prove-worker-identity.ts:92) is a fresh mkdtemp run directory. It is bound together with four files created inside it: the identity helper, helper.py, the filesystem filter and setup.sh. That's about 5 new mounts per run, plus any per-fault variants.
  • (const artifact, scripts/prove-native-host.ts:30) is a fresh run directory per backend/uid.
  • (staging = await mkdtemp, scripts/capture.ts:79) is a fresh capture staging directory per run.
  • tests/workspace-lease.test.ts binds its fixture root into the pinned container ("-v", \${root}:/repo``). Whether that root is fresh per run needs checking.
  • The native diagram producer creates a fresh private state directory (prepareDiagramNativeArtifacts, scripts/diagram-export-producer.mjs:30-50) and binds its snapshot, filtered installation, captured Bun and output paths. Those are fresh per invocation, not stable per checkout. Its original-repository and ambient-canary absence assertions forbid exposing a shared parent containing extra data; use the already allowed private copy/volume transport instead.
  • scripts/visual.mjs, scripts/captures/check-runtime-raster.ts and the audio probe bind stable per-checkout/cache/output paths. Those paths leak once per distinct checkout or explicit output, rather than once per inner run identity.

Scope

Make each Docker caller bind only stable source paths. Use one fixed directory per checkout (for example tmp/docker-share/) and select the per-run subdirectory inside the container through an argument or environment variable. Bind directories, never freshly created single files. Where a caller only needs inputs copied in and outputs copied out, docker cp or a tar-filled named volume is acceptable instead. Record which form each caller uses and why.

Out of scope

  • Fixing Docker Desktop, or cleaning up existing leaked mounts from the tooling.
  • The per-new-worktree leak of stable paths. It is bounded by worktree count and needs a different design, such as copying instead of binding.
  • Changing what the proofs assert, their images or their isolation flags.

Acceptance criteria

  • Running each changed proof twice in a row on WSL2 with Docker Desktop leaves /proc/self/mountinfo unchanged by the second run. Record before/after counts per script in the PR.
  • A unit test per changed caller asserts that the bind source paths in its constructed docker run argv are identical across two invocations with different run identities. Mutation receipt: reintroducing a fresh mkdtemp bind fails that test.
  • The proofs pass unchanged, with the same assertions and native receipts as before.

Verification

Per-script double-run mount counts on a Docker Desktop WSL2 host, the argv-stability unit tests, and the affected native proofs (bun scripts/prove-worker-identity.ts, bun scripts/prove-native-host.ts, bun run capture:verify).

Contract impact

None. Local tooling only.

2026-10-18 source audit correction: verified the named callers at the starting main above. Corrected the worker artifact citation and the mistaken claim that diagram export sources were stable; that producer binds four fresh private paths. The existing scope explicitly allows copied inputs/outputs or tar-filled named volumes, so its correction preserves private input visibility and every original absence assertion without mounting the private snapshot’s ambient parent. The workspace lease test also creates a fresh fixture root; its Docker fixture now belongs beneath a stable checkout-owned share.

Delivered prerequisite: #2365 through PR #2374.

2026-10-18 independent proof-gate finding: the unchanged tests/native-diagram-inputs.test.ts fails COMPLETE_NATIVE_QUALIFICATION_WITHOUT_DOCKER_OR_IMAGE_LOOKUP (expected 859, received undefined). A temporary observer in an owned copy of that test records the exact fixture diagnostic: status source-mismatch; 859 source inputs; all three source pins match; expected source e8502b6ffb3c575f2e9a1c93dea8dbf5ceda93ece367da9d542cb10b1c9ac32b, actual 48bd2c18d694da6e627023f713eeed58960fcb0698812784bc9a073c318a6f25; expected installation 0c2758c046f2b69d5f33b66521baf3098db5f44256b73ec96961c98d381e6245, actual 0f4614b41729d5f1c974a88f0497b77c088c045de6b71a3012ff91d543c2df51. The observer copy is removed and the original test is untouched. This frozen-positive/current-source assumption is already owned by #2365; no qualification hash or refusal guard is changed here. The live candidate's changed producer must execute fresh, rather than inherit old native qualification. #2365 is already higher priority p1 than this p2.

Independent functional/mount proof is complete for the changed callers: pinned lease 85→85→85; native host admission 85→89→89; worker identity 89→90→90; private diagram export 90→90→90; full capture verification 90→93→93. Every invocation passes. Final committed diagram proof repeats at 93→93→93. Detailed identities and limitations belong to the draft PR proof. Final candidate e6f1716bd33e6c4f36637bf0e8db1360a9d80aa7 on parent 63fde3c8fe76e754aa4524f95b23f9786f4447fe passes lint, leakage, strict types and 956 fast tests; focused tests pass 28/28. Full local preflight remains blocked by the sole affected-suite failure owned by #2365 (176 pass, one fail). No frozen hashes or refusal assertions were changed. Resume after #2365 lands by refreshing the branch, rerunning preflight and obtaining complete selected hosted candidate proof. These local receipts do not substitute for that qualification.

2026-10-18 handoff correction: captures and final preflight are now reconciled above; earlier source-mismatch hashes describe their explicitly recorded local checkpoint, not a newly qualified final source tree.

2026-10-18 source refresh: delivered #2375 option 1 through #2381, squash 268b76b, and integrated current main into the preserved #2376 branch. Refreshed local candidate 16c9ed0 has parent main 268b76b and tree 7f768552cabb7aed28c90d66116975a649591835. The merge is conflict-free and retains stable-bind transport alongside the newly delivered SDK leaf and complete default identity phase selection. Build, strict types and focused transport/citation checks are running. Historical double-run measurements above keep their original source identities. #2365 remains open with its current owner; this campaign does not take over that work. No new combined hosted qualification or passing preflight is claimed yet.

2026-10-18 refreshed candidate 16c9ed0: build, strict types and six focused transport/output/citation suites pass (73 tests, 229 assertions). Consecutive real Docker proofs pass: lease total mounts 85→85→85 (Desktop shares 15→15→15), native-host 85→88→88 (15→18→18), complete default identity 88→89→89 (18→19→19). All four backend/UID native-host cells and the intended retained-index negative are verified in both invocations; namespace quota and full loop/mount/backing cleanup are confirmed. Each identity invocation has complete scope, all four default app/control phases, no skipped categories, confirmed container removal and zero remaining owned resources. No second-run mount growth is observed.

The subsequent first diagram invocation fails before rendering while copying its 1.9 GiB installed dependencies: actual ENOSPC on the host filesystem (about 1.3 GiB free). It creates no new host mount. Preserve this failed receipt; it is not a sandbox/diagram assertion failure or passing diagram proof. Removed only regenerable dependency installs from five clean, completed, merged worktrees owned by this campaign after verifying no active process used them; source and receipts remain. Available host capacity is now over 7 GiB. The same diagram and complete capture double-runs resume with unchanged source, assertions, image and deadlines. #2365 remains with its current owner, and current preflight/hosted confirmation remain pending.

2026-10-18 restored-capacity diagram proof passes twice, both engines, 26.09 / 22.53 seconds, total mounts 89→89→89 (Desktop 19→19→19). Complete capture verification also passes twice, 362.08 / 368.87 seconds, all five recipes / 13 exact output files and ten paint-history pairs. The aggregate Desktop counter grows 19→26→39, so the runner correctly reports a failed mount-growth measurement despite both capture commands exiting 0. Do not report those raw counts as zero growth.

Inspection finds concurrent foreign-workspace browser/native-proof shares in the global mount table. This checkout has exactly one mount for each of its eight stable source roots, including capture fonts/share/paint-history, with no duplicate source identities. A bounded attribution run now records complete before/after mount IDs/root paths and the actual Docker argv (a transparent logging client, no argument changes) while rerunning full capture verification. Preserve the failed aggregate measurement; only attribute growth after that direct source evidence is inspected. No assertion, isolation flag, output comparison or native qualification hash is changed.

2026-10-18 capture attribution complete: the transparent client logs seven Docker run calls using only five stable bind sources, with no fresh staging/runner path binds. The complete diagnostic capture verification passes (375.12 seconds), five recipes / 13 exact files and all ten paint-history pairs. Before/after mount IDs are identical: 39 total Desktop shares, eight owned by this checkout, zero new own or foreign shares. The earlier aggregate growth remains a failed unscoped measurement with concurrent foreign-workspace shares; actual root identity evidence and this instrumented complete run establish no further capture mounts from this caller. The earlier product capture runs remain passing output comparisons, not zero-global-growth receipts.

Final prepared draft head 6cc530a integrates main c4d31c6; tree 27d9b66065f9fd0d53ade446d18b3c7dea5982cf. The latest base change is the independently delivered CI shard/weight/headroom adjustment; native-host/identity/diagram/capture implementations and their runtime inputs are unchanged from the measured 16c9ed0 candidate. Preserve those runtime receipts with their actual source identity. Current focused checks pass (73 tests / 229 assertions), strict types and lint pass; complete preflight and selected hosted confirmation still require #2365's current owner to deliver that frozen-positive prerequisite. Do not reinterpret any earlier green as proof of a different base or refresh qualification hashes. The remaining dependency remains canonical above.

2026-10-18 blocked handoff: current implementation and all scoped native/capture receipts are preserved in draft #2385. The only remaining declared prerequisite is #2365 under its existing owner. After delivery, integrate verified main, run complete local preflight and obtain the new ready candidate's selected hosted proof with actual head/base/job checkout and release milestone readbacks before matching-head merge. Claim chain2376-20261018 is released after fresh ownership reconciliation.

Coordinated resumption — 2026-10-19

Claim release044-2376-r2-20261019, intended branch fix/2376-campaign-r2. The previous claim explicitly released custody; fresh readback confirms this issue is open and unassigned. The scoped #2365 correction is delivered by PR #2374, squash 90a95123cfceb169bd2ecf7c213544db742eaed7, whose parent/tree exactly match complete selected PR confirmation. Automatic main-push CI 39175300737 is still running and remains pending; no main-push success is claimed. Other named historical prerequisites are delivered.

Remove the satisfied dependency label and resume the preserved draft implementation, without reimplementing its completed work or relabelling historical runtime measurements. A dedicated GPT-6.1-Sol worker will review the diff and integrate verified main; installation and local/native execution require the parent's resource window. The parent serializes candidate confirmation and merges. Existing deadlines, assertions, images and qualification guards remain unchanged. Fresh candidate preflight and complete selected hosted proof remain required before delivery.

Activity

  1. added
    type: fixSomething is broken and must behave the way it is documented.
    area: ciCI workflows, the gates they run, release scripts, packaging.
    priority: p2Wanted, not scheduled. Pulled in when a milestone has room.
    agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.
    on Oct 17, 2026
  2. self-assigned this
    on Oct 17, 2026
  3. schlessera commented on Oct 17, 2026

    @schlessera
    OwnerAuthor

    Claim mount2376-20261018, intended branch fix/2376-stable-docker-mounts. Working the explicitly requested stable-bind correction in a dedicated checkout from fetched main. I will audit every named caller, preserve images/assertions/isolation flags and collect argv-stability mutation plus WSL2 Docker Desktop double-run mount receipts. #2375's supply-path ruling is separate.

  4. schlessera commented on Oct 17, 2026

    @schlessera
    OwnerAuthor

    Source audit corrected two routine factual gaps in the body. The lease fixture is fresh per test, and diagram export also creates fresh private paths (four host binds per engine), rather than stable checkout paths. The producer's private-source and ambient-canary absence assertions rule out mounting its entire parent. The issue already permits copy-in/copy-out and tar-filled named volumes; I am applying that form to diagram export, keeping its actual source/install visibility, captured Bun bytes, read-only root/input mounts, network boundary and original assertions. No frozen source audit hash is refreshed to admit this transport change; unsupported source stays fresh.

  5. added
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 17, 2026
  6. added this to the 0.44.0 milestone on Oct 17, 2026
  7. schlessera commented on Oct 17, 2026

    @schlessera
    OwnerAuthor

    Blocked handoff for claim mount2376-20261018: preserved implementation in draft PR #2385, head e6f1716bd33e6c4f36637bf0e8db1360a9d80aa7, parent 63fde3c8fe76e754aa4524f95b23f9786f4447fe. All five changed callers pass twice with no second-run mount growth; the PR records actual counts, source identities and limitations. Focused tests and mutations, lint, leakage, strict types, capture verification and final real diagram exports pass. Final preflight has 956 fast passes and 176 affected passes, with one existing frozen-positive native qualification failure owned by #2365. The issue body is updated; original assertions, isolation and qualification hashes remain intact.

    Next: deliver #2365, refresh this preserved branch, rerun focused preflight and obtain complete selected hosted proof for the actual ready candidate before merge. This draft uses Refs and does not close #2376. PR and issue milestone 0.44.0 are verified. Releasing only this invocation's uncontested assignment; implementation remains resumable on fix/2376-stable-docker-mounts.

  8. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    Claim chain2376-20261018 resumes this invocation's preserved branch fix/2376-stable-docker-mounts. Fresh comments/assignment show no competing owner. Refresh source against delivered prerequisites and verify unchanged transport/cleanup assertions while #2365's current owner finishes that independent dependency. No takeover of #2365 or its PR is intended; current hosted qualification remains required before #2376 delivery.

  9. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    Blocked handoff for claim chain2376-20261018: draft #2385 preserves refreshed head 6cc530a, main c4d31c6. Scoped native/capture double-run, cleanup, mutation and exact mount-attribution receipts are recorded in the issue/PR with their actual source identities; final focused 73-test/229-assertion, strict-type and lint checks pass. #2365 remains open with its current owner. After delivery, integrate verified main, complete local preflight and new selected hosted candidate proof before matching-head merge/main reconciliation. Releasing only this invocation's uncontested assignment and preserving the branch/receipts.

  10. removed their assignment
    on Oct 18, 2026
  11. removed
    blockedWaiting on another issue or an upstream dependency. Say which in the body.
    on Oct 18, 2026
  12. self-assigned this
    on Oct 18, 2026
  13. schlessera commented on Oct 18, 2026

    @schlessera
    OwnerAuthor

    Coordinated resumption — 2026-10-19

    Claim release044-2376-r2-20261019, intended branch fix/2376-campaign-r2. The previous claim explicitly released custody; fresh readback confirms this issue is open and unassigned. The scoped #2365 correction is delivered by PR #2374, squash 90a95123cfceb169bd2ecf7c213544db742eaed7, whose parent/tree exactly match complete selected PR confirmation. Automatic main-push CI 39175300737 is still running and remains pending; no main-push success is claimed. Other named historical prerequisites are delivered.

    Remove the satisfied dependency label and resume the preserved draft implementation, without reimplementing its completed work or relabelling historical runtime measurements. A dedicated GPT-6.1-Sol worker will review the diff and integrate verified main; installation and local/native execution require the parent's resource window. The parent serializes candidate confirmation and merges. Existing deadlines, assertions, images and qualification guards remain unchanged. Fresh candidate preflight and complete selected hosted proof remain required before delivery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyContext and acceptance criteria are complete enough for an agent to take this unattended.area: ciCI workflows, the gates they run, release scripts, packaging.priority: p2Wanted, not scheduled. Pulled in when a milestone has room.type: fixSomething is broken and must behave the way it is documented.

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions