tests: full suites intermittently lose writable scratch #3010

Activity

1. schlessera

   added

   type: testCoverage, harness or proof work with no behaviour change.Coverage, harness or proof work with no behaviour change.

   area: ui-serverHono app factory: turn coordinator, auth, session catalog, routes, cron.Hono app factory: turn coordinator, auth, session catalog, routes, cron.

   needs: reproBlocked on a reproduction.Blocked on a reproduction.

   on Oct 26, 2026
2. schlessera

   changed the title [-]ui-server: strict startup-order test fails only in the package suite[/-] [+]tests: full suites intermittently lose writable scratch[/+] on Oct 26, 2026
3. schlessera

   added this to the 0.45.0 milestone on Oct 26, 2026
4. schlessera commented on Oct 26, 2026

   

   schlessera

   on Oct 26, 2026

   OwnerAuthor

   More actions

   Additional completed controls while proving #3001, with Bun 1.4.2 and frozen dependencies:

   • Full head b9b7d099ba362838f22407902c9d3a4b9fc70cc6, based on c47483196f02c1098128207ce68c4f0de6b571ab: bun run test ./tests failed 1963 pass / 3 fail, 74210 assertions, 848.55 seconds. The failures are the Claude interactive, Claude voice and pi interactive cases at tests/policy-write-boundary.test.ts:80: AREAS_WRITABLE, expected [true, true], received [false, true].
   • The complete isolated policy-boundary file passed on that merged base and on the unchanged head: 19 pass / 0 fail, 587 assertions each.
   • Full merged-base root suite passed 1966/1966, 74412 assertions, 909.12 seconds.
   • An unchanged full-head root retry in the same checkout passed 1966/1966, 74414 assertions, 743.99 seconds. The failed full-head receipt remains retained. No source, assertion, timeout, scratch rule or offline guard changed between the failed and passing head runs. All full-suite commands used the shared heavy-command slots.

   These controls establish variability, not its cause or a repair. They do not reproduce a failing full current-main invocation. The missing correlation remains the failed request/worker's scratch resolution, Git-ignore result, mount facts and access result; the full-suite failure shares the symptom recorded here, with the voice and pi interactive observations also tracked by #2923 and #2987.
5. schlessera commented on Oct 26, 2026

   

   schlessera

   on Oct 26, 2026

   OwnerAuthor

   More actions

   Additional evidence from #3003 (2026-10-26):

   • Full root command at head 942897313c06a31499c2509cbc3357598aaa436d, merged base c47483196f02c1098128207ce68c4f0de6b571ab: 1965 pass / 1 fail, 74319 assertions. The Claude interactive policy-boundary case fails AREAS_WRITABLE, expected [true, true], received [false, true] (executorsHeld, tests/policy-write-boundary.test.ts:71-83). The schedules changes do not edit that test, scratch qualification or the worker sandbox.
   • After the coordinator-requested refresh, the complete isolated policy-boundary file passes on pristine main 07fbce896099b87903bb994e3b6a939f232669fd and on schedules head 7405e78084f4138eee9362df96f94dc92a3efbe0: 19 pass / 0 fail, 587 assertions each.
   • The refreshed full root suite is submitted through the shared heavy-command wrapper. The earlier failed full receipt remains retained. No assertion, timeout, scratch rule or offline guard was changed.

   This matches the recorded scratch-writability symptom. The isolated controls do not reproduce a failing full current-main invocation or identify a cause or repair.
6. schlessera

   added

   priority: p2Wanted, not scheduled. Pulled in when a milestone has room.Wanted, not scheduled. Pulled in when a milestone has room.

   priority: p1Scheduled into the current or next milestone.Scheduled into the current or next milestone.

   and removed

   priority: p2Wanted, not scheduled. Pulled in when a milestone has room.Wanted, not scheduled. Pulled in when a milestone has room.

   on Oct 26, 2026
7. schlessera

   added

   priority: p2Wanted, not scheduled. Pulled in when a milestone has room.Wanted, not scheduled. Pulled in when a milestone has room.

   and removed

   priority: p1Scheduled into the current or next milestone.Scheduled into the current or next milestone.

   on Nov 3, 2026
8. schlessera commented on Nov 4, 2026

   

   schlessera

   on Nov 4, 2026

   OwnerAuthor

   More actions

   Reproduction triage — 2026-11-04: inconclusive; retain needs: repro.

   Inspected and executed unchanged HEAD 53b862123d7b8e4c9d039859fabdc9fc6ad4cd36, matching origin/main at the initial fetch. Linux x86_64, kernel 7.0.0-42-generic, UID/GID 1001/1001, Bun 1.4.4 (38afee6c8), Git 2.53.0, bubblewrap 0.11.1, Python 3.14.4, Node 26.13.0, libseccomp; temporary fixture filesystem: tmpfs. Dependencies were initially absent; bun install --offline --frozen-lockfile --backend=copyfile --ignore-scripts under umask 022 installed 922 packages from the existing cache. Lock SHA-256 remained ecf6986227406c1fd9d835ae3c119d561e012df5a1bedcae599907332fc20a3e. Runtime commands used an empty environment plus explicit PATH, temporary HOME/XDG directories, UTC and C.UTF-8; no provider credentials, model calls or deployments.

   Read the entire issue and both comments through all comment pages, the no-parent response, the relevant historical reports (#2994/#3000/#3001/#3003, #2923/#2987/#3007/#3019, #3030) and the binding writable-area, worker-isolation, startup-view/native-storage and testing guidance. The parent endpoint confirms HTTP 404 “No parent issue found”; #736 is a related epic, not this issue's parent. Later #3314 adds the test's managed-view stand-in; it does not remove scratch validation. Current navigation: the original scratch-writable assertion is now at expect(hold.sandbox.areas.find, packages/ui-server/tests/native-worker-startup-order.test.ts:23; scratchIgnoredByGit remains at packages/ui-sdk/src/server/worker-sandbox.ts:409-423. The native access assertion remains AREAS_WRITABLE, tests/policy-write-boundary.test.ts:80.

   Completed bounded runtime controls, with original assertions and preload guards intact:

   Selection	Actual result
   Startup-order file, pass-through observation of the real Git query and resolved layout	1 pass / 0 fail / 7 assertions, exit 0, 227 ms test process
   Uninstrumented startup-order file, --rerun-each=50 in one Bun test process	50 pass / 0 fail / 350 assertions, exit 0, 3.85 s
   Existing own-repository and parent-repository Git-ignore controls, followed by startup-order	3 pass / 0 fail / 33 filtered / 13 assertions, exit 0, 502 ms
   Claude interactive policy-boundary case inside a fresh outer offline bubblewrap namespace	0 pass / 1 fail / 18 filtered / 1 assertion, exit 1, 20 ms; stopped at the child setup assertion on line 30, before AREAS_WRITABLE

   The passing observed startup invocation produced this tuple for both sandbox resolutions: scratch is a canonical real directory, mode 0700, no mount points at/below it, host accessSync(W_OK) succeeds, and the resulting scratch plan has writable: true. The first captured startup view lacks scratch; the refreshed view contains it. The test's original checkpoint, replacement-view cleanup and gate-release assertions all pass.

   The actual credential-free maintenance queries are git rev-parse --git-dir, exit 128 both times; the fixture has no .git. A separate same-shape query through runMaintenanceWorkerSync returned exit 128 with fatal: not a git repository (or any of the parent directories): .git; direct fixture Git also returned 128. This exercises the expected no-repository branch, not check-ignore for a repository. The two companion existing SDK controls independently pass the ignored/unignored assertions for brains with their own and a parent repository. Host W_OK here is not an in-agent access measurement.

   Harness limits and excluded attempts:

   • The repository's standard scripts/test-offline.sh -- bun run test ./packages/ui-server/tests/native-worker-startup-order.test.ts refuses before test discovery: unshare: write failed /proc/self/uid_map: Operation not permitted.
   • For the three completed controls above, a command-scoped inherited seccomp filter denied IPv4/IPv6/packet sockets and io_uring setup; real IPv4/IPv6 socket sentinels returned EPERM, while Unix and route-netlink sentinels passed. This leaves the original Bun preload and real local maintenance worker enabled. An earlier overbroad version also denied netlink, causing a bubblewrap loopback-lookup refusal and Git-worker exit 1; its passing startup repetitions and its failed policy attempt are excluded from the completed controls. The corrected run's Git exit is 128.
   • Without that socket filter, the native policy attempt used a verified separate outer network namespace with empty IPv4/IPv6 main routing tables. Its real nested bubblewrap child refused namespace creation. The exact failed assertion is expect(exit, err || out).toBe(0), tests/policy-write-boundary.test.ts:30: expected 0, received 1, with bwrap: No permissions to create a new namespace. A separate nested bwrap ... bwrap ... /bin/true control produces the same setup refusal. This is a host prerequisite failure, not the reported scratch-false failure.
   • An initial credential-free launch selected an unresolved Bun shim and ran no tests; subsequent commands used the real executable.
   • No full UI-server/root-suite ordering was replayed. These focused and repeated passes neither disprove the historical failures nor establish their state boundary. Bun also differs from the original 1.4.2 reports.

   Reusable diagnostic/control recipe (from the inspected checkout; all added files stay outside authored source). The executable lookup below assumes mise supplies Bun/Node; use their real executable paths instead on another host. The observation wrappers only delegate to the original functions and restore themselves; they do not inject a refusal or alter assertions.

   repro_dir=$(mktemp -d /tmp/odysseus-3010-XXXXXX)
   repro_bun=$(mise which bun)
   repro_node=$(mise which node)
   repro_path="$(dirname "$repro_bun"):$(dirname "$repro_node"):/usr/bin:/bin"
   mkdir -m 700 "$repro_dir/home" "$repro_dir/config" "$repro_dir/cache"
   cat > "$repro_dir/offline.py" <<'PY'
   import ctypes, errno, os, socket, sys
   lib = ctypes.CDLL("libseccomp.so.2", use_errno=True)
   lib.seccomp_init.argtypes = [ctypes.c_uint32]
   lib.seccomp_init.restype = ctypes.c_void_p
   lib.seccomp_syscall_resolve_name.argtypes = [ctypes.c_char_p]
   lib.seccomp_syscall_resolve_name.restype = ctypes.c_int
   lib.seccomp_rule_add.argtypes = [ctypes.c_void_p, ctypes.c_uint32, ctypes.c_int, ctypes.c_uint]
   lib.seccomp_rule_add.restype = ctypes.c_int
   class Compare(ctypes.Structure):
       _fields_ = [("arg",ctypes.c_uint),("op",ctypes.c_int),("datum_a",ctypes.c_uint64),("datum_b",ctypes.c_uint64)]
   lib.seccomp_rule_add_array.argtypes = [ctypes.c_void_p,ctypes.c_uint32,ctypes.c_int,ctypes.c_uint,ctypes.POINTER(Compare)]
   lib.seccomp_rule_add_array.restype = ctypes.c_int
   lib.seccomp_load.argtypes = [ctypes.c_void_p]
   lib.seccomp_load.restype = ctypes.c_int
   lib.seccomp_release.argtypes = [ctypes.c_void_p]
   ctx = lib.seccomp_init(0x7fff0000)  # SECCOMP_RET_ALLOW, linux/seccomp.h
   assert ctx
   try:
       number = lib.seccomp_syscall_resolve_name(b"socket")
       assert number >= 0
       for family in (socket.AF_INET,socket.AF_INET6,socket.AF_PACKET):
           comparison = Compare(0,4,family,0) # SCMP_CMP_EQ; checked by sentinels below
           assert lib.seccomp_rule_add_array(ctx,0x00050000 | errno.EPERM,number,1,ctypes.byref(comparison)) == 0
       number = lib.seccomp_syscall_resolve_name(b"io_uring_setup")
       assert number >= 0
       assert lib.seccomp_rule_add(ctx,0x00050000 | errno.EPERM,number,0) == 0
       assert lib.seccomp_load(ctx) == 0
   finally:
       lib.seccomp_release(ctx)
   for family in (socket.AF_INET, socket.AF_INET6):
       try:
           socket.socket(family, socket.SOCK_STREAM)
       except OSError as error:
           assert error.errno == errno.EPERM
       else:
           raise AssertionError("network socket was not denied")
   for family,kind,protocol in ((socket.AF_UNIX,socket.SOCK_STREAM,0),(socket.AF_NETLINK,socket.SOCK_RAW,socket.NETLINK_ROUTE)):
       with socket.socket(family,kind,protocol): pass
   print("offline seccomp: IPv4/IPv6/packet socket + io_uring_setup denied; IPv4/IPv6 sentinels EPERM; Unix/route-netlink sentinels pass", file=sys.stderr, flush=True)
   os.execvp(sys.argv[1], sys.argv[1:])
   PY
   cat > "$repro_dir/observe.ts" <<'TS'
   import { afterAll, spyOn } from "bun:test";
   import { accessSync, constants, existsSync, lstatSync, realpathSync } from "node:fs";
   import { join } from "node:path";
   import { pathToFileURL } from "node:url";
   const load = (rel: string) => import(pathToFileURL(join(process.cwd(), rel)).href);
   const maintenance = await load("packages/ui-sdk/src/server/maintenance-worker.ts");
   const launcher = await load("packages/ui-sdk/src/server/worker-launcher.ts");
   const sandbox = await load("packages/ui-sdk/src/server/worker-sandbox.ts");
   const steps = await load("packages/ui-sdk/src/server/worker-steps.ts");
   const git = maintenance.brainGitSteps;
   const prepare = launcher.prepareWorkerSandboxAsync;
   const spawnStep = steps.workerStepSpawn;
   const clean = (value: unknown) => JSON.parse(JSON.stringify(value, (_key, item) =>
     typeof item === "string" ? item.replace(/\/tmp\/odysseus-startup-order-[^/\s]+/g,"<brain>").replace(/\/tmp\/brain-startup-input-[^/\s]+\/startup-[^/\s]+\/brain/g,"<view>") : item));
   const log = (event: string, data: unknown) => console.error("ODYSSEUS_3010 "+JSON.stringify({event,...clean(data)}));
   const spawns = spyOn(steps,"workerStepSpawn").mockImplementation(function* (command: string[], options: any): any {
     const result = yield* spawnStep(command, options);
     log("maintenance_spawn", {exitCode:result.exitCode});
     return result;
   });
   const gitQueries = spyOn(maintenance,"brainGitSteps").mockImplementation(function* (brain: string,args: string[],options: any): any {
     try {
       const exitCode = yield* git(brain,args,options);
       log("git_query", {args,exitCode,ownGit:existsSync(join(brain,".git")),viewHasScratch:existsSync(join(options.startupView.root,".brain/scratch"))});
       return exitCode;
     } catch (error) {
       log("git_error", {args,error:String(error)});
       throw error;
     }
   });
   const layouts = spyOn(launcher,"prepareWorkerSandboxAsync").mockImplementation(async (brain: string,options: any) => {
     const result = await prepare(brain,options);
     const path = join(brain,".brain/scratch");
     const info = lstatSync(path,{throwIfNoEntry:false});
     let writeAccess: boolean | string = true;
     try { accessSync(path,constants.W_OK); } catch(error) {writeAccess=(error as any).code;}
     log("layout", {areas:result.areas,scratch: {exists:!!info,directory:info?.isDirectory(),symlink:info?.isSymbolicLink(),
       mode:info && (info.mode & 0o777).toString(8),canonical:existsSync(path) && realpathSync(path)===path,
       mountPoints:sandbox.mountPointsUnder(path),writeAccess},viewHasScratch:existsSync(join(options.maintenance.startupView.root,".brain/scratch"))});
     return result;
   });
   afterAll(()=>{layouts.mockRestore();gitQueries.mockRestore();spawns.mockRestore();});
   TS
   repro_run() {
     timeout --signal=TERM --kill-after=5s 45s env -i \
       PATH="$repro_path" LANG=C.UTF-8 TZ=UTC HOME="$repro_dir/home" \
       XDG_CONFIG_HOME="$repro_dir/config" XDG_CACHE_HOME="$repro_dir/cache" \
       python3 "$repro_dir/offline.py" "$repro_bun" run test "$@"
   }
   repro_run ./packages/ui-server/tests/native-worker-startup-order.test.ts \
     --preload "$repro_dir/observe.ts"
   repro_run ./packages/ui-server/tests/native-worker-startup-order.test.ts --rerun-each=50
   repro_run ./packages/ui-sdk/tests/worker-sandbox.test.ts \
     ./packages/ui-server/tests/native-worker-startup-order.test.ts \
     -t 'scratch in a.*repository is writable|a first strict admission'

   Retained corrected-control log SHA-256 values: startup observation 3eec0346e05c62dba0c5c9718f7d14550287b8abbd3d6dd1ac2741a6843ddc3d; 50-repeat control cb93005a9167dfca0daf60349e9833e3e64d6bea441b8bd164ab39c03650bd81; Git controls dd82082e4c4e615ec745f2a7fb624c61c3b942e95bae29723275118362554df4; native setup refusal a990e8f5ee4f94dbacecf9599766f8256ad0705ed8e14a0f3b346b4e62944624. These identify local receipts, not a passing full-suite proof.

   Next step: on a host that qualifies the required nested native boundary, retain the first naturally failing package/root ordering with pass-through capture of the same request's resolved scratch reason, Git command/exit/stderr, topology/mount facts and in-worker W_OK result. Distinguish startup's no-repository query from repository check-ignore, and distinguish an admission refusal from false in-worker access. Do not force a bad ledger, stub Git to fail, or treat a namespace refusal as the original reproduction. The historical linkage to other scratch symptoms remains unproven.

   No source fix, mutation, commit, PR, push, closure or assignment change. Checkout remains clean at the tested SHA. During triage the tracking ref advanced to 1398da138c7a9d6a58b01e935da151c981c92bf7; only the worker-var-sockets fixture/test differ from the tested snapshot, and these controls are not relabeled as a run on that newer SHA.

   Read-back labels are type: test, area: ui-server, priority: p2, needs: repro; no label changes. Milestone 0.45.0, open state and empty assignments are preserved. Keep agent-ready absent: same-failure scratch/Git/access/mount attribution is still missing.