From 9a40179f07d35c4cce2bbc12c1aeb485bdcbacbf Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 2 Nov 2026 08:44:46 -0800 Subject: [PATCH 1/7] Docs: design shared filesystem platform adapters and native APFS support --- README.md | 4 + docs/adr/0010-filesystem-platform-adapters.md | 81 ++++++ docs/adr/README.md | 1 + docs/architecture/filesystem-platforms.md | 263 ++++++++++++++++++ 4 files changed, 349 insertions(+) create mode 100644 docs/adr/0010-filesystem-platform-adapters.md create mode 100644 docs/architecture/filesystem-platforms.md diff --git a/README.md b/README.md index 39286307..6b49639e 100644 --- a/README.md +++ b/README.md @@ -170,6 +170,10 @@ The core holds no clock, no caller identity, no paths, and no application policy. Retention proves a *physical reconstruction* claim only — never what the content means, who owns it, or whether deleting it is legally safe. +The [filesystem platform design](docs/architecture/filesystem-platforms.md) +defines the shared Linux/macOS adapter boundary and native APFS acceptance +work. It is a design in implementation, not a claim of shipped macOS support. + ## Try it Keep is `0.0.0` and unpublished; build from source. The in-memory diff --git a/docs/adr/0010-filesystem-platform-adapters.md b/docs/adr/0010-filesystem-platform-adapters.md new file mode 100644 index 00000000..02445062 --- /dev/null +++ b/docs/adr/0010-filesystem-platform-adapters.md @@ -0,0 +1,81 @@ +# ADR-0010: Filesystem Platform Adapters + +- Status: Accepted design; implementation and native acceptance in progress +- Date: 2026-11-02 +- Owners: Keep storage implementation and repository owner +- Related issue: none; directly requested platform design and implementation + +## Context + +Keep's production admission accepts only Linux ext4. Several focused storage +ports already separate protocol ordering from filesystem effects, but their +implementations share concrete Linux assumptions without one native mechanism +boundary. Adding macOS separately to initialization, publication, retention, +migration, and recovery would duplicate authority and durability decisions. + +The design baseline is main `3887f2a`, including its existing distinction +between live mount identity and restart device/inode coordinates. + +## Decision + +Retain the focused semantic storage ports. Implement their filesystem effects +through common Unix protocol adapters over one pinned store authority, backed +by a `PlatformFilesystem` interface. Native implementations supply observations +and OS mechanisms; shared admission policy establishes which profiles are +supported and refuses incomplete evidence. + +The [architecture design](../architecture/filesystem-platforms.md) defines +the operations, ownership, Linux profile extension rules, APFS naming and +durability obligations, and acceptance gates. Backend selection happens once. +It must not alter logical identities or duplicate protocol state machines. + +Native macOS support targets writable local APFS. Exact namespace spelling +must remain enforced on case-insensitive volumes. Other Linux filesystems +are distinct admitted profiles with their own evidence, rather than being +accepted because they provide familiar POSIX operations. Windows and network +filesystems are excluded from this implementation. + +Keep existing public admission constructors and protocol entry points during +the refactor. A future convenience facade may construct typed protocol views +from retained authority, but a general object factory is not necessary for OS +selection and must never create independent competing writer authorities. + +## Alternatives considered + +- **A single broad `StorageInterface`:** conflates initialization, reads, + mutation, and recovery authority; implementations must expose operations + that are invalid in their current state. Keep narrower capability ports. +- **A factory returning unrelated storage objects:** chooses implementations + without sharing their pinned root, observations, or lock lifetime. A + factory can compose the design but cannot replace the platform boundary. +- **A full generic virtual filesystem:** duplicates existing capability file + APIs and spreads type parameters through every protocol. Share Unix file + operations and abstract only the native mechanisms that actually differ. +- **A complete protocol adapter per OS:** duplicates ordered state machines + and lets platform implementations drift on exact-byte and recovery laws. +- **Remove the Linux gate and try ordinary file operations:** turns successful + individual syscalls into an unsupported durability claim. Admission must + remain tied to a named, tested filesystem profile. +- **Require a separate case-sensitive volume without evaluating default + APFS:** avoids a useful native deployment but leaves ordinary Mac storage + unsupported. Exact canonical namespace handling is part of the target. +- **Claim support from Linux containers on a Mac:** exercises Linux semantics, + not Darwin/APFS. Native acceptance requires native production-path evidence. + +## Consequences + +The OS port is an infrastructure boundary; semantic domain ports remain free +of descriptors and filesystem-library types. Adding an OS entails one adapter, +one selector entry, and its conformance evidence, rather than changing every +durability protocol. Filesystem support stays explicit and fail-closed. + +Existing Linux wire bytes remain unchanged. Darwin physical recovery +coordinates need a documented mapping and compatibility analysis; logical +content identity must not incorporate native metadata. Any format change +discovered necessary during implementation requires an explicit revision of +this decision and the format contract before it ships. + +The native validation environment is separately bounded and serialized within +the same project resource budget. Successful process-death recovery is not +evidence of physical power-loss survival. Design acceptance does not claim +that implementation, resource enforcement, or platform validation is complete. diff --git a/docs/adr/README.md b/docs/adr/README.md index 1348e0ca..3f2ca2ce 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,3 +61,4 @@ encryption, concurrency, or public-API surface it governs. - [ADR-0007: Terminal signal process-group guard](0007-terminal-signal-process-group-guard.md) - [ADR-0008: Deadline-bounded reader retirement](0008-deadline-bounded-reader-retirement.md) - [ADR-0009: Retention roots, release, and GC liveness](0009-retention-roots-release-and-gc-liveness.md) +- [ADR-0010: Filesystem platform adapters](0010-filesystem-platform-adapters.md) diff --git a/docs/architecture/filesystem-platforms.md b/docs/architecture/filesystem-platforms.md new file mode 100644 index 00000000..12259dbc --- /dev/null +++ b/docs/architecture/filesystem-platforms.md @@ -0,0 +1,263 @@ +# Filesystem platform adapters + +Status: implementation design. The checked-in Linux adapter remains the only +production-admitted backend until the acceptance gates below have evidence. + +This design adds native macOS storage while retaining Keep's exact-byte, +exclusive-writer, immutable-publication, and recovery contracts. It also defines +the boundary for admitting additional Linux filesystems without copying every +storage protocol. Windows and network filesystems are outside this work. + +## The boundary to introduce + +Keep has several storage **capabilities**, not four competing storage engines. +Initialization, catalog publication, retention publication, and migration each +need a different subset of operations and a different authority state. Keep +those focused ports. Implement them over one admitted filesystem session. + +```mermaid +flowchart TD + C[Identity and transition laws] --> P[Semantic protocol ports] + U[Common Unix filesystem protocol adapters] --> P + U --> S[Pinned store authority and capability handles] + S --> F[PlatformFilesystem mechanism boundary] + L[Linux adapter and filesystem profiles] --> F + M[Darwin adapter and APFS profiles] --> F +``` + +The diagram's arrows mean dependency or implementation, not execution order. +The protocol orchestrator calls its semantic port; the common Unix adapter +implements that port using the selected platform implementation. + +The two boundaries have different purposes: + +| Boundary | Owns | Must not own | +| --- | --- | --- | +| Semantic protocol ports | Required effects, ordering obligations, typed outcomes | OS selection, descriptors, filesystem detection | +| Common filesystem adapters | Canonical namespace, pinned authority, exact verification, protocol effects | Linux or Darwin syscall policy | +| `PlatformFilesystem` | Root traversal, descriptor observations, durability, native locking semantics | Catalog generations, retention policy, codecs, recovery decisions | +| Filesystem profile admission | Which observed configurations satisfy the supported contract | Caller-selected claims of safety, best-effort fallback | + +The platform boundary is an internal infrastructure interface, not a domain +port pretending that POSIX descriptors are domain objects. Descriptor types +and capability-library types stay on the infrastructure side. Domain ports +continue to exchange semantic values. Existing codec-bearing protocol ports +are not a precedent for moving filesystem types into the domain. + +## Why a factory is not the main abstraction + +A factory can choose an implementation, but returning four unrelated objects +would let their locks, root handles, filesystem observations, or recovery state +diverge. It would also duplicate the operating-system work in four adapters. + +Use one admitted root and retained writer authority as the construction root. +An initialization or reopen operation establishes that authority; subsequent +protocol adapters consume it or borrow it exclusively. Migration changes the +permitted session state. It does not mint another independent writer. + +Keep's existing `FilesystemPlatformAdmission` and version-two admission values +already establish much of this ownership discipline. Preserve those public +entry points during the implementation. A convenience `FilesystemStore` facade +is justified only if it removes repeated consumer orchestration; it must not +be required merely to select Linux versus macOS. + +Backend selection occurs once through a native-platform alias. Adding a new +OS requires a platform implementation, its selection entry, and conformance +evidence. It must not require another catalog, retention, migration, or +recovery state machine. Adding another filesystem on the same OS ordinarily +requires a profile and its evidence, not another implementation of every port. + +## Platform interface + +The implementation separates shared profile admission from native observations. +The interface has these semantic responsibilities; final Rust names may be +refined without changing the contract: + +| Operation | Required result | +| --- | --- | +| Open a selected root | Pinned readable directory, no symlink traversal | +| Observe a directory | Filesystem kind, writable/local observations, naming behavior, device/mount/object coordinates | +| Observe an existing object | Exact descriptor-bound identity; no path reopening as a substitute | +| Synchronize a regular file | All prior writes reach the platform's documented durable boundary, or an exact error | +| Synchronize a directory | Prior namespace effects reach that boundary, or an exact error | +| Acquire a native lock | Nonblocking kernel exclusion tied to the retained handle; typed contention | +| Validate entry spelling where necessary | The actual namespace entry has the exact protocol spelling | + +Use concrete observation records, checked constructors, and explicit enums for +naming and mount behavior. No `supports_durability: bool` supplied by a caller, +no catch-all string capability map, and no public boolean parameters. + +Common code owns the protocol-directory list, checks every existing child, +and compares its device and mount with the root. Native implementations own +the evidence producers, not the decision to skip a required check. Required +but unreported properties are refusal, not a default value. + +Do not add a trait for every POSIX operation. `cap-std` already provides shared +capability-relative file access, hard links, renames, and directory iteration. +Keep a common Unix implementation for those operations and introduce a native +hook only where semantics differ. No dynamic dispatch, per-operation heap +allocation, or reference-counted writer authority is needed for OS selection. + +## Admission and continued authority + +Admission is ordered as follows: + +1. Pin the root through a no-follow traversal. +2. Observe its supported filesystem profile and every existing protocol child. +3. Refuse unrecognized names, wrong kinds, foreign mounts, and unsupported + naming behavior before creating protocol artifacts. +4. Acquire root and writer-file exclusion, retaining the exact descriptors. +5. Revalidate the namespace and identities under writer authority. +6. Execute initialization, publication, or explicit recovery in existing order. +7. Return success only after all required synchronization succeeds. + +Subsequent protocol admission must not revert to ambient path lookup. Newly +created children are verified through retained parent capabilities before use. +Existing substitution and uncertain-recovery refusals remain mandatory. + +An adapter's successful admission does not prove that the operator has not +shared a block device with another host, or that hardware honestly implements +flush commands. Local exclusive administration remains a deployment condition. +There is no claim of exclusion against arbitrary uncooperative mutation. + +## Linux profiles + +The first implementation preserves existing ext4 behavior: writable, +non-casefolded directories, no symlink traversal, and one device/mount across +the protocol namespace. `openat2` and `statx` remain native implementation +details. Missing required `STATX_MNT_ID` continues to refuse in production. + +Do not promote every filesystem with a working `fsync` to supported status. + +| Candidate | Design treatment | Admission requirement | +| --- | --- | --- | +| ext4 | Preserve current profile | Existing laws plus adapter-equivalence regression evidence | +| XFS | First additional Linux candidate | Separate profile, real mounted-filesystem publication/recovery and sync evidence | +| Btrfs | Separate candidate | Subvolume identity, rename/link restrictions, and durability evidence | +| tmpfs | Test-only, not durable storage | Never admitted merely because synchronization returns success | +| OverlayFS | Not admitted by this change | Copy-up identity and backing-filesystem semantics require a distinct contract | +| NFS, SMB, FUSE, unknown types | Refuse | A future explicit contract and evidence, not a permissive fallback | + +XFS and Btrfs consideration is part of this design; enabling them without +their evidence is not part of native macOS acceptance. The interface must +allow adding either without editing the protocol orchestrators. + +## macOS profiles + +The intended native target is local writable APFS, including the ordinary +case-insensitive variant and case-sensitive APFS. HFS+ and other filesystems +remain explicit future profiles. APFS snapshots and read-only mounts refuse +writer admission. + +Root traversal uses descriptor-relative, no-follow component opens. Handle +identity uses Darwin observations; it must not pretend that device identity +is independently observed mount identity. Symbolic links in the selected path +remain a refusal. The documentation must explain that `/tmp` on macOS is a +symlink and use the actual selected directory instead of silently resolving it. + +Case-insensitive APFS needs an explicit naming contract. Keep-generated names +are canonical ASCII; their grammar must be injective under APFS name matching. +The actual names observed during namespace admission must match the grammar +byte-for-byte. `HEAD` must not be admitted through a foreign `head` entry, +and uppercase digest names must not silently alias canonical lowercase names. +The same obligation applies to reads, recovery, and publication reuse, not +only to store initialization. Implement and test exact-spelling checks before +admitting this profile. Unicode normalization is irrelevant to generated names +but must not turn malformed foreign names into admitted protocol entries. + +Regular-file synchronization preserves the pinned Rust toolchain's full-flush +semantics. Rust 1.96's Apple `File::sync_all` calls `F_FULLFSYNC`. Directory +synchronization requires native validation of the same guarantee; do not +catch `EINVAL` or `ENOTSUP` and return success, and do not silently substitute +plain `fsync` for a stronger promised operation. + +The native adapter must use safe existing wrappers where available. If a +required descriptor-bound observation has no safe wrapper, isolate the minimal +FFI in a dedicated crate with documented invariants and boundary tests. Adding +a broad platform dependency merely for one syscall is not the default choice. + +## Physical coordinates and durable compatibility + +Content identities, canonical segment/catalog bytes, and storage profiles do +not depend on the OS. Existing Linux bytes and golden fixtures remain stable. + +Physical recovery coordinates are separate from logical content identity. +The current migration record stores device, mount, and file coordinates as +three big-endian `u64` values. Current main treats mount identity as live +same-process evidence and compares device/inode for restart. Preserve that +distinction; never replace a persisted intent merely because a mount number +changed after restart. + +Document a named Darwin coordinate mapping and its validation before emitting +such records. It must be deterministic, checked, and must reject copied roots. +Do not hash arbitrary OS structs or truncate native fields. If the existing +wire grammar cannot distinguish a necessary coordinate domain, make a +versioned format decision rather than silently reinterpreting existing bytes. + +Cross-OS movement of a physical store is not ordinary reopen. A store whose +recovery evidence binds it to another physical root must refuse. Logical +content export/import remains independent of those coordinates. + +## Failure model + +The platform adapter cannot skip or combine named durability phases. An I/O +failure retains its source and protocol phase, and no success receipt is +created. Dropping an adapter releases resources; it does not commit or repair. + +Publication retains the existing forward ordering and immutable artifacts. +Every interrupted prefix must reopen to the documented snapshot or explicit +recovery/refusal. Failure during recovery remains uncertain until re-observed. +Unknown filesystems, absent observation fields, wrong kinds, alias spellings, +foreign mounts, failed flushes, and contested locks all refuse. + +The existing deterministic process-death harness is reusable, but a harness +that bypasses production platform admission cannot establish native support. +Native acceptance must enter through the ordinary public production path. +Process death is not a physical power-loss experiment. + +## Implementation and acceptance + +The work proceeds through independently working checkpoints, without enabling +an unfinished backend in production: + +1. Extract native mechanism observation and shared admission policy while + preserving Linux acceptance and every public signature. +2. Route OS-dependent durability and authority operations through that boundary. +3. Implement Darwin observations and exact namespace semantics, with typed + refusal tests and a bounded native test environment. +4. Admit APFS only after production initialization, publication, reopen, reads, + lock exclusion, corruption refusal, and recovery tests pass natively. +5. Run Linux regression validation, native debug/release checks, formatting, + strict Clippy, dependency policy, and documentation checks. Add repeatable + macOS CI coverage with explicit supported filesystem profiles. + +An independently useful new adapter must pass the same semantic conformance +laws. Test both success and the boundary failures: symlinked roots/children, +case aliases, missing fields, read-only mounts, foreign devices/mounts, +replaced lock files, copied roots, failed file/directory synchronization, and +each affected crash/recovery prefix. Calibrate refusal assertions against +deliberately weakened evidence, separately from product execution results. + +Build output is bounded within the aggregate 20 GiB Keep budget. Test stores, +scratch data, and logs share the separate 4 GiB budget, including a 128 MiB log +ceiling. Use one serialized reusable cache per incompatible OS toolchain; +native validation cannot reuse Linux object files. Every launch records actual +usage, free space, lock, enforced limits, timeout, and descendant cleanup. + +The implementation is complete only when the design is reflected in code, +Linux behavior remains verified, native production APFS paths are exercised, +and copies of this design and its decision record have intact Reader receipts. + +## Sources and related contracts + +- [Hexagonal boundaries](https://github.com/flyingrobots/keep/blob/3887f2a/docs/adr/0004-hexagonal-boundary-architecture.md). +- [Platform adapter decision](../adr/0010-filesystem-platform-adapters.md). +- [Current filesystem contract](https://github.com/flyingrobots/keep/blob/3887f2a/docs/formats/segment-store-v1/recovery.md#platform-contract). +- [Migration recovery](https://github.com/flyingrobots/keep/blob/3887f2a/docs/formats/segment-store-v2/migration-recovery.md). +- [Apple APFS naming variants](https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html). +- [Apple file synchronization](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fsync.2.html). +- [Apple descriptor controls](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html). +- [Linux ext4 format documentation](https://www.kernel.org/doc/html/latest/filesystems/ext4/index.html). + +Apple's archived manual pages describe the syscall contract; actual APFS +support still requires native execution evidence on the recorded OS version. From 7c68122fd9d4e29219a040f58277783527300c9e Mon Sep 17 00:00:00 2001 From: James Ross Date: Mon, 2 Nov 2026 09:08:59 -0800 Subject: [PATCH 2/7] Refactor: isolate native filesystem admission behind a platform interface --- .../filesystem-platform-adapters.md | 94 +++++ scripts/native/check-worker.sh | 32 ++ scripts/native/check.sh | 83 +++++ .../filesystem_mount_admission_tests.rs | 51 +++ src/adapters/filesystem_mount_refusal.rs | 26 ++ src/adapters/filesystem_platform_profile.rs | 325 ++++-------------- .../filesystem_platform_profile_tests.rs | 40 +-- src/adapters/linux_filesystem.rs | 184 ++++++++++ src/adapters/mod.rs | 7 + src/adapters/native_filesystem.rs | 7 + src/adapters/platform_filesystem.rs | 25 ++ src/adapters/unsupported_filesystem.rs | 50 +++ 12 files changed, 621 insertions(+), 303 deletions(-) create mode 100644 docs/testing-evidence/filesystem-platform-adapters.md create mode 100644 scripts/native/check-worker.sh create mode 100644 scripts/native/check.sh create mode 100644 src/adapters/filesystem_mount_admission_tests.rs create mode 100644 src/adapters/filesystem_mount_refusal.rs create mode 100644 src/adapters/linux_filesystem.rs create mode 100644 src/adapters/native_filesystem.rs create mode 100644 src/adapters/platform_filesystem.rs create mode 100644 src/adapters/unsupported_filesystem.rs diff --git a/docs/testing-evidence/filesystem-platform-adapters.md b/docs/testing-evidence/filesystem-platform-adapters.md new file mode 100644 index 00000000..8b69d25b --- /dev/null +++ b/docs/testing-evidence/filesystem-platform-adapters.md @@ -0,0 +1,94 @@ +# Filesystem platform adapter evidence + +This is an implementation record, not platform certification. The design is +in [filesystem platforms](../architecture/filesystem-platforms.md). + +## Claims and current status + +- Product: native production storage on ordinary and case-sensitive APFS. + **Pending implementation and native runtime evidence.** +- Product: Linux ext4 admission, authority, and durability laws survive the + extraction of the native mechanism boundary. **Runtime validation pending.** +- Static/API: focused protocol ports and public constructors remain available. + Source inspection establishes the intended preservation; compilation pending. +- Tool: bounded native compilation without writes outside admitted outputs. + Runner implemented; execution and guard calibration pending. +- Documentation: architecture and decision record delivered unchanged to Reader. + Receipt `f8a83855-f07e-4622-ac91-25b5bcf72b34`, `awaiting_filing`, + `integrity: intact`, for design commit `ba99881`. + +The author owns the evidence; repository owner `@flyingrobots` owns policy. +No missing gate is treated as an approved waiver. + +## Native mechanism observations + +On the workstation's existing writable APFS data volume, read-only descriptor +observations established that `README.md` and `readme.md` resolve to the same +device/inode. `F_GETPATH` returned the actual stored spelling in both cases. +A bounded one-byte fixture with two hard links returned each opened link's +actual spelling, including when one was opened using an uppercase alias. +The fixture was removed. `F_FULLFSYNC` on an open repository directory returned +success. These observations motivate the adapter; they are not permanent +production tests, a concurrency proof, or a power-loss experiment. + +The pinned Rust 1.96 standard library's Apple implementation of +`File::sync_all` calls `F_FULLFSYNC`. A new adapter must not weaken that call. + +## Native compilation launch contract + +The initial compiler-only runner is `scripts/native/check.sh`. It does not +launch test executables or crash campaigns. Its admitted child tree is Cargo, +the pinned compiler, linker, and admitted dependency build scripts; workloads +that detach process groups are excluded from this runner. + +- Reused Linux worker: `keep-validation`, image `rust:1.96.0`; idle at inspection. + The stopped historical documentation worker is `keep114-closure-docs`. +- Workstation lock store: `/Users/Shared/git-locks/workstation.git`. + Hold `host/cache/keep-native-validation/`, `host/docker/keep-validation/`, and + `host/docker/keep114-closure-docs/` together for project-budget admission. +- Measured Docker writable layers: 19,150,114,816 and 271,781,888 bytes. + Counting their entire layers conservatively gives 19,421,896,704 bytes. +- Native build allocation: one reusable 1.5 GiB APFS sparse image, mounted under + `$HOME/Library/Caches/keep-native-validation/build`. Target and Cargo-local + metadata are the only compiler-writable build paths. Existing registry + contents and toolchains are read-only inputs. No second target is permitted. +- The conservative existing-layer total plus the full native build capacity + is 21,032,509,440 bytes, below the 21,474,836,480-byte project build budget. +- Native scratch: a separate 256 MiB APFS image at the adjacent `data` mount; + `TMPDIR` points to its `tmp` directory. It is scratch, not durability evidence. +- Logs: adjacent owned `logs/`; each logger has a 4 MiB file-size ceiling. + Launch refuses above 20 MiB already retained, bounding native logs at 24 MiB. + Existing worker log rotation permits at most 100 MiB; combined ceilings stay + below 128 MiB. Evidence must be exported before owned logs are recycled. +- Disk admission observed about 557 GiB host and 530 GiB Docker backing free. + The native run monitors host free space and stops below 50 GiB. It does not + generate Docker output while holding both worker locks. +- Hard filesystem capacities bound build and temporary output. The compiler + sandbox denies network and all filesystem writes except target, Cargo-local + metadata, scratch, and `/dev/null`; its log writer is separately size-limited. +- Two Cargo jobs; inherited 900-second per-process CPU ceiling and 4 GiB + per-process virtual-address-space ceiling. The parent monitors aggregate + process-group RSS and refuses above 6 GiB. Whole-command deadline: 1,200 seconds. +- The compiler tree has a fresh process group. Limit breach, monitor failure, + interruption, or exit kills the owned group and waits for its leader. + The reservation must exceed the deadline and cleanup time. + +The native images are retained reusable caches, not per-revision artifacts. +Their measured usage and execution receipts must be added after validation. +This contract does not authorize another cache when one fills. + +## Oracles and remaining acceptance + +Shared mount-admission laws compare exact typed device/mount contradictions +and permit a child inode distinct from its parent on the same mount. Existing +Linux profile laws retain writable ext4 and casefold rejection. The old combined +child-policy unit law is separated into native profile laws and common mount +laws because those responsibilities now have different owners. Production +directory traversal and child-admission coverage still require execution. + +Native product acceptance must exercise public production admission, exact +content reads, publication, retention, migration/reopen, writer exclusion, +case aliases, malformed namespaces, substitutions, and interrupted recovery. +Both APFS naming variants require actual runs. Debug/release, strict Clippy, +dependency checks, Linux regressions, CI configuration, and replayable crash +evidence remain pending. No native support or regression-free claim is made. diff --git a/scripts/native/check-worker.sh b/scripts/native/check-worker.sh new file mode 100644 index 00000000..e20595b2 --- /dev/null +++ b/scripts/native/check-worker.sh @@ -0,0 +1,32 @@ +#!/usr/bin/env bash +# Compiler-only child: all writable locations are on the bounded native images. +set -euo pipefail +readonly source_dir="$1" +readonly build_dir="$2" +readonly scratch_dir="$3" +readonly log_path="$4" +readonly sandbox_profile="$5" + +export CARGO_HOME="$build_dir/cargo-home" +export CARGO_TARGET_DIR="$build_dir/target" +export CARGO_INCREMENTAL=0 +export CARGO_BUILD_JOBS=2 +export CARGO_NET_OFFLINE=true +export CARGO_PROFILE_DEV_DEBUG=0 +export CARGO_PROFILE_TEST_DEBUG=0 +export TMPDIR="$scratch_dir/" +unset RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER +cd "$source_dir" + +# The monitor caps aggregate RSS; these inherited ceilings also bound each +# compiler process independently. No test executable is launched by this tool. +( + ulimit -t 900 + ulimit -v 4194304 + sandbox-exec -f "$sandbox_profile" \ + cargo check --lib --all-features --locked --offline +) 2>&1 | ( + # This limit applies to the log writer, not compiler artifact sizes. + ulimit -f 4096 + cat > "$log_path" +) diff --git a/scripts/native/check.sh b/scripts/native/check.sh new file mode 100644 index 00000000..f7360fb7 --- /dev/null +++ b/scripts/native/check.sh @@ -0,0 +1,83 @@ +#!/usr/bin/env bash +# Bounded native compilation. Invoke under the workstation resource locks. +set -euo pipefail + +readonly source_dir="$(cd -- "${1:?usage: check.sh CHECKOUT}" && pwd)" +readonly native_root="${KEEP_NATIVE_ROOT:-$HOME/Library/Caches/keep-native-validation}" +readonly build_dir="$native_root/build" +readonly scratch_dir="$native_root/data/tmp" +readonly logs_dir="$native_root/logs" +readonly profile="$native_root/compiler.sb" +readonly log_path="$logs_dir/$(date -u +%Y%m%dT%H%M%SZ)-check.log" +readonly worker="$source_dir/scripts/native/check-worker.sh" +readonly minimum_free_kib=52428800 + +[[ "$(uname -s)" == Darwin && -f "$source_dir/Cargo.toml" ]] +[[ -d "$build_dir/target" && -d "$scratch_dir" && -d "$logs_dir" ]] +# Setup is deliberate and separate. Refuse an absent quota filesystem instead +# of compiling into an ordinary directory left behind by an unmount. +[[ "$(df -Pk "$build_dir" | awk 'END {print $6}')" == "$build_dir" ]] +[[ "$(df -Pk "$scratch_dir" | awk 'END {print $6}')" == "$native_root/data" ]] +[[ "$(df -Pk "$build_dir" | awk 'END {print $2}')" -le 1572864 ]] +[[ "$(df -Pk "$scratch_dir" | awk 'END {print $2}')" -le 262144 ]] +[[ "$(df -Pk "$source_dir" | awk 'END {print $4}')" -ge "$minimum_free_kib" ]] +[[ "$(du -sk "$logs_dir" | awk '{print $1}')" -lt 20480 ]] + +cat > "$profile" </dev/null || true + wait "$task_pid" 2>/dev/null || true + fi +} +trap stop_child EXIT INT TERM + +# setsid gives the compiler tree one owned process group. This runner admits +# compilation only, excluding test harnesses that create their own groups. +/usr/bin/perl -MPOSIX=setsid -e \ + 'setsid() >= 0 or die "setsid failed"; exec @ARGV or die "exec failed";' \ + /bin/bash "$worker" "$source_dir" "$build_dir" "$scratch_dir" \ + "$log_path" "$profile" & +task_pid=$! +readonly task_started=$SECONDS +failure='' +while kill -0 "$task_pid" 2>/dev/null; do + if ! process_snapshot="$(ps -axo pgid=,rss=)"; then + failure='process monitor failed' + break + fi + task_rss="$(awk -v group="$task_pid" '$1 == group {sum += $2} END {print sum+0}' <<< "$process_snapshot")" + if ! free_kib="$(df -Pk "$source_dir" | awk 'END {print $4}')"; then + failure='disk monitor failed' + break + fi + if (( task_rss > 6291456 || SECONDS - task_started > 1200 || free_kib < minimum_free_kib )); then + failure='memory, timeout, or free-space guard refused' + break + fi + sleep 1 +done + +if [[ -n "$failure" ]]; then + printf '%s\n' "$failure" >&2 + exit 1 +fi +result=0 +wait "$task_pid" || result=$? +stop_child +task_pid='' +tail -60 "$log_path" +printf 'Native compilation log: %s\n' "$log_path" +exit "$result" diff --git a/src/adapters/filesystem_mount_admission_tests.rs b/src/adapters/filesystem_mount_admission_tests.rs new file mode 100644 index 00000000..6a609ce5 --- /dev/null +++ b/src/adapters/filesystem_mount_admission_tests.rs @@ -0,0 +1,51 @@ +//! Device and mount separation laws, independent of the host operating system. +//! +//! The oracle is the platform contract's single-device/single-mount rule; +//! child inode identity is deliberately not required to equal the root inode. + +use std::{error::Error, io}; + +use super::admit_child_identity; +use crate::adapters::filesystem_mount_refusal::FilesystemMountRefusal; +use crate::adapters::filesystem_root_identity::FilesystemRootIdentity; + +#[test] +fn a_distinct_child_inode_on_the_same_mount_is_admitted() -> Result<(), Box> { + let root = FilesystemRootIdentity::new(8, 41, 73); + let child = FilesystemRootIdentity::new(8, 41, 74); + admit_child_identity(root, child)?; + Ok(()) +} + +#[test] +fn a_child_on_another_device_is_refused() { + let root = FilesystemRootIdentity::new(8, 41, 73); + let child = FilesystemRootIdentity::new(9, 41, 74); + assert_refusal( + admit_child_identity(root, child), + FilesystemMountRefusal::Device { + expected: 8, + observed: 9, + }, + ); +} + +#[test] +fn a_child_on_another_mount_of_the_same_device_is_refused() { + let root = FilesystemRootIdentity::new(8, 41, 73); + let child = FilesystemRootIdentity::new(8, 42, 74); + assert_refusal( + admit_child_identity(root, child), + FilesystemMountRefusal::Mount { + expected: 41, + observed: 42, + }, + ); +} + +fn assert_refusal(result: io::Result<()>, expected: FilesystemMountRefusal) { + assert!(matches!(result, Err(source) + if source.kind() == io::ErrorKind::Unsupported + && source.get_ref().and_then(|error| error.downcast_ref::()) + == Some(&expected))); +} diff --git a/src/adapters/filesystem_mount_refusal.rs b/src/adapters/filesystem_mount_refusal.rs new file mode 100644 index 00000000..59da7c10 --- /dev/null +++ b/src/adapters/filesystem_mount_refusal.rs @@ -0,0 +1,26 @@ +//! This module owns observed device and mount contradictions during admission. + +use std::{error::Error, fmt}; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum FilesystemMountRefusal { + Device { expected: u64, observed: u64 }, + Mount { expected: u64, observed: u64 }, +} + +impl fmt::Display for FilesystemMountRefusal { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Device { expected, observed } => write!( + formatter, + "protocol directory device is {observed}, expected {expected}" + ), + Self::Mount { expected, observed } => write!( + formatter, + "protocol directory mount is {observed}, expected {expected}" + ), + } + } +} + +impl Error for FilesystemMountRefusal {} diff --git a/src/adapters/filesystem_platform_profile.rs b/src/adapters/filesystem_platform_profile.rs index 896ec543..19ac8ebb 100644 --- a/src/adapters/filesystem_platform_profile.rs +++ b/src/adapters/filesystem_platform_profile.rs @@ -1,21 +1,16 @@ -//! This module owns fail-closed filesystem platform-profile admission. +//! This module owns shared filesystem profile and protocol-directory admission. -use std::io; -use std::path::Path; +use std::{io, path::Path}; use cap_std::fs::Dir; +use super::filesystem_mount_refusal::FilesystemMountRefusal; use super::filesystem_root_identity::FilesystemRootIdentity; +use super::native_filesystem::NativeFilesystem; +use super::platform_filesystem::PlatformFilesystem; -#[cfg(target_os = "linux")] -const PROTOCOL_DIRECTORIES: [&str; 3] = ["staging", "segments", "catalogs"]; -/// Every version-two protocol directory, including nested immutable pools. -/// -/// A migrated root receives writer authority only when each of these shares -/// the root's filesystem type, device, and mount identity and is not -/// casefolded or read-only. -#[cfg(target_os = "linux")] -const VERSION_TWO_PROTOCOL_DIRECTORIES: [&str; 9] = [ +pub(super) const PROTOCOL_DIRECTORIES: [&str; 3] = ["staging", "segments", "catalogs"]; +pub(super) const VERSION_TWO_PROTOCOL_DIRECTORIES: [&str; 9] = [ "staging", "segments", "catalogs", @@ -27,108 +22,61 @@ const VERSION_TWO_PROTOCOL_DIRECTORIES: [&str; 9] = [ "recovery/dispositions", ]; -#[cfg(target_os = "linux")] -#[derive(Clone, Copy)] -struct LinuxDirectoryProperties { - filesystem_type: rustix::fs::FsWord, - mount_flags: rustix::fs::StatVfsMountFlags, - inode_flags: u32, - device_major: u32, - device_minor: u32, - mount_id: u64, -} - -#[cfg(target_os = "linux")] pub(super) fn open(store_root: &Path) -> io::Result { - use std::fs::File; - - use rustix::fs::{CWD, Mode, OFlags, ResolveFlags, openat2}; + open_profile(&NativeFilesystem, store_root, &PROTOCOL_DIRECTORIES) +} - let descriptor = openat2( - CWD, +pub(super) fn open_version_two(store_root: &Path) -> io::Result { + open_profile( + &NativeFilesystem, store_root, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::CLOEXEC, - Mode::empty(), - ResolveFlags::NO_MAGICLINKS | ResolveFlags::NO_SYMLINKS, - )?; - let directory = Dir::from_std_file(File::from(descriptor)); - admit_linux_profile(&directory, &PROTOCOL_DIRECTORIES)?; - Ok(directory) + &VERSION_TWO_PROTOCOL_DIRECTORIES, + ) } -/// Applies the production profile to the root already pinned by writer authority. -#[cfg(all(target_os = "linux", feature = "repository-tasks"))] +#[cfg(feature = "repository-tasks")] pub(super) fn admit_locked_root(directory: &Dir) -> io::Result { - admit_linux_profile(directory, &PROTOCOL_DIRECTORIES)?; + admit_profile(&NativeFilesystem, directory, &PROTOCOL_DIRECTORIES)?; root_identity(directory) } -#[cfg(all(not(target_os = "linux"), feature = "repository-tasks"))] -pub(super) fn admit_locked_root(_directory: &Dir) -> io::Result { - Err(io::Error::new( - io::ErrorKind::Unsupported, - "catalog publication requires the admitted Linux ext4 profile", - )) -} - -/// Opens one version-two store root under the admitted Linux profile. -/// -/// Identical to [`open`], but every version-two protocol directory that -/// exists must satisfy the same filesystem, mount, and inode-flag laws as the -/// root; absence is left to namespace admission. -#[cfg(target_os = "linux")] -pub(super) fn open_version_two(store_root: &Path) -> io::Result { - use std::fs::File; - - use rustix::fs::{CWD, Mode, OFlags, ResolveFlags, openat2}; - - let descriptor = openat2( - CWD, - store_root, - OFlags::RDONLY | OFlags::DIRECTORY | OFlags::CLOEXEC, - Mode::empty(), - ResolveFlags::NO_MAGICLINKS | ResolveFlags::NO_SYMLINKS, - )?; - let directory = Dir::from_std_file(File::from(descriptor)); - admit_linux_profile(&directory, &VERSION_TWO_PROTOCOL_DIRECTORIES)?; - Ok(directory) +pub(super) fn root_identity(directory: &Dir) -> io::Result { + NativeFilesystem.root_identity(directory) } -#[cfg(not(target_os = "linux"))] -pub(super) fn open(_store_root: &Path) -> io::Result { - Err(io::Error::new( - io::ErrorKind::Unsupported, - "filesystem initialization currently requires the admitted Linux ext4 profile", - )) +#[cfg(any(test, feature = "repository-tasks"))] +pub(super) fn root_identity_lenient(directory: &Dir) -> io::Result { + NativeFilesystem.repository_identity(directory) } -#[cfg(not(target_os = "linux"))] -pub(super) fn open_version_two(_store_root: &Path) -> io::Result { - Err(io::Error::new( - io::ErrorKind::Unsupported, - "version-two reopen currently requires the admitted Linux ext4 profile", - )) +fn open_profile( + platform: &impl PlatformFilesystem, + store_root: &Path, + protocol_directories: &[&str], +) -> io::Result { + let directory = platform.open_root(store_root)?; + admit_profile(platform, &directory, protocol_directories)?; + Ok(directory) } -#[cfg(target_os = "linux")] -fn admit_linux_profile(directory: &Dir, protocol_directories: &[&str]) -> io::Result<()> { - let file = directory.try_clone()?.into_std_file(); - let root = linux_directory_properties(&file)?; - admit_linux_properties(root.filesystem_type, root.mount_flags, root.inode_flags)?; +fn admit_profile( + platform: &impl PlatformFilesystem, + directory: &Dir, + protocol_directories: &[&str], +) -> io::Result<()> { + let root = platform.admit_directory(directory)?; for name in protocol_directories { let child = match open_protocol_directory(directory, name) { Ok(child) => child, Err(source) if source.kind() == io::ErrorKind::NotFound => continue, Err(source) => return Err(source), }; - let child = linux_directory_properties(&child.into_std_file())?; - admit_linux_child_properties(root, child)?; + let child = platform.admit_directory(&child)?; + admit_child_identity(root, child)?; } - file.sync_all() + directory.try_clone()?.into_std_file().sync_all() } -/// Opens a possibly nested protocol directory one no-follow component at a time. -#[cfg(target_os = "linux")] fn open_protocol_directory(root: &Dir, name: &str) -> io::Result { let mut components = name.split('/'); let first = components @@ -143,186 +91,31 @@ fn open_protocol_directory(root: &Dir, name: &str) -> io::Result { Ok(current) } -#[cfg(target_os = "linux")] -fn linux_directory_properties(file: &std::fs::File) -> io::Result { - use rustix::fs::{AtFlags, StatxFlags, fstatfs, fstatvfs, ioctl_getflags, statx}; - - let filesystem = fstatfs(file)?; - let mount = fstatvfs(file)?; - let inode_flags = ioctl_getflags(file)?; - let required = StatxFlags::BASIC_STATS | StatxFlags::MNT_ID; - let status = statx(file, ".", AtFlags::empty(), required)?; - let observed = StatxFlags::from_bits_retain(status.stx_mask); - if !observed.contains(required) { - return Err(unsupported_linux_profile()); - } - Ok(LinuxDirectoryProperties { - filesystem_type: filesystem.f_type, - mount_flags: mount.f_flag, - inode_flags: inode_flags.bits(), - device_major: status.stx_dev_major, - device_minor: status.stx_dev_minor, - mount_id: status.stx_mnt_id, - }) -} - -/// Whether an identity probe may proceed when `statx` omits `STATX_MNT_ID`. -/// -/// Production admission requires the mount identity: without it a bind-mounted -/// or relocated root cannot be told apart from the original. The test and -/// repository-task bypass records an unreported mount identity as zero instead, -/// so the suite runs on kernels older than 5.8 while every production probe -/// stays strict. -#[cfg(any(target_os = "linux", test))] -#[derive(Clone, Copy, Debug, Eq, PartialEq)] -enum MountIdentityPolicy { - Required, - #[cfg(any(test, feature = "repository-tasks"))] - Lenient, -} - -/// Selects the recorded mount identity from what `statx` reported. -/// -/// Returns `None` exactly when the policy requires a mount identity the kernel -/// did not report. -#[cfg(any(target_os = "linux", test))] -const fn admit_mount_identity( - policy: MountIdentityPolicy, - reported: bool, - mount_id: u64, -) -> Option { - match (policy, reported) { - (_, true) => Some(mount_id), - (MountIdentityPolicy::Required, false) => None, - #[cfg(any(test, feature = "repository-tasks"))] - (MountIdentityPolicy::Lenient, false) => Some(0), - } -} - -#[cfg(target_os = "linux")] -pub(super) fn root_identity(directory: &Dir) -> io::Result { - let file = directory.try_clone()?.into_std_file(); - linux_file_identity(&file, MountIdentityPolicy::Required) -} - -/// Probes root identity for the test and repository-task admission bypass. -/// -/// Identical to [`root_identity`] except that an unreported mount identity is -/// recorded as zero instead of refusing; production admission never uses it. -#[cfg(all(target_os = "linux", any(test, feature = "repository-tasks")))] -pub(super) fn root_identity_lenient(directory: &Dir) -> io::Result { - let file = directory.try_clone()?.into_std_file(); - linux_file_identity(&file, MountIdentityPolicy::Lenient) -} - -#[cfg(target_os = "linux")] -fn linux_file_identity( - file: &std::fs::File, - policy: MountIdentityPolicy, -) -> io::Result { - use rustix::fs::{AtFlags, StatxFlags, statx}; - - let requested = StatxFlags::BASIC_STATS | StatxFlags::MNT_ID; - let status = statx(file, ".", AtFlags::empty(), requested)?; - let observed = StatxFlags::from_bits_retain(status.stx_mask); - if !observed.contains(StatxFlags::BASIC_STATS) { - return Err(unsupported_linux_profile()); - } - let reported = observed.contains(StatxFlags::MNT_ID); - let mount_id = admit_mount_identity(policy, reported, status.stx_mnt_id) - .ok_or_else(unsupported_linux_profile)?; - Ok(linux_root_identity( - status.stx_dev_major, - status.stx_dev_minor, - mount_id, - status.stx_ino, - )) -} - -#[cfg(target_os = "linux")] -fn linux_root_identity( - device_major: u32, - device_minor: u32, - mount_id: u64, - inode: u64, -) -> FilesystemRootIdentity { - let device = rustix::fs::makedev(device_major, device_minor); - FilesystemRootIdentity::new(device, mount_id, inode) -} - -#[cfg(all(not(target_os = "linux"), any(test, feature = "repository-tasks")))] -pub(super) fn root_identity(directory: &Dir) -> io::Result { - use cap_fs_ext::MetadataExt; - - let metadata = directory.dir_metadata()?; - Ok(FilesystemRootIdentity::new( - metadata.dev(), - metadata.dev(), - metadata.ino(), - )) -} - -#[cfg(all(not(target_os = "linux"), any(test, feature = "repository-tasks")))] -pub(super) fn root_identity_lenient(directory: &Dir) -> io::Result { - root_identity(directory) -} - -#[cfg(all(not(target_os = "linux"), not(any(test, feature = "repository-tasks"))))] -pub(super) fn root_identity(_directory: &Dir) -> io::Result { - Err(io::Error::new( - io::ErrorKind::Unsupported, - "filesystem root identity currently requires the admitted Linux ext4 profile", - )) -} - -#[cfg(target_os = "linux")] -fn admit_linux_properties( - filesystem_type: rustix::fs::FsWord, - mount_flags: rustix::fs::StatVfsMountFlags, - inode_flags: u32, +pub(super) fn admit_child_identity( + root: FilesystemRootIdentity, + child: FilesystemRootIdentity, ) -> io::Result<()> { - // These values are the Linux UAPI ext4 superblock magic and per-directory - // casefold inode flag. Keeping them local makes the admitted profile - // visible at the exact decision boundary. - const EXT4_SUPER_MAGIC: rustix::fs::FsWord = 0x0000_ef53; - const EXT4_CASEFOLD_FLAG: u32 = 0x4000_0000; - - if filesystem_type != EXT4_SUPER_MAGIC - || mount_flags.contains(rustix::fs::StatVfsMountFlags::RDONLY) - || inode_flags & EXT4_CASEFOLD_FLAG != 0 - { - return Err(unsupported_linux_profile()); + if root.device() != child.device() { + return Err(io::Error::new( + io::ErrorKind::Unsupported, + FilesystemMountRefusal::Device { + expected: root.device(), + observed: child.device(), + }, + )); } - Ok(()) -} - -#[cfg(target_os = "linux")] -fn admit_linux_child_properties( - root: LinuxDirectoryProperties, - child: LinuxDirectoryProperties, -) -> io::Result<()> { - admit_linux_properties(child.filesystem_type, child.mount_flags, child.inode_flags)?; - if root.device_major != child.device_major - || root.device_minor != child.device_minor - || root.mount_id != child.mount_id - { - return Err(unsupported_linux_profile()); + if root.mount() != child.mount() { + return Err(io::Error::new( + io::ErrorKind::Unsupported, + FilesystemMountRefusal::Mount { + expected: root.mount(), + observed: child.mount(), + }, + )); } Ok(()) } -#[cfg(target_os = "linux")] -fn unsupported_linux_profile() -> io::Error { - io::Error::new( - io::ErrorKind::Unsupported, - "store namespace does not satisfy one local writable case-sensitive ext4 profile", - ) -} - -#[cfg(all(test, target_os = "linux"))] -#[path = "filesystem_platform_profile_tests.rs"] -mod tests; - #[cfg(test)] -#[path = "filesystem_platform_profile_policy_tests.rs"] -mod policy_tests; +#[path = "filesystem_mount_admission_tests.rs"] +mod tests; diff --git a/src/adapters/filesystem_platform_profile_tests.rs b/src/adapters/filesystem_platform_profile_tests.rs index 5ae74201..96554d07 100644 --- a/src/adapters/filesystem_platform_profile_tests.rs +++ b/src/adapters/filesystem_platform_profile_tests.rs @@ -1,8 +1,8 @@ //! Linux filesystem platform-profile laws. -use super::{ - LinuxDirectoryProperties, PROTOCOL_DIRECTORIES, VERSION_TWO_PROTOCOL_DIRECTORIES, - admit_linux_child_properties, admit_linux_properties, linux_root_identity, +use super::{admit_linux_properties, linux_root_identity}; +use crate::adapters::filesystem_platform_profile::{ + PROTOCOL_DIRECTORIES, VERSION_TWO_PROTOCOL_DIRECTORIES, }; use rustix::fs::{NFS_SUPER_MAGIC, StatVfsMountFlags}; @@ -37,25 +37,6 @@ fn only_writable_case_sensitive_ext4_is_admitted() { )); } -#[test] -fn every_protocol_child_must_share_the_root_filesystem_and_mount() { - assert_eq!(PROTOCOL_DIRECTORIES, ["staging", "segments", "catalogs"]); - let root = properties(8, 1, 41); - let mut casefolded = root; - casefolded.inode_flags = EXT4_CASEFOLD_FLAG; - let mut read_only = root; - read_only.mount_flags = StatVfsMountFlags::RDONLY; - let mut foreign_format = root; - foreign_format.filesystem_type = NFS_SUPER_MAGIC; - - assert!(admit_linux_child_properties(root, root).is_ok()); - assert_unsupported(&admit_linux_child_properties(root, properties(8, 2, 41))); - assert_unsupported(&admit_linux_child_properties(root, properties(8, 1, 42))); - assert_unsupported(&admit_linux_child_properties(root, casefolded)); - assert_unsupported(&admit_linux_child_properties(root, read_only)); - assert_unsupported(&admit_linux_child_properties(root, foreign_format)); -} - #[test] fn root_identity_uses_linux_device_mount_and_inode_coordinates() { let identity = linux_root_identity(8, 1, 41, 73); @@ -74,21 +55,6 @@ fn assert_unsupported(result: &std::io::Result<()>) { )); } -const fn properties( - device_major: u32, - device_minor: u32, - mount_id: u64, -) -> LinuxDirectoryProperties { - LinuxDirectoryProperties { - filesystem_type: EXT4_SUPER_MAGIC, - mount_flags: StatVfsMountFlags::empty(), - inode_flags: 0, - device_major, - device_minor, - mount_id, - } -} - #[test] fn version_two_admission_covers_every_protocol_directory() { assert_eq!( diff --git a/src/adapters/linux_filesystem.rs b/src/adapters/linux_filesystem.rs new file mode 100644 index 00000000..1e4cd31a --- /dev/null +++ b/src/adapters/linux_filesystem.rs @@ -0,0 +1,184 @@ +//! This module owns Linux descriptor observations and ext4 admission. + +use std::{fs::File, io, path::Path}; + +use cap_std::fs::Dir; +use rustix::fs::{CWD, Mode, OFlags, ResolveFlags, openat2}; + +use super::filesystem_root_identity::FilesystemRootIdentity; +use super::platform_filesystem::PlatformFilesystem; + +pub(super) struct LinuxFilesystem; + +impl PlatformFilesystem for LinuxFilesystem { + fn open_root(&self, store_root: &Path) -> io::Result { + let descriptor = openat2( + CWD, + store_root, + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::CLOEXEC, + Mode::empty(), + ResolveFlags::NO_MAGICLINKS | ResolveFlags::NO_SYMLINKS, + )?; + Ok(Dir::from_std_file(File::from(descriptor))) + } + + fn admit_directory(&self, directory: &Dir) -> io::Result { + let file = directory.try_clone()?.into_std_file(); + let observed = linux_directory_properties(&file)?; + admit_linux_properties( + observed.filesystem_type, + observed.mount_flags, + observed.inode_flags, + )?; + Ok(linux_root_identity( + observed.device_major, + observed.device_minor, + observed.mount_id, + observed.inode, + )) + } + + fn root_identity(&self, directory: &Dir) -> io::Result { + let file = directory.try_clone()?.into_std_file(); + linux_file_identity(&file, MountIdentityPolicy::Required) + } + + #[cfg(any(test, feature = "repository-tasks"))] + fn repository_identity(&self, directory: &Dir) -> io::Result { + let file = directory.try_clone()?.into_std_file(); + linux_file_identity(&file, MountIdentityPolicy::Lenient) + } +} + +#[derive(Clone, Copy)] +struct LinuxDirectoryProperties { + filesystem_type: rustix::fs::FsWord, + mount_flags: rustix::fs::StatVfsMountFlags, + inode_flags: u32, + device_major: u32, + device_minor: u32, + mount_id: u64, + inode: u64, +} + +fn linux_directory_properties(file: &std::fs::File) -> io::Result { + use rustix::fs::{AtFlags, StatxFlags, fstatfs, fstatvfs, ioctl_getflags, statx}; + + let filesystem = fstatfs(file)?; + let mount = fstatvfs(file)?; + let inode_flags = ioctl_getflags(file)?; + let required = StatxFlags::BASIC_STATS | StatxFlags::MNT_ID; + let status = statx(file, ".", AtFlags::empty(), required)?; + let observed = StatxFlags::from_bits_retain(status.stx_mask); + if !observed.contains(required) { + return Err(unsupported_linux_profile()); + } + Ok(LinuxDirectoryProperties { + filesystem_type: filesystem.f_type, + mount_flags: mount.f_flag, + inode_flags: inode_flags.bits(), + device_major: status.stx_dev_major, + device_minor: status.stx_dev_minor, + mount_id: status.stx_mnt_id, + inode: status.stx_ino, + }) +} + +/// Whether an identity probe may proceed when `statx` omits `STATX_MNT_ID`. +/// +/// Production admission requires the mount identity: without it a bind-mounted +/// or relocated root cannot be told apart from the original. The test and +/// repository-task bypass records an unreported mount identity as zero instead, +/// so the suite runs on kernels older than 5.8 while every production probe +/// stays strict. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +enum MountIdentityPolicy { + Required, + #[cfg(any(test, feature = "repository-tasks"))] + Lenient, +} + +/// Selects the recorded mount identity from what `statx` reported. +/// +/// Returns `None` exactly when the policy requires a mount identity the kernel +/// did not report. +const fn admit_mount_identity( + policy: MountIdentityPolicy, + reported: bool, + mount_id: u64, +) -> Option { + match (policy, reported) { + (_, true) => Some(mount_id), + (MountIdentityPolicy::Required, false) => None, + #[cfg(any(test, feature = "repository-tasks"))] + (MountIdentityPolicy::Lenient, false) => Some(0), + } +} + +fn linux_file_identity( + file: &std::fs::File, + policy: MountIdentityPolicy, +) -> io::Result { + use rustix::fs::{AtFlags, StatxFlags, statx}; + + let requested = StatxFlags::BASIC_STATS | StatxFlags::MNT_ID; + let status = statx(file, ".", AtFlags::empty(), requested)?; + let observed = StatxFlags::from_bits_retain(status.stx_mask); + if !observed.contains(StatxFlags::BASIC_STATS) { + return Err(unsupported_linux_profile()); + } + let reported = observed.contains(StatxFlags::MNT_ID); + let mount_id = admit_mount_identity(policy, reported, status.stx_mnt_id) + .ok_or_else(unsupported_linux_profile)?; + Ok(linux_root_identity( + status.stx_dev_major, + status.stx_dev_minor, + mount_id, + status.stx_ino, + )) +} + +fn linux_root_identity( + device_major: u32, + device_minor: u32, + mount_id: u64, + inode: u64, +) -> FilesystemRootIdentity { + let device = rustix::fs::makedev(device_major, device_minor); + FilesystemRootIdentity::new(device, mount_id, inode) +} + +fn admit_linux_properties( + filesystem_type: rustix::fs::FsWord, + mount_flags: rustix::fs::StatVfsMountFlags, + inode_flags: u32, +) -> io::Result<()> { + // These values are the Linux UAPI ext4 superblock magic and per-directory + // casefold inode flag. Keeping them local makes the admitted profile + // visible at the exact decision boundary. + const EXT4_SUPER_MAGIC: rustix::fs::FsWord = 0x0000_ef53; + const EXT4_CASEFOLD_FLAG: u32 = 0x4000_0000; + + if filesystem_type != EXT4_SUPER_MAGIC + || mount_flags.contains(rustix::fs::StatVfsMountFlags::RDONLY) + || inode_flags & EXT4_CASEFOLD_FLAG != 0 + { + return Err(unsupported_linux_profile()); + } + Ok(()) +} + +fn unsupported_linux_profile() -> io::Error { + io::Error::new( + io::ErrorKind::Unsupported, + "store namespace does not satisfy one local writable case-sensitive ext4 profile", + ) +} + +#[cfg(test)] +#[path = "filesystem_platform_profile_tests.rs"] +mod tests; + +#[cfg(test)] +#[path = "filesystem_platform_profile_policy_tests.rs"] +mod policy_tests; diff --git a/src/adapters/mod.rs b/src/adapters/mod.rs index 31a2b951..47dc6630 100644 --- a/src/adapters/mod.rs +++ b/src/adapters/mod.rs @@ -91,6 +91,7 @@ mod filesystem_catalog_storage; mod filesystem_exact_record; mod filesystem_initialization_namespace; mod filesystem_initialization_storage; +mod filesystem_mount_refusal; mod filesystem_namespace_refusal; mod filesystem_platform_admission; mod filesystem_platform_admission_error; @@ -160,12 +161,16 @@ mod layout_record_decoder; mod layout_record_encoder; mod layout_record_format; mod layout_record_framing; +#[cfg(target_os = "linux")] +mod linux_filesystem; mod loaded_segment; mod lower_hex; +mod native_filesystem; #[cfg(feature = "repository-tasks")] mod observed_segment_stage; mod opened_reusable_segment; mod physical_pool_name; +mod platform_filesystem; mod publication_head_decode_error; mod publication_head_decode_error_display; mod publication_head_decoder; @@ -246,6 +251,8 @@ mod sync_capable_directory; #[cfg(test)] #[path = "../../tests/support/mod.rs"] mod test_support; +#[cfg(not(target_os = "linux"))] +mod unsupported_filesystem; mod verification_admission; mod verification_decode_error; mod verification_error; diff --git a/src/adapters/native_filesystem.rs b/src/adapters/native_filesystem.rs new file mode 100644 index 00000000..9076eb28 --- /dev/null +++ b/src/adapters/native_filesystem.rs @@ -0,0 +1,7 @@ +//! This module owns compile-time selection of the native filesystem adapter. + +#[cfg(target_os = "linux")] +pub(super) use super::linux_filesystem::LinuxFilesystem as NativeFilesystem; + +#[cfg(not(target_os = "linux"))] +pub(super) use super::unsupported_filesystem::UnsupportedFilesystem as NativeFilesystem; diff --git a/src/adapters/platform_filesystem.rs b/src/adapters/platform_filesystem.rs new file mode 100644 index 00000000..67d79a20 --- /dev/null +++ b/src/adapters/platform_filesystem.rs @@ -0,0 +1,25 @@ +//! This module owns the native filesystem mechanism interface. +//! +//! This infrastructure boundary stays below the semantic storage ports. Its +//! capability handles never enter domain identity or transition APIs. + +use std::{io, path::Path}; + +use cap_std::fs::Dir; + +use super::filesystem_root_identity::FilesystemRootIdentity; + +pub(super) trait PlatformFilesystem { + /// Pins a readable directory without traversing symbolic links. + fn open_root(&self, path: &Path) -> io::Result; + + /// Requires a supported writable profile and observes its live coordinates. + fn admit_directory(&self, directory: &Dir) -> io::Result; + + /// Observes descriptor-bound coordinates without admitting a new profile. + fn root_identity(&self, directory: &Dir) -> io::Result; + + /// Supplies historical repository-fixture coordinates, never production proof. + #[cfg(any(test, feature = "repository-tasks"))] + fn repository_identity(&self, directory: &Dir) -> io::Result; +} diff --git a/src/adapters/unsupported_filesystem.rs b/src/adapters/unsupported_filesystem.rs new file mode 100644 index 00000000..2b471e0a --- /dev/null +++ b/src/adapters/unsupported_filesystem.rs @@ -0,0 +1,50 @@ +//! This module owns refusal on platforms without a production adapter. + +use std::{io, path::Path}; + +use cap_std::fs::Dir; + +use super::filesystem_root_identity::FilesystemRootIdentity; +use super::platform_filesystem::PlatformFilesystem; + +pub(super) struct UnsupportedFilesystem; + +impl PlatformFilesystem for UnsupportedFilesystem { + fn open_root(&self, _path: &Path) -> io::Result { + Err(unsupported()) + } + + fn admit_directory(&self, _directory: &Dir) -> io::Result { + Err(unsupported()) + } + + fn root_identity(&self, directory: &Dir) -> io::Result { + #[cfg(any(test, feature = "repository-tasks"))] + return self.repository_identity(directory); + + #[cfg(not(any(test, feature = "repository-tasks")))] + { + let _ = directory; + Err(unsupported()) + } + } + + #[cfg(any(test, feature = "repository-tasks"))] + fn repository_identity(&self, directory: &Dir) -> io::Result { + use cap_fs_ext::MetadataExt; + + let metadata = directory.dir_metadata()?; + Ok(FilesystemRootIdentity::new( + metadata.dev(), + metadata.dev(), + metadata.ino(), + )) + } +} + +fn unsupported() -> io::Error { + io::Error::new( + io::ErrorKind::Unsupported, + "filesystem initialization currently requires the admitted Linux ext4 profile", + ) +} From 9368beed057d254601b57c4d981a2a5d8b0f2709 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 21:30:07 -0800 Subject: [PATCH 3/7] Admit macOS APFS as a production storage platform Route protocol lookups through FilesystemDirectory, which requires each existing name byte-for-byte in a listing of its parent, and select the Darwin adapter. Add a native macOS CI job. No format, dependency, or unsafe change. --- .github/workflows/ci.yml | 41 ++++ CHANGELOG.md | 2 + README.md | 9 +- ROADMAP.md | 15 +- docs/adr/README.md | 2 +- ...ers.md => filesystem-platform-adapters.md} | 34 +++- docs/architecture/filesystem-platforms.md | 39 ++-- docs/formats/segment-store-v1/recovery.md | 13 +- docs/formats/segment-store-v1/requirements.md | 2 +- .../filesystem-platform-adapters.md | 170 ++++++++-------- repository-process-spawn/src/lib.rs | 2 +- scripts/native/check-worker.sh | 29 ++- scripts/native/check.sh | 19 +- src/adapters/catalog_restart_io.rs | 3 +- src/adapters/catalog_restart_loader.rs | 4 +- src/adapters/catalog_restart_segments.rs | 2 +- src/adapters/darwin_filesystem.rs | 86 ++++++++ src/adapters/darwin_filesystem_tests.rs | 189 ++++++++++++++++++ src/adapters/durable/snapshot.rs | 4 +- src/adapters/durable/store.rs | 2 +- src/adapters/filesystem_catalog_artifact.rs | 3 +- src/adapters/filesystem_catalog_current.rs | 4 +- src/adapters/filesystem_catalog_publisher.rs | 3 +- src/adapters/filesystem_directory.rs | 132 ++++++++++++ src/adapters/filesystem_entry_spelling.rs | 63 ++++++ src/adapters/filesystem_exact_record.rs | 3 +- src/adapters/filesystem_exact_record_tests.rs | 4 +- .../filesystem_initialization_namespace.rs | 3 +- .../filesystem_initialization_storage.rs | 2 +- src/adapters/filesystem_namespace_refusal.rs | 3 +- src/adapters/filesystem_platform_profile.rs | 12 +- .../filesystem_recovery_inventory_reader.rs | 4 +- .../filesystem_recovery_inventory_scan.rs | 2 +- src/adapters/filesystem_recovery_namespace.rs | 2 +- ...filesystem_recovery_next_head_finalizer.rs | 2 +- src/adapters/filesystem_recovery_stage.rs | 3 +- ...lesystem_recovery_stage_completion_pool.rs | 2 +- ...lesystem_recovery_stage_discard_storage.rs | 2 +- .../filesystem_recovery_stage_discarder.rs | 2 +- .../filesystem_recovery_stage_sync.rs | 2 +- src/adapters/filesystem_root_identity.rs | 3 +- src/adapters/filesystem_store_initializer.rs | 16 +- .../filesystem_store_initializer_tests.rs | 2 +- .../filesystem_version_two_admission.rs | 3 +- .../filesystem_version_two_records.rs | 2 +- src/adapters/filesystem_writer_lock.rs | 3 +- src/adapters/filesystem_writer_lock_tests.rs | 2 +- src/adapters/mod.rs | 6 +- src/adapters/native_filesystem.rs | 5 +- src/adapters/retention.rs | 2 + .../retention/filesystem_retention_attempt.rs | 2 +- .../filesystem_retention_authority.rs | 2 +- .../retention/filesystem_retention_catalog.rs | 4 +- .../filesystem_retention_closure_admission.rs | 2 +- .../retention/filesystem_retention_current.rs | 3 +- .../filesystem_retention_namespace.rs | 3 +- .../filesystem_retention_recovery.rs | 3 +- ...lesystem_retention_recovery_observation.rs | 3 +- .../filesystem_retention_recovery_roots.rs | 2 +- .../filesystem_retention_snapshot.rs | 7 +- ...tem_retention_snapshot_coordinate_tests.rs | 3 +- ...m_retention_snapshot_moving_error_tests.rs | 3 +- ...system_retention_snapshot_pinning_tests.rs | 3 +- .../filesystem_retention_snapshot_tests.rs | 2 +- .../retention/filesystem_retention_stage.rs | 3 +- .../retention/filesystem_retention_storage.rs | 3 +- .../filesystem_version_two_admission_tests.rs | 4 +- src/adapters/retention/reader_fence.rs | 3 +- src/adapters/retention/reader_fence_tests.rs | 2 +- .../reader_platform_authority_tests.rs | 33 +++ .../retention/reader_platform_law_tests.rs | 17 -- .../filesystem_inventory_catalogs.rs | 2 +- ...esystem_inventory_catalogs_test_fixture.rs | 2 +- .../filesystem_inventory_directory.rs | 2 +- .../filesystem_inventory_file.rs | 3 +- .../filesystem_inventory_file_tests.rs | 2 +- .../filesystem_inventory_names.rs | 2 +- .../filesystem_inventory_names_tests.rs | 2 +- .../filesystem_inventory_reader.rs | 2 +- .../filesystem_inventory_segments.rs | 2 +- ...esystem_inventory_segments_test_fixture.rs | 2 +- .../filesystem_migration_authority.rs | 4 +- .../filesystem_migration_fixed_artifact.rs | 3 +- .../filesystem_migration_namespace.rs | 2 +- ...ilesystem_migration_namespace_directory.rs | 2 +- .../filesystem_migration_reader_fence.rs | 3 +- .../filesystem_migration_recovery.rs | 2 +- .../filesystem_migration_repository_tasks.rs | 2 +- .../filesystem_migration_residue.rs | 3 +- src/adapters/sync_capable_directory.rs | 3 +- src/adapters/unsupported_filesystem.rs | 2 +- src/lib.rs | 2 +- tests/catalog_platform_admission.rs | 12 +- tests/darwin_fifo_refusal.rs | 50 +++++ tests/darwin_store_spelling.rs | 63 ++++++ tests/durable_locator.rs | 2 +- tests/golden_file_worldline.rs | 2 +- .../durable_locator_laws.rs | 22 +- tests/golden_file_worldline/suite.rs | 26 +-- tests/migration_descriptor_exhaustion.rs | 25 ++- tests/observed_segment_stage.rs | 5 +- tests/reader_fence_process.rs | 5 +- tests/reader_fence_process/reader.rs | 29 ++- tests/recovery_inventory.rs | 10 +- tests/segment_filesystem_stage.rs | 2 +- tests/store_initialization.rs | 6 +- 106 files changed, 1066 insertions(+), 313 deletions(-) rename docs/adr/{0010-filesystem-platform-adapters.md => filesystem-platform-adapters.md} (71%) create mode 100644 src/adapters/darwin_filesystem.rs create mode 100644 src/adapters/darwin_filesystem_tests.rs create mode 100644 src/adapters/filesystem_directory.rs create mode 100644 src/adapters/filesystem_entry_spelling.rs create mode 100644 src/adapters/retention/reader_platform_authority_tests.rs create mode 100644 tests/darwin_fifo_refusal.rs create mode 100644 tests/darwin_store_spelling.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3f758cf3..fa9151d4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -87,6 +87,47 @@ jobs: cargo check --manifest-path fuzz/Cargo.toml --bins --locked cargo clippy --manifest-path fuzz/Cargo.toml --bins --locked -- -D warnings + macos: + name: Native macOS storage + runs-on: macos-15 + timeout-minutes: 90 + steps: + - name: Check out repository + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + persist-credentials: false + + - name: Install pinned toolchain + run: rustup show + + # The default macOS TMPDIR traverses the /var symlink, which admission refuses. + - name: Select an admitted scratch directory + run: echo "TMPDIR=$RUNNER_TEMP/" >> "$GITHUB_ENV" + + - name: Lint all features + run: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings + + - name: Run process-death crash matrix on default APFS + run: cargo xtask durability-crash-matrix + + - name: Test debug profile on default APFS + run: cargo test --all-features --locked + + - name: Test release profile on default APFS + run: cargo test --all-features --release --locked + + - name: Mount case-sensitive APFS + run: | + hdiutil create -size 1g -type SPARSE -fs "Case-sensitive APFS" -volname keep-cs "$RUNNER_TEMP/keep-cs.sparseimage" + mkdir "$RUNNER_TEMP/keep-cs" + hdiutil attach -nobrowse -mountpoint "$RUNNER_TEMP/keep-cs" "$RUNNER_TEMP/keep-cs.sparseimage" + + - name: Test debug profile on case-sensitive APFS + run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo test --all-features --locked + + - name: Run process-death crash matrix on case-sensitive APFS + run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo xtask durability-crash-matrix + documentation: name: Documentation and workflow integrity runs-on: ubuntu-latest diff --git a/CHANGELOG.md b/CHANGELOG.md index ce726ece..d2619b2e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established. ## [Unreleased] +- Admit macOS APFS as a production storage platform, on both default and case-sensitive volumes. Keep requires every existing protocol name byte-for-byte in a listing of its parent, so `head` never answers for `HEAD`. Adds a native macOS CI job; no format, dependency, or unsafe change. The crash matrix passes natively, but no power-loss experiment has been run on APFS. + - Update the development-only Markdown validation graph and exact admission policy to markdownlint-cli2 0.24.0. On the 2026-11-03 audit this removes the markdown-it and smol-toml reports and adds the high-severity re2js advisory GHSA-gg6h-894w-4m9c through markdownlint's exact pin. This is a currency update, not a demonstrated security gain: markdown-it was not loaded on Keep's path, re2js now is, and the admission records why its affected path appears unreachable. MD011 also reports slightly less, and #209 tracks removing re2js (#188). - Update Rustix to 1.1.6, which fixes the high-severity GHSA-jh2m-fhpj-3vr7 extended-attribute undefined behavior with no Keep exposure found, and cap-fs-ext to 4.0.3 in both locked workspaces. Renew their dependency admission and pin the `RDONLY` mount-flag value in a law; selected features and the Rust floor are unchanged (#207, #189). diff --git a/README.md b/README.md index 6b49639e..e0b9b9d3 100644 --- a/README.md +++ b/README.md @@ -50,7 +50,8 @@ Keep is required to refuse all three, before mutating anything. complete framing, checksum, and identity verification. Immutable segments, generation-versioned catalogs, and a fixed-width `HEAD` are published through an ordered protocol whose every step is a named crash point. - Platform admission is Linux ext4, non-casefolded, one writer. + Platform admission is non-casefolded Linux ext4 or local writable macOS APFS, + one writer. - **Process-death recovery evidence.** The crash matrix kills real writer processes at 156 before/during/after coordinates (`KEEP-CRASH-001`–`052`) and verifies the store lands in exactly one documented lawful state each time, @@ -171,8 +172,10 @@ policy. Retention proves a *physical reconstruction* claim only — never what the content means, who owns it, or whether deleting it is legally safe. The [filesystem platform design](docs/architecture/filesystem-platforms.md) -defines the shared Linux/macOS adapter boundary and native APFS acceptance -work. It is a design in implementation, not a claim of shipped macOS support. +defines the shared Linux/macOS adapter boundary. Both APFS naming variants are +admitted; the [evidence record](docs/testing-evidence/filesystem-platform-adapters.md) +states what has been exercised, and process-death evidence is not power-loss +evidence. ## Try it diff --git a/ROADMAP.md b/ROADMAP.md index 0fe6874d..447f03da 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -147,7 +147,7 @@ names; use those in code, tests, and commits. - [ ] [F-39 Compression representation codec](#f-39-compression-representation-codec) — Proposed - [ ] [F-40 Additional chunking profiles and hierarchical layouts](#f-40-additional-chunking-profiles-and-hierarchical-layouts) — Proposed -- [ ] [F-41 Platform adapters beyond Linux ext4](#f-41-platform-adapters-beyond-linux-ext4) — Proposed +- [ ] [F-41 Platform adapters beyond Linux ext4](#f-41-platform-adapters-beyond-linux-ext4) — In progress; macOS APFS done - [ ] [F-42 Operator surfaces](#f-42-operator-surfaces) — Proposed (CLI and MCP adapters) - [ ] [F-43 Public release and API stability](#f-43-public-release-and-api-stability) — Proposed - [ ] [F-44 Multi-writer, replication, and remote tiers](#f-44-multi-writer-replication-and-remote-tiers) — Out of scope @@ -582,7 +582,7 @@ in place. One writer holds kernel advisory locks on the store root and **Status:** Done (issue #17, M3). Production initialization admits only a writable, non-casefolded Linux -ext4 root on a single local host. Opening is observational; recovery is +ext4 root or a writable local macOS APFS root, on a single local host. Opening is observational; recovery is explicit and planned against storage ports: inventory, name classification, stage fingerprint and assessment, then discard, completion, next-head finalization, or segment resume. `cargo xtask durability-crash-matrix` @@ -2396,11 +2396,11 @@ plan or needs bounded plan streaming. ### F-41 Platform adapters beyond Linux ext4 -**Status:** Proposed. `recovery.md` defers Windows "until an adapter and -crash harness prove equivalent semantics"; macOS is not admitted for -production; the crash matrix does not simulate power loss. +**Status:** In progress. `recovery.md` defers Windows "until an adapter and +crash harness prove equivalent semantics"; macOS APFS is admitted (T-41.1); +the crash matrix does not simulate power loss. -- [ ] T-41.1 macOS APFS production admission. +- [x] T-41.1 macOS APFS production admission. - **Requirements:** a platform profile that probes APFS for the same capabilities (atomic no-clobber link, atomic same-filesystem replacement, durable `fsync` semantics including `F_FULLFSYNC`, @@ -2409,7 +2409,8 @@ production; the crash matrix does not simulate power loss. - **Acceptance criteria:** `KEEP-RECOVERY-003` gains a macOS row; the 105-case matrix (and the v2 matrices) green on a macOS runner. - **Scope:** in — `filesystem_platform_profile.rs` sibling. Out — - case-insensitive volumes (refuse), network volumes. + network volumes. Case-insensitive volumes are in, with exact-spelling + checks (see the platform adapter decision). - **User stories:** Human — a developer on a Mac runs a durable store locally. API user — same API. MCP user — same. Agent — same. - **Interface:** none new. diff --git a/docs/adr/README.md b/docs/adr/README.md index 3f2ca2ce..c504455b 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -61,4 +61,4 @@ encryption, concurrency, or public-API surface it governs. - [ADR-0007: Terminal signal process-group guard](0007-terminal-signal-process-group-guard.md) - [ADR-0008: Deadline-bounded reader retirement](0008-deadline-bounded-reader-retirement.md) - [ADR-0009: Retention roots, release, and GC liveness](0009-retention-roots-release-and-gc-liveness.md) -- [ADR-0010: Filesystem platform adapters](0010-filesystem-platform-adapters.md) +- [Filesystem platform adapters](filesystem-platform-adapters.md) diff --git a/docs/adr/0010-filesystem-platform-adapters.md b/docs/adr/filesystem-platform-adapters.md similarity index 71% rename from docs/adr/0010-filesystem-platform-adapters.md rename to docs/adr/filesystem-platform-adapters.md index 02445062..4956002e 100644 --- a/docs/adr/0010-filesystem-platform-adapters.md +++ b/docs/adr/filesystem-platform-adapters.md @@ -1,6 +1,6 @@ -# ADR-0010: Filesystem Platform Adapters +# Filesystem Platform Adapters -- Status: Accepted design; implementation and native acceptance in progress +- Status: Accepted; implemented for Linux ext4 and macOS APFS - Date: 2026-11-02 - Owners: Keep storage implementation and repository owner - Related issue: none; directly requested platform design and implementation @@ -29,8 +29,11 @@ the operations, ownership, Linux profile extension rules, APFS naming and durability obligations, and acceptance gates. Backend selection happens once. It must not alter logical identities or duplicate protocol state machines. -Native macOS support targets writable local APFS. Exact namespace spelling -must remain enforced on case-insensitive volumes. Other Linux filesystems +Native macOS support admits writable local APFS in both naming variants, with +no format difference between them. Case-insensitive volumes keep exact namespace +spelling: a name the lookup resolves must also appear byte-for-byte in a listing +of its parent, or the operation refuses before it reads, reuses, or mutates the +entry. No unsafe code or Darwin-specific dependency is needed. Other Linux filesystems are distinct admitted profiles with their own evidence, rather than being accepted because they provide familiar POSIX operations. Windows and network filesystems are excluded from this implementation. @@ -59,6 +62,12 @@ selection and must never create independent competing writer authorities. - **Require a separate case-sensitive volume without evaluating default APFS:** avoids a useful native deployment but leaves ordinary Mac storage unsupported. Exact canonical namespace handling is part of the target. +- **Check spelling with `F_GETPATH`:** it reports a name cached on the vnode. + With hard links, which publication uses, it can name another link of the same + file, so it both refuses valid entries and misreports aliases. +- **Detect the naming variant and scan only on case-insensitive volumes:** needs + an unsafe `fpathconf` binding to save a scan, and adds a branch whose untaken + side no single volume exercises. - **Claim support from Linux containers on a Mac:** exercises Linux semantics, not Darwin/APFS. Native acceptance requires native production-path evidence. @@ -69,13 +78,16 @@ of descriptors and filesystem-library types. Adding an OS entails one adapter, one selector entry, and its conformance evidence, rather than changing every durability protocol. Filesystem support stays explicit and fail-closed. -Existing Linux wire bytes remain unchanged. Darwin physical recovery -coordinates need a documented mapping and compatibility analysis; logical -content identity must not incorporate native metadata. Any format change -discovered necessary during implementation requires an explicit revision of -this decision and the format contract before it ships. +Existing Linux wire bytes remain unchanged. Darwin recovery coordinates set the +top device bit and store `st_dev` as both device and mount, because each mounted +APFS volume has its own `st_dev`; a record written on one OS can never match a +root on the other. Logical content identity does not incorporate native metadata. + +On macOS, opening an existing entry costs one streaming scan of its parent +directory. A rename that replaces a name during that scan could in principle hide +it and cause a false refusal; a bounded stress law did not observe one, and such +a refusal fails closed. The native validation environment is separately bounded and serialized within the same project resource budget. Successful process-death recovery is not -evidence of physical power-loss survival. Design acceptance does not claim -that implementation, resource enforcement, or platform validation is complete. +evidence of physical power-loss survival. diff --git a/docs/architecture/filesystem-platforms.md b/docs/architecture/filesystem-platforms.md index 12259dbc..e7e0e760 100644 --- a/docs/architecture/filesystem-platforms.md +++ b/docs/architecture/filesystem-platforms.md @@ -1,7 +1,8 @@ # Filesystem platform adapters -Status: implementation design. The checked-in Linux adapter remains the only -production-admitted backend until the acceptance gates below have evidence. +Status: implemented. Production admits Linux ext4 and macOS APFS; the +[evidence record](../testing-evidence/filesystem-platform-adapters.md) states +what has been exercised and what has not. This design adds native macOS storage while retaining Keep's exact-byte, exclusive-writer, immutable-publication, and recovery contracts. It also defines @@ -144,7 +145,7 @@ allow adding either without editing the protocol orchestrators. ## macOS profiles -The intended native target is local writable APFS, including the ordinary +The native target is local writable APFS, including the ordinary case-insensitive variant and case-sensitive APFS. HFS+ and other filesystems remain explicit future profiles. APFS snapshots and read-only mounts refuse writer admission. @@ -161,20 +162,21 @@ The actual names observed during namespace admission must match the grammar byte-for-byte. `HEAD` must not be admitted through a foreign `head` entry, and uppercase digest names must not silently alias canonical lowercase names. The same obligation applies to reads, recovery, and publication reuse, not -only to store initialization. Implement and test exact-spelling checks before -admitting this profile. Unicode normalization is irrelevant to generated names -but must not turn malformed foreign names into admitted protocol entries. +only to store initialization. The adapter resolves a name, then requires it +byte-for-byte in a listing of the parent, on either naming variant. The listing +is the only observation of the stored spelling: `F_GETPATH` reports a name cached +on the vnode, which can belong to another hard link of the same file. Case +folding beyond ASCII and Unicode normalization need no special handling, because +the comparison is byte equality against whatever the lookup resolved. Regular-file synchronization preserves the pinned Rust toolchain's full-flush semantics. Rust 1.96's Apple `File::sync_all` calls `F_FULLFSYNC`. Directory -synchronization requires native validation of the same guarantee; do not +synchronization uses the same call on the directory descriptor; do not catch `EINVAL` or `ENOTSUP` and return success, and do not silently substitute plain `fsync` for a stronger promised operation. -The native adapter must use safe existing wrappers where available. If a -required descriptor-bound observation has no safe wrapper, isolate the minimal -FFI in a dedicated crate with documented invariants and boundary tests. Adding -a broad platform dependency merely for one syscall is not the default choice. +The native adapter uses existing safe wrappers and needs no FFI: `fsid` only +repeats `st_dev` beside the filesystem type, so it is not read. ## Physical coordinates and durable compatibility @@ -188,11 +190,10 @@ same-process evidence and compares device/inode for restart. Preserve that distinction; never replace a persisted intent merely because a mount number changed after restart. -Document a named Darwin coordinate mapping and its validation before emitting -such records. It must be deterministic, checked, and must reject copied roots. -Do not hash arbitrary OS structs or truncate native fields. If the existing -wire grammar cannot distinguish a necessary coordinate domain, make a -versioned format decision rather than silently reinterpreting existing bytes. +The Darwin mapping sets the top bit of the device coordinate over the checked +32-bit `st_dev`, uses that value for both device and mount, and stores the inode +unchanged. A negative `st_dev` refuses instead of being truncated. The top bit +keeps Darwin coordinates apart from Linux ones without changing the wire grammar. Cross-OS movement of a physical store is not ordinary reopen. A store whose recovery evidence binds it to another physical root must refuse. Logical @@ -244,14 +245,10 @@ ceiling. Use one serialized reusable cache per incompatible OS toolchain; native validation cannot reuse Linux object files. Every launch records actual usage, free space, lock, enforced limits, timeout, and descendant cleanup. -The implementation is complete only when the design is reflected in code, -Linux behavior remains verified, native production APFS paths are exercised, -and copies of this design and its decision record have intact Reader receipts. - ## Sources and related contracts - [Hexagonal boundaries](https://github.com/flyingrobots/keep/blob/3887f2a/docs/adr/0004-hexagonal-boundary-architecture.md). -- [Platform adapter decision](../adr/0010-filesystem-platform-adapters.md). +- [Platform adapter decision](../adr/filesystem-platform-adapters.md). - [Current filesystem contract](https://github.com/flyingrobots/keep/blob/3887f2a/docs/formats/segment-store-v1/recovery.md#platform-contract). - [Migration recovery](https://github.com/flyingrobots/keep/blob/3887f2a/docs/formats/segment-store-v2/migration-recovery.md). - [Apple APFS naming variants](https://developer.apple.com/library/archive/documentation/FileManagement/Conceptual/APFS_Guide/FAQ/FAQ.html). diff --git a/docs/formats/segment-store-v1/recovery.md b/docs/formats/segment-store-v1/recovery.md index e2761679..609a9d50 100644 --- a/docs/formats/segment-store-v1/recovery.md +++ b/docs/formats/segment-store-v1/recovery.md @@ -366,7 +366,7 @@ from an untrusted count before bounds and exact total length agree. ## Platform contract -The initial production adapter is supported only on Linux when it proves: +The Linux production adapter is supported when it proves: - capability-relative no-follow access to regular files and directories; - an ext4 store root whose inode does not enable ext4 casefolding; @@ -384,13 +384,20 @@ The initial production adapter is supported only on Linux when it proves: The adapter refuses every non-ext4 filesystem, read-only mount, casefolded store or protocol directory, protocol-directory mount point, foreign -protocol-directory device, symlinked selected path, or platform other than -Linux. A single local host is an explicit deployment precondition: filesystem +protocol-directory device, symlinked selected path, or platform with no adapter. A single local host is an explicit deployment precondition: filesystem metadata cannot prove that an administrator has not exposed one block device to another host. Shared ext4 is therefore unsupported even though the adapter cannot distinguish it from a valid local mount. Windows support is deferred until an adapter and crash harness prove equivalent semantics. +The macOS production adapter applies the same protocol to writable local APFS, +in either naming variant. It refuses read-only and non-local mounts, every other +filesystem, foreign protocol-directory devices, and symlinked selected paths. +Because a case-insensitive lookup can resolve a different spelling, it requires +each existing protocol name byte-for-byte in a listing of its parent before +reading, reusing, or mutating it. File and directory synchronization use +`F_FULLFSYNC`. + The protocol cannot compensate for hardware or an operating system that acknowledges synchronization without honoring it. Documentation and receipts state this assumption; they do not upgrade it into proof. diff --git a/docs/formats/segment-store-v1/requirements.md b/docs/formats/segment-store-v1/requirements.md index 35f59bae..081f0c12 100644 --- a/docs/formats/segment-store-v1/requirements.md +++ b/docs/formats/segment-store-v1/requirements.md @@ -117,7 +117,7 @@ device fault evidence. | --- | --- | --- | --- | --- | | `KEEP-RECOVERY-001` | Crash identifiers `KEEP-CRASH-001` through `KEEP-CRASH-035` form one contiguous typed vocabulary, map to the exact owning protocol sequence, and admit an occurrence counter only for record append | Ordered identifier-and-sequence ledger | `xtask/tests/durability_crash_point_contract.rs` | Implemented in #17 | | `KEEP-RECOVERY-002` | Initialization admits the platform before mutation, opens and locks the writer file, admits `staging`, `segments`, and `catalogs` in order, and returns a receipt only after root synchronization; every failed operation retains its exact phase and prevents later transitions | Fault-recording initialization port | `tests/store_initialization.rs` | Implemented in #17 | -| `KEEP-RECOVERY-003` | Production initialization admits only one writable, non-casefolded Linux ext4 store profile; every existing protocol directory must independently satisfy that profile and share the root's device and mount identity; initialization refuses any noncanonical root entry before mutation, completes an empty or partial canonical namespace without replacing evidence, excludes a second initializer, and retains writer authority through the synchronized receipt; published-store reopen performs no mutation, reacquires the writer lock, and requires the complete initialized root plus regular `HEAD` | Capability-relative initialization, published-reopen, child-profile, and exact platform-profile matrix | `src/adapters/filesystem_store_initializer_tests.rs`, `src/adapters/filesystem_catalog_publisher_tests.rs`, `src/adapters/filesystem_platform_profile.rs`, `tests/store_initialization.rs` | Implemented in #17 | +| `KEEP-RECOVERY-003` | Production initialization admits only one writable, non-casefolded Linux ext4 store profile or writable local macOS APFS store profile; every existing protocol directory must independently satisfy that profile and share the root's device and mount identity; initialization refuses any noncanonical root entry before mutation, completes an empty or partial canonical namespace without replacing evidence, excludes a second initializer, and retains writer authority through the synchronized receipt; published-store reopen performs no mutation, reacquires the writer lock, and requires the complete initialized root plus regular `HEAD` | Capability-relative initialization, published-reopen, child-profile, and exact platform-profile matrix | `src/adapters/filesystem_store_initializer_tests.rs`, `src/adapters/filesystem_catalog_publisher_tests.rs`, `src/adapters/filesystem_platform_profile.rs`, `tests/store_initialization.rs` | Implemented in #17 | | `KEEP-RECOVERY-004` | Writer authority is returned only when the locked handle still has the exact device and inode resolved by the canonical `writer.lock` entry after kernel acquisition | Deterministic replacement through the shared authority-producing acquisition boundary; calibrated ignored-refusal mutant | `src/adapters/filesystem_writer_lock_tests.rs`; [acquisition evidence](../../testing-evidence/writer-acquisition-identity.md) | Implemented in #17; acquisition-boundary evidence strengthened in #169 | | `KEEP-RECOVERY-005` | Recovery counts the root and three protocol directories in fixed order before retaining names, refuses at the configured or protocol entry ceiling with the exact observed-at-least count, then returns one duplicate-free inventory sorted by namespace and raw name bytes | Fault-recording inventory port | `tests/recovery_inventory.rs` | Implemented in #17 | | `KEEP-RECOVERY-006` | Filesystem inventory pins the admitted root and protocol directories without following links, verifies child-directory identity before and after scanning, stops each count at the remaining global budget plus one, preserves raw Linux entry-name bytes, and performs no protocol mutation | Capability-relative filesystem fixture | `src/adapters/filesystem_recovery_inventory_tests.rs`, `tests/recovery_inventory.rs` | Implemented in #17 | diff --git a/docs/testing-evidence/filesystem-platform-adapters.md b/docs/testing-evidence/filesystem-platform-adapters.md index 8b69d25b..e4b215ac 100644 --- a/docs/testing-evidence/filesystem-platform-adapters.md +++ b/docs/testing-evidence/filesystem-platform-adapters.md @@ -1,94 +1,82 @@ # Filesystem platform adapter evidence This is an implementation record, not platform certification. The design is -in [filesystem platforms](../architecture/filesystem-platforms.md). - -## Claims and current status - -- Product: native production storage on ordinary and case-sensitive APFS. - **Pending implementation and native runtime evidence.** -- Product: Linux ext4 admission, authority, and durability laws survive the - extraction of the native mechanism boundary. **Runtime validation pending.** -- Static/API: focused protocol ports and public constructors remain available. - Source inspection establishes the intended preservation; compilation pending. -- Tool: bounded native compilation without writes outside admitted outputs. - Runner implemented; execution and guard calibration pending. -- Documentation: architecture and decision record delivered unchanged to Reader. - Receipt `f8a83855-f07e-4622-ac91-25b5bcf72b34`, `awaiting_filing`, - `integrity: intact`, for design commit `ba99881`. - -The author owns the evidence; repository owner `@flyingrobots` owns policy. -No missing gate is treated as an approved waiver. - -## Native mechanism observations - -On the workstation's existing writable APFS data volume, read-only descriptor -observations established that `README.md` and `readme.md` resolve to the same -device/inode. `F_GETPATH` returned the actual stored spelling in both cases. -A bounded one-byte fixture with two hard links returned each opened link's -actual spelling, including when one was opened using an uppercase alias. -The fixture was removed. `F_FULLFSYNC` on an open repository directory returned -success. These observations motivate the adapter; they are not permanent -production tests, a concurrency proof, or a power-loss experiment. - -The pinned Rust 1.96 standard library's Apple implementation of -`File::sync_all` calls `F_FULLFSYNC`. A new adapter must not weaken that call. - -## Native compilation launch contract - -The initial compiler-only runner is `scripts/native/check.sh`. It does not -launch test executables or crash campaigns. Its admitted child tree is Cargo, -the pinned compiler, linker, and admitted dependency build scripts; workloads -that detach process groups are excluded from this runner. - -- Reused Linux worker: `keep-validation`, image `rust:1.96.0`; idle at inspection. - The stopped historical documentation worker is `keep114-closure-docs`. -- Workstation lock store: `/Users/Shared/git-locks/workstation.git`. - Hold `host/cache/keep-native-validation/`, `host/docker/keep-validation/`, and - `host/docker/keep114-closure-docs/` together for project-budget admission. -- Measured Docker writable layers: 19,150,114,816 and 271,781,888 bytes. - Counting their entire layers conservatively gives 19,421,896,704 bytes. -- Native build allocation: one reusable 1.5 GiB APFS sparse image, mounted under - `$HOME/Library/Caches/keep-native-validation/build`. Target and Cargo-local - metadata are the only compiler-writable build paths. Existing registry - contents and toolchains are read-only inputs. No second target is permitted. -- The conservative existing-layer total plus the full native build capacity - is 21,032,509,440 bytes, below the 21,474,836,480-byte project build budget. -- Native scratch: a separate 256 MiB APFS image at the adjacent `data` mount; - `TMPDIR` points to its `tmp` directory. It is scratch, not durability evidence. -- Logs: adjacent owned `logs/`; each logger has a 4 MiB file-size ceiling. - Launch refuses above 20 MiB already retained, bounding native logs at 24 MiB. - Existing worker log rotation permits at most 100 MiB; combined ceilings stay - below 128 MiB. Evidence must be exported before owned logs are recycled. -- Disk admission observed about 557 GiB host and 530 GiB Docker backing free. - The native run monitors host free space and stops below 50 GiB. It does not - generate Docker output while holding both worker locks. -- Hard filesystem capacities bound build and temporary output. The compiler - sandbox denies network and all filesystem writes except target, Cargo-local - metadata, scratch, and `/dev/null`; its log writer is separately size-limited. -- Two Cargo jobs; inherited 900-second per-process CPU ceiling and 4 GiB - per-process virtual-address-space ceiling. The parent monitors aggregate - process-group RSS and refuses above 6 GiB. Whole-command deadline: 1,200 seconds. -- The compiler tree has a fresh process group. Limit breach, monitor failure, - interruption, or exit kills the owned group and waits for its leader. - The reservation must exceed the deadline and cleanup time. - -The native images are retained reusable caches, not per-revision artifacts. -Their measured usage and execution receipts must be added after validation. -This contract does not authorize another cache when one fills. - -## Oracles and remaining acceptance - -Shared mount-admission laws compare exact typed device/mount contradictions -and permit a child inode distinct from its parent on the same mount. Existing -Linux profile laws retain writable ext4 and casefold rejection. The old combined -child-policy unit law is separated into native profile laws and common mount -laws because those responsibilities now have different owners. Production -directory traversal and child-admission coverage still require execution. - -Native product acceptance must exercise public production admission, exact -content reads, publication, retention, migration/reopen, writer exclusion, -case aliases, malformed namespaces, substitutions, and interrupted recovery. -Both APFS naming variants require actual runs. Debug/release, strict Clippy, -dependency checks, Linux regressions, CI configuration, and replayable crash -evidence remain pending. No native support or regression-free claim is made. +in [filesystem platforms](../architecture/filesystem-platforms.md) and the +decision in [filesystem platform adapters](../adr/filesystem-platform-adapters.md). + +## Claims and status + +- Native production storage on default and case-sensitive APFS: **implemented**. + Process-death evidence below. No power-loss experiment has been run on APFS. +- Linux ext4 admission, authority, and durability laws survive extraction of the + native boundary: established by the Linux CI jobs on the merge candidate. +- No format change: no golden fixture changed, and Linux coordinates are untouched. +- No unsafe code and no new dependency: `Cargo.toml` and `Cargo.lock` are unchanged. + +## Native execution + +Darwin 27.0.0, arm64, pinned `rustc 1.96.0`, under `scripts/native/check.sh`. + +| Check | Default APFS | Case-sensitive APFS | +| --- | --- | --- | +| `cargo test --all-features` (keep package, debug) | 959 passed, 0 failed | 959 passed, 0 failed | +| `cargo xtask durability-crash-matrix` | exit 0 | exit 0 | + +The volumes are sparse images: default APFS `data` and case-sensitive APFS +`data-cs`, 256 MiB each, selected with `KEEP_NATIVE_DATA_IMAGE`. Release-profile +tests, `xtask` tests, and lint run in the `Native macOS storage` CI job. + +An earlier set of native runs used Homebrew's Rust 1.98 ahead of rustup on `PATH` +and is not counted. The worker now selects the pinned toolchain and asserts its +version before running. + +## Spelling oracle + +`F_GETPATH` was rejected. With a descriptor retained on `a/HEAD`, a probe of its +hard link reported `b/head` for it, so the name is vnode state, not bound to the +descriptor, and publication uses hard links. Directory listings return the stored spelling, so the check is +byte equality against a listing of the parent. + +Calibration with the check disabled, on default APFS: + +- `a_foreign_spelling_never_answers_for_a_protocol_name`, + `a_non_ascii_fold_never_answers_for_a_protocol_name`, and + `mutation_through_a_foreign_spelling_leaves_the_entry_unchanged` **fail**. +- `darwin_store_spelling` laws **still pass**. Namespace admission independently + refuses the foreign entry, so those laws do not witness the per-lookup check. + +## Laws that remain Linux-only, and why + +- tmpfs refusal (`reader_platform_law_tests`, `a_writer_lock_cannot_mint_...`): + no tmpfs. Darwin refusal of non-APFS uses devfs and of read-only APFS uses the + sealed system volume, both at the adapter rather than the public boundary. +- Raw non-UTF-8 names: APFS refuses to create them. +- `filesystem_retention_fifo_tests`: Rustix omits `mknodat` on Apple targets and + `std`'s `mkfifo` is unstable. `tests/darwin_fifo_refusal.rs` covers a FIFO at + `HEAD` only. +- O_PATH directory synchronization and ext4 profile laws: Linux mechanisms. + +## Weaker on macOS + +- The reader-fence law cannot observe a queued `flock`, because Darwin has no + `/proc/locks`. It requires that no snapshot escapes for two seconds, so a late + child can weaken it but cannot falsely fail it. +- Isolated laws use `ulimit -t` where Linux uses wall-clock `timeout`. + +## Not measured + +- The cost of the parent scan per existing-entry lookup. It is linear in the + directory size and matters most for a large segment pool. +- `F_FULLFSYNC` durability. A successful call is not proof that hardware honored + it. +- A real non-APFS or network volume at the public boundary. + +## Runner bounds + +Compilation and test output stays on the reusable 1.5 GiB `build` image, 349 MiB +used after a clean run. Each data image stayed under 4 MiB used. Logs total +1.1 MiB. The sandbox denies network except local Unix sockets, which the process +laws need, and denies writes outside the build and selected data images. + +The runner watches only its own process group, so a law that starts its own group +is bounded by that law's watchdog, not by the runner. diff --git a/repository-process-spawn/src/lib.rs b/repository-process-spawn/src/lib.rs index 978492d9..a5ce4178 100644 --- a/repository-process-spawn/src/lib.rs +++ b/repository-process-spawn/src/lib.rs @@ -1,6 +1,6 @@ //! Isolated child-process setup for descriptor-bound working directories. //! -//! This crate contains Keep's only admitted unsafe boundary. It exists because +//! This crate contains Keep's child-process unsafe boundary. It exists because //! [`std::process::Command`] exposes child setup through an unsafe hook, while //! an opened directory descriptor is required to prevent pathname-replacement //! races. No filesystem, storage, identity, or durable-format code belongs diff --git a/scripts/native/check-worker.sh b/scripts/native/check-worker.sh index e20595b2..91661c89 100644 --- a/scripts/native/check-worker.sh +++ b/scripts/native/check-worker.sh @@ -1,11 +1,12 @@ #!/usr/bin/env bash -# Compiler-only child: all writable locations are on the bounded native images. +# Compile or test child: all writable locations are on the bounded native images. set -euo pipefail readonly source_dir="$1" readonly build_dir="$2" readonly scratch_dir="$3" readonly log_path="$4" readonly sandbox_profile="$5" +readonly check_scope="$6" export CARGO_HOME="$build_dir/cargo-home" export CARGO_TARGET_DIR="$build_dir/target" @@ -15,16 +16,34 @@ export CARGO_NET_OFFLINE=true export CARGO_PROFILE_DEV_DEBUG=0 export CARGO_PROFILE_TEST_DEBUG=0 export TMPDIR="$scratch_dir/" +export CARGO_TARGET_TMPDIR="$scratch_dir/test" unset RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER cd "$source_dir" -# The monitor caps aggregate RSS; these inherited ceilings also bound each -# compiler process independently. No test executable is launched by this tool. +# A package-manager cargo can precede rustup on PATH. Evidence must come from the +# repository's pinned toolchain, including its clippy. +readonly toolchain="$(awk -F'"' '/^channel/ {print $2}' rust-toolchain.toml)" +PATH="$(dirname "$(rustup which --toolchain "$toolchain" cargo)"):$PATH" +export PATH +[[ "$(rustc --version)" == "rustc $toolchain "* ]] + +if [[ "$check_scope" == --run-tests ]]; then + cargo_command=(cargo test --all-features --locked --offline --no-fail-fast) +elif [[ "$check_scope" == --crash-matrix ]]; then + cargo_command=(cargo xtask durability-crash-matrix) +elif [[ "$check_scope" == --clippy ]]; then + cargo_command=(cargo clippy --workspace --all-targets --all-features --locked --offline -- -D warnings) +else + cargo_command=(cargo check "$check_scope" --all-features --locked --offline) +fi + +# The parent monitor caps aggregate RSS. The inherited CPU ceiling also bounds +# each compiler or test process. ( ulimit -t 900 - ulimit -v 4194304 + rustc --version sandbox-exec -f "$sandbox_profile" \ - cargo check --lib --all-features --locked --offline + "${cargo_command[@]}" ) 2>&1 | ( # This limit applies to the log writer, not compiler artifact sizes. ulimit -f 4096 diff --git a/scripts/native/check.sh b/scripts/native/check.sh index f7360fb7..4527f2dc 100644 --- a/scripts/native/check.sh +++ b/scripts/native/check.sh @@ -1,11 +1,16 @@ #!/usr/bin/env bash -# Bounded native compilation. Invoke under the workstation resource locks. +# Bounded native compilation and test execution. Invoke under the workstation resource locks. set -euo pipefail readonly source_dir="$(cd -- "${1:?usage: check.sh CHECKOUT}" && pwd)" +readonly check_scope="${2:---lib}" +case "$check_scope" in --lib|--tests|--run-tests|--crash-matrix|--clippy) ;; *) exit 2 ;; esac readonly native_root="${KEEP_NATIVE_ROOT:-$HOME/Library/Caches/keep-native-validation}" readonly build_dir="$native_root/build" -readonly scratch_dir="$native_root/data/tmp" +# data is default case-insensitive APFS; data-cs is the case-sensitive variant. +readonly data_image="${KEEP_NATIVE_DATA_IMAGE:-data}" +case "$data_image" in data|data-cs) ;; *) exit 2 ;; esac +readonly scratch_dir="$native_root/$data_image/tmp" readonly logs_dir="$native_root/logs" readonly profile="$native_root/compiler.sb" readonly log_path="$logs_dir/$(date -u +%Y%m%dT%H%M%SZ)-check.log" @@ -17,7 +22,7 @@ readonly minimum_free_kib=52428800 # Setup is deliberate and separate. Refuse an absent quota filesystem instead # of compiling into an ordinary directory left behind by an unmount. [[ "$(df -Pk "$build_dir" | awk 'END {print $6}')" == "$build_dir" ]] -[[ "$(df -Pk "$scratch_dir" | awk 'END {print $6}')" == "$native_root/data" ]] +[[ "$(df -Pk "$scratch_dir" | awk 'END {print $6}')" == "$native_root/$data_image" ]] [[ "$(df -Pk "$build_dir" | awk 'END {print $2}')" -le 1572864 ]] [[ "$(df -Pk "$scratch_dir" | awk 'END {print $2}')" -le 262144 ]] [[ "$(df -Pk "$source_dir" | awk 'END {print $4}')" -ge "$minimum_free_kib" ]] @@ -27,6 +32,8 @@ cat > "$profile" <= 0 or die "setsid failed"; exec @ARGV or die "exec failed";' \ /bin/bash "$worker" "$source_dir" "$build_dir" "$scratch_dir" \ - "$log_path" "$profile" & + "$log_path" "$profile" "$check_scope" & task_pid=$! readonly task_started=$SECONDS failure='' diff --git a/src/adapters/catalog_restart_io.rs b/src/adapters/catalog_restart_io.rs index 24c721dc..38623d58 100644 --- a/src/adapters/catalog_restart_io.rs +++ b/src/adapters/catalog_restart_io.rs @@ -3,9 +3,10 @@ use std::io::{self, Read}; use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, OpenOptionsFollowExt, OpenOptionsSyncExt}; use cap_std::ambient_authority; -use cap_std::fs::{Dir, File, OpenOptions}; +use cap_std::fs::{File, OpenOptions}; use super::{CatalogRestartArtifact, CatalogRestartError, CatalogRestartPhase}; diff --git a/src/adapters/catalog_restart_loader.rs b/src/adapters/catalog_restart_loader.rs index ce96d44d..175c3bde 100644 --- a/src/adapters/catalog_restart_loader.rs +++ b/src/adapters/catalog_restart_loader.rs @@ -2,8 +2,6 @@ use std::path::Path; -use cap_fs_ext::DirExt; - use crate::CatalogLength; use super::{ @@ -26,7 +24,7 @@ pub(super) fn load( } pub(super) fn load_from_directory( - directory: &cap_std::fs::Dir, + directory: &crate::adapters::filesystem_directory::FilesystemDirectory, head_name: &str, policy: CatalogRestartPolicy, ) -> Result { diff --git a/src/adapters/catalog_restart_segments.rs b/src/adapters/catalog_restart_segments.rs index 116b6f67..756e45c0 100644 --- a/src/adapters/catalog_restart_segments.rs +++ b/src/adapters/catalog_restart_segments.rs @@ -1,6 +1,6 @@ //! This module owns bounded loading of catalog-selected immutable segments. -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::loaded_segment::LoadedSegment; use super::segment_header::MAXIMUM_SEGMENT_LENGTH; diff --git a/src/adapters/darwin_filesystem.rs b/src/adapters/darwin_filesystem.rs new file mode 100644 index 00000000..78eb855c --- /dev/null +++ b/src/adapters/darwin_filesystem.rs @@ -0,0 +1,86 @@ +//! This module owns Darwin directory observations and local APFS admission. + +use std::{ + fs::File, + io, + path::{Component, Path}, +}; + +use cap_fs_ext::MetadataExt; +use cap_std::fs::Dir; +use rustix::fs::{CWD, Mode, OFlags, fstatfs, openat}; + +use super::filesystem_root_identity::FilesystemRootIdentity; +use super::platform_filesystem::PlatformFilesystem; + +/// Separates Darwin device coordinates from Linux ones in persisted records, so +/// a record written on one OS can never match a live root on the other. +const DARWIN_COORDINATE_DOMAIN: u64 = 0x8000_0000_0000_0000; + +pub(super) struct DarwinFilesystem; + +impl PlatformFilesystem for DarwinFilesystem { + fn open_root(&self, path: &Path) -> io::Result { + let start = if path.is_absolute() { "/" } else { "." }; + let mut current = openat(CWD, start, directory_flags(), Mode::empty())?; + for component in path.components() { + match component { + Component::RootDir | Component::CurDir => {} + Component::Normal(name) => { + current = openat(¤t, name, directory_flags(), Mode::empty())?; + } + Component::ParentDir => { + current = openat(¤t, "..", directory_flags(), Mode::empty())?; + } + Component::Prefix(_) => return Err(unsupported("unsupported root prefix")), + } + } + Ok(Dir::from_std_file(File::from(current))) + } + + fn admit_directory(&self, directory: &Dir) -> io::Result { + // Darwin's public mount flags. Native APFS admission excludes snapshots + // mounted read-only and network filesystems even if a name resembles APFS. + const READ_ONLY: u32 = 0x0000_0001; + const LOCAL: u32 = 0x0000_1000; + let observed = fstatfs(directory)?; + let apfs = observed + .f_fstypename + .into_iter() + .take(5) + .eq([97, 112, 102, 115, 0]); + if !apfs || observed.f_flags & READ_ONLY != 0 || observed.f_flags & LOCAL == 0 { + return Err(unsupported( + "directory is not on a writable local APFS volume", + )); + } + self.root_identity(directory) + } + + fn root_identity(&self, directory: &Dir) -> io::Result { + let metadata = directory.dir_metadata()?; + let device = u32::try_from(metadata.dev()) + .map_err(|source| io::Error::new(io::ErrorKind::Unsupported, source))?; + // Each mounted APFS volume has its own st_dev, and fsid only repeats it + // beside the filesystem type, so one observation serves both coordinates. + let device = DARWIN_COORDINATE_DOMAIN | u64::from(device); + Ok(FilesystemRootIdentity::new(device, device, metadata.ino())) + } + + #[cfg(any(test, feature = "repository-tasks"))] + fn repository_identity(&self, directory: &Dir) -> io::Result { + self.root_identity(directory) + } +} + +fn directory_flags() -> OFlags { + OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC +} + +fn unsupported(message: &'static str) -> io::Error { + io::Error::new(io::ErrorKind::Unsupported, message) +} + +#[cfg(test)] +#[path = "darwin_filesystem_tests.rs"] +mod tests; diff --git a/src/adapters/darwin_filesystem_tests.rs b/src/adapters/darwin_filesystem_tests.rs new file mode 100644 index 00000000..5aedae94 --- /dev/null +++ b/src/adapters/darwin_filesystem_tests.rs @@ -0,0 +1,189 @@ +//! Native APFS laws for root coordinates and exact protocol-entry spelling. + +use std::{error::Error, fs, io, path::Path, thread}; + +use cap_fs_ext::MetadataExt; +use cap_std::{ambient_authority, fs::OpenOptions}; + +use super::{DARWIN_COORDINATE_DOMAIN, DarwinFilesystem}; +use crate::adapters::filesystem_directory::FilesystemDirectory; +use crate::adapters::filesystem_entry_spelling::EntrySpellingMismatch; +use crate::adapters::filesystem_test_sandbox::TestDirectory; +use crate::adapters::platform_filesystem::PlatformFilesystem; + +fn open(path: &Path) -> io::Result { + FilesystemDirectory::open_ambient_dir(path, ambient_authority()) +} + +fn read_options() -> OpenOptions { + let mut options = OpenOptions::new(); + options.read(true); + options +} + +/// Observes this run's volume, so one law states both APFS naming variants. +fn folds_case(root: &Path) -> io::Result { + fs::write(root.join("variant-probe"), [])?; + let folds = root.join("VARIANT-PROBE").try_exists()?; + fs::remove_file(root.join("variant-probe"))?; + Ok(folds) +} + +fn assert_alias_refused(result: io::Result, expected: &str) { + let source = result.err(); + let mismatch = source + .as_ref() + .and_then(io::Error::get_ref) + .and_then(|inner| inner.downcast_ref::()); + assert_eq!( + mismatch, + Some(&EntrySpellingMismatch { + expected: expected.into() + }) + ); +} + +#[test] +fn apfs_root_coordinates_bind_the_opened_directory() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-root-coordinate")?; + let path = fs::canonicalize(sandbox.path())?; + let directory = DarwinFilesystem.open_root(&path)?; + let identity = DarwinFilesystem.admit_directory(&directory)?; + let metadata = directory.dir_metadata()?; + assert_eq!(identity.file(), metadata.ino()); + assert_eq!(identity.device(), DARWIN_COORDINATE_DOMAIN | metadata.dev()); + assert_eq!(identity.mount(), identity.device()); + assert_eq!(identity, DarwinFilesystem.root_identity(&directory)?); + Ok(()) +} + +#[test] +fn symbolic_root_components_are_refused() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-symbolic-root")?; + let root = fs::canonicalize(sandbox.path())?; + let real = root.join("real"); + let alias = root.join("alias"); + fs::create_dir(&real)?; + std::os::unix::fs::symlink(&real, &alias)?; + let result = DarwinFilesystem.open_root(&alias); + assert!(matches!(result, Err(source) + if source.raw_os_error() == Some(rustix::io::Errno::NOTDIR.raw_os_error()))); + Ok(()) +} + +#[test] +fn a_foreign_spelling_never_answers_for_a_protocol_name() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-case-alias")?; + let folds = folds_case(sandbox.path())?; + fs::write(sandbox.path().join("head"), b"foreign")?; + let directory = open(sandbox.path())?; + let result = directory.open_with("HEAD", &read_options()); + if folds { + assert_alias_refused(result, "HEAD"); + } else { + assert!(matches!(result, Err(source) if source.kind() == io::ErrorKind::NotFound)); + } + Ok(()) +} + +#[test] +fn a_non_ascii_fold_never_answers_for_a_protocol_name() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-unicode-alias")?; + // U+017F LATIN SMALL LETTER LONG S folds to ASCII `s` under full case folding. + fs::write(sandbox.path().join("\u{17f}tage"), b"foreign")?; + let directory = open(sandbox.path())?; + let result = directory.open_with("stage", &read_options()); + if sandbox.path().join("stage").try_exists()? { + assert_alias_refused(result, "stage"); + } else { + assert!(matches!(result, Err(source) if source.kind() == io::ErrorKind::NotFound)); + } + Ok(()) +} + +#[test] +fn mutation_through_a_foreign_spelling_leaves_the_entry_unchanged() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-mutation-alias")?; + let folds = folds_case(sandbox.path())?; + fs::write(sandbox.path().join("head"), b"foreign")?; + let directory = open(sandbox.path())?; + let removal = directory.remove_file("HEAD"); + assert!(removal.is_err()); + if folds { + assert_eq!( + removal.err().map(|source| source.kind()), + Some(io::ErrorKind::InvalidData) + ); + } + assert_eq!(fs::read(sandbox.path().join("head"))?, b"foreign"); + Ok(()) +} + +#[test] +fn hard_linked_publication_names_stay_admitted_while_another_link_is_open() +-> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-hard-link")?; + fs::write(sandbox.path().join("stage"), b"published")?; + fs::hard_link(sandbox.path().join("stage"), sandbox.path().join("HEAD"))?; + let directory = open(sandbox.path())?; + let held = directory.open_with("stage", &read_options())?; + let published = directory.open_with("HEAD", &read_options())?; + assert_eq!(held.metadata()?.ino(), published.metadata()?.ino()); + directory.open_with("stage", &read_options())?; + Ok(()) +} + +#[test] +fn absent_names_remain_available_to_create() -> Result<(), Box> { + let sandbox = TestDirectory::create("darwin-absent")?; + let directory = open(sandbox.path())?; + let mut options = OpenOptions::new(); + options.write(true).create_new(true); + directory.open_with("FORMAT", &options)?; + assert_eq!(fs::read_dir(sandbox.path())?.count(), 1); + Ok(()) +} + +#[test] +fn replacing_an_entry_does_not_refuse_a_concurrent_exact_lookup() -> Result<(), Box> { + const REPLACEMENTS: usize = 5_000; + let sandbox = TestDirectory::create("darwin-replace-race")?; + let root = sandbox.path().to_owned(); + for index in 0..64 { + fs::write(root.join(format!("neighbour-{index}")), [])?; + } + fs::write(root.join("HEAD"), b"0")?; + let directory = open(&root)?; + let publisher = thread::spawn(move || -> io::Result<()> { + for _ in 0..REPLACEMENTS { + fs::write(root.join("HEAD.next"), b"1")?; + fs::rename(root.join("HEAD.next"), root.join("HEAD"))?; + } + Ok(()) + }); + while !publisher.is_finished() { + directory.open_with("HEAD", &read_options())?; + } + publisher + .join() + .map_err(|_| io::Error::other("publisher panicked"))??; + Ok(()) +} + +#[test] +fn filesystems_other_than_apfs_are_refused() -> Result<(), Box> { + // devfs is always mounted and never writable through this adapter. + let directory = DarwinFilesystem.open_root(Path::new("/dev"))?; + let refusal = DarwinFilesystem.admit_directory(&directory); + assert!(matches!(refusal, Err(source) if source.kind() == io::ErrorKind::Unsupported)); + Ok(()) +} + +#[test] +fn read_only_apfs_volumes_are_refused() -> Result<(), Box> { + // The sealed system volume is APFS and mounted read-only. + let directory = DarwinFilesystem.open_root(Path::new("/System"))?; + let refusal = DarwinFilesystem.admit_directory(&directory); + assert!(matches!(refusal, Err(source) if source.kind() == io::ErrorKind::Unsupported)); + Ok(()) +} diff --git a/src/adapters/durable/snapshot.rs b/src/adapters/durable/snapshot.rs index dca8c1e3..bff8caf1 100644 --- a/src/adapters/durable/snapshot.rs +++ b/src/adapters/durable/snapshot.rs @@ -76,8 +76,8 @@ impl DurableSnapshot { /// reader fence, double-collects one consistent view within `limit` /// attempts, and verifies every retained root's closure. /// - /// The reader enforces the existing local writable, case-sensitive Linux - /// ext4 profile without acquiring writer authority. The admitted directory + /// The reader enforces an admitted local writable Linux ext4 or macOS + /// APFS profile without acquiring writer authority. The admitted directory /// capability is retained through view collection. Platform admission /// synchronizes the root directory before fencing; a synchronization /// failure is preserved under the snapshot's admission error. diff --git a/src/adapters/durable/store.rs b/src/adapters/durable/store.rs index 98950acc..1d9b1e61 100644 --- a/src/adapters/durable/store.rs +++ b/src/adapters/durable/store.rs @@ -34,7 +34,7 @@ use crate::{BlobId, ByteRange, LayoutId}; /// catalog bytes and decoded metadata allocate separately. /// /// ```no_run -/// #[cfg(target_os = "linux")] +/// #[cfg(any(target_os = "linux", target_os = "macos"))] /// fn copy_retained_blob( /// root: &std::path::Path, /// target: keep::BlobId, diff --git a/src/adapters/filesystem_catalog_artifact.rs b/src/adapters/filesystem_catalog_artifact.rs index 7adf710a..8339d3a9 100644 --- a/src/adapters/filesystem_catalog_artifact.rs +++ b/src/adapters/filesystem_catalog_artifact.rs @@ -3,8 +3,9 @@ use std::error::Error; use std::io; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, OpenOptionsFollowExt, OpenOptionsSyncExt}; -use cap_std::fs::{Dir, File, OpenOptions}; +use cap_std::fs::{File, OpenOptions}; use super::segment_header::MAXIMUM_SEGMENT_LENGTH; use super::{ diff --git a/src/adapters/filesystem_catalog_current.rs b/src/adapters/filesystem_catalog_current.rs index cf56a399..773c1b78 100644 --- a/src/adapters/filesystem_catalog_current.rs +++ b/src/adapters/filesystem_catalog_current.rs @@ -114,7 +114,7 @@ fn require_empty_durable_pools(publisher: &FilesystemCatalogPublisher) -> io::Re } fn require_empty_pool( - directory: &cap_std::fs::Dir, + directory: &crate::adapters::filesystem_directory::FilesystemDirectory, error: FilesystemCatalogPublicationError, ) -> io::Result<()> { match directory.entries()?.next() { @@ -125,7 +125,7 @@ fn require_empty_pool( } fn require_absent( - directory: &cap_std::fs::Dir, + directory: &crate::adapters::filesystem_directory::FilesystemDirectory, name: &str, error: FilesystemCatalogPublicationError, ) -> io::Result<()> { diff --git a/src/adapters/filesystem_catalog_publisher.rs b/src/adapters/filesystem_catalog_publisher.rs index 6034f662..875cb181 100644 --- a/src/adapters/filesystem_catalog_publisher.rs +++ b/src/adapters/filesystem_catalog_publisher.rs @@ -2,7 +2,8 @@ use std::io; -use cap_std::fs::{Dir, File}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; +use cap_std::fs::File; use super::filesystem_publisher_authority::FilesystemPublisherAuthority; use super::{ diff --git a/src/adapters/filesystem_directory.rs b/src/adapters/filesystem_directory.rs new file mode 100644 index 00000000..c92940e0 --- /dev/null +++ b/src/adapters/filesystem_directory.rs @@ -0,0 +1,132 @@ +//! This module owns capability-relative access to exact protocol entry names. +//! +//! No Deref exposes unchecked pathname operations. Each operation accepts one +//! normal component; nested protocol directories are traversed with new pinned +//! handles. Native spelling observations precede even truncation and removal. + +use std::{ + ffi::OsStr, + io, + path::{Component, Path}, +}; + +use cap_fs_ext::DirExt; +use cap_std::fs::{Dir as RawDirectory, File, Metadata, OpenOptions, ReadDir}; + +use super::filesystem_entry_spelling; + +#[derive(Debug)] +pub(super) struct FilesystemDirectory(RawDirectory); + +impl FilesystemDirectory { + pub(super) const fn from_raw(directory: RawDirectory) -> Self { + Self(directory) + } + + pub(super) const fn raw(&self) -> &RawDirectory { + &self.0 + } + + pub(super) fn from_std_file(file: std::fs::File) -> Self { + Self(RawDirectory::from_std_file(file)) + } + + pub(super) fn into_std_file(self) -> std::fs::File { + self.0.into_std_file() + } + + pub(super) fn open_ambient_dir( + path: impl AsRef, + authority: cap_std::AmbientAuthority, + ) -> io::Result { + RawDirectory::open_ambient_dir(path, authority).map(Self) + } + + pub(super) fn try_clone(&self) -> io::Result { + self.0.try_clone().map(Self) + } + + pub(super) fn dir_metadata(&self) -> io::Result { + self.0.dir_metadata() + } + + pub(super) fn entries(&self) -> io::Result { + self.0.entries() + } + + pub(super) fn open_with( + &self, + path: impl AsRef, + options: &OpenOptions, + ) -> io::Result { + let name = self.require_exact(path.as_ref())?; + self.0.open_with(name, options) + } + + pub(super) fn symlink_metadata(&self, path: impl AsRef) -> io::Result { + let name = self.require_exact(path.as_ref())?; + self.0.symlink_metadata(name) + } + + pub(super) fn open_dir_nofollow(&self, path: impl AsRef) -> io::Result { + let name = self.require_exact(path.as_ref())?; + self.0.open_dir_nofollow(name).map(Self) + } + + pub(super) fn create_dir(&self, path: impl AsRef) -> io::Result<()> { + let name = self.require_exact(path.as_ref())?; + self.0.create_dir(name) + } + + pub(super) fn remove_file(&self, path: impl AsRef) -> io::Result<()> { + let name = self.require_exact(path.as_ref())?; + self.0.remove_file(name) + } + + pub(super) fn hard_link( + &self, + source: impl AsRef, + destination: &Self, + target: impl AsRef, + ) -> io::Result<()> { + let source = self.require_exact(source.as_ref())?; + let target = destination.require_exact(target.as_ref())?; + self.0.hard_link(source, &destination.0, target) + } + + pub(super) fn rename( + &self, + source: impl AsRef, + destination: &Self, + target: impl AsRef, + ) -> io::Result<()> { + let source = self.require_exact(source.as_ref())?; + let target = destination.require_exact(target.as_ref())?; + self.0.rename(source, &destination.0, target) + } + + fn require_exact<'a>(&self, path: &'a Path) -> io::Result<&'a OsStr> { + // The directory itself has no entry to spell; synchronization reopens it. + if path == Path::new(".") { + return Ok(path.as_os_str()); + } + let mut components = path.components(); + let name = match (components.next(), components.next()) { + (Some(Component::Normal(name)), None) if path.as_os_str() == name => name, + _ => { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "protocol entry must be one normal component", + )); + } + }; + filesystem_entry_spelling::require_exact(&self.0, name)?; + Ok(name) + } +} + +impl std::os::fd::AsFd for FilesystemDirectory { + fn as_fd(&self) -> std::os::fd::BorrowedFd<'_> { + self.0.as_fd() + } +} diff --git a/src/adapters/filesystem_entry_spelling.rs b/src/adapters/filesystem_entry_spelling.rs new file mode 100644 index 00000000..a5262dd6 --- /dev/null +++ b/src/adapters/filesystem_entry_spelling.rs @@ -0,0 +1,63 @@ +//! This module owns exact protocol-entry spelling at the native lookup boundary. + +use std::{ffi::OsStr, io}; + +use cap_std::fs::Dir; + +/// Verifies an existing entry before a lookup may read, reuse, or mutate it. +/// Absence is permitted so create-new operations can retain their atomicity. +/// +/// Case-insensitive APFS resolves `head` when asked for `HEAD`, and folds more +/// than ASCII case. The directory listing is the only observation of the stored +/// spelling: `F_GETPATH` reports a name cached on the vnode, which for Keep's +/// hard-link publication can belong to another link of the same file. So a name +/// the lookup resolves must also appear byte-for-byte in the listing. This +/// costs one streaming scan of the parent per existing entry, allocates one name +/// at a time, and has no durability effect. +#[cfg(target_os = "macos")] +pub(super) fn require_exact(directory: &Dir, name: &OsStr) -> io::Result<()> { + match directory.symlink_metadata(name) { + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(()), + Err(source) => return Err(source), + Ok(_) => {} + } + for entry in directory.entries()? { + if entry?.file_name() == name { + return Ok(()); + } + } + Err(io::Error::new( + io::ErrorKind::InvalidData, + EntrySpellingMismatch { + expected: name.to_owned(), + }, + )) +} + +// Production Linux admission excludes casefold directories. Other platforms +// have no production admission; their repository fixtures retain old behavior. +#[cfg(not(target_os = "macos"))] +pub(super) fn require_exact(_directory: &Dir, _name: &OsStr) -> io::Result<()> { + Ok(()) +} + +/// The lookup resolved `expected`, but no directory entry has that spelling. +#[cfg(target_os = "macos")] +#[derive(Debug, Eq, PartialEq)] +pub(super) struct EntrySpellingMismatch { + pub(super) expected: std::ffi::OsString, +} + +#[cfg(target_os = "macos")] +impl std::fmt::Display for EntrySpellingMismatch { + fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!( + formatter, + "lookup resolved {} but no entry has that exact spelling", + self.expected.display() + ) + } +} + +#[cfg(target_os = "macos")] +impl std::error::Error for EntrySpellingMismatch {} diff --git a/src/adapters/filesystem_exact_record.rs b/src/adapters/filesystem_exact_record.rs index 44707fab..0150802d 100644 --- a/src/adapters/filesystem_exact_record.rs +++ b/src/adapters/filesystem_exact_record.rs @@ -12,8 +12,9 @@ use std::error::Error; use std::fmt; use std::io::{self, Read}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, MetadataExt, OpenOptionsFollowExt, OpenOptionsSyncExt}; -use cap_std::fs::{Dir, File, Metadata, OpenOptions}; +use cap_std::fs::{File, Metadata, OpenOptions}; /// Device and inode identity of one directory entry or open handle. #[derive(Clone, Copy, Debug, Eq, PartialEq)] diff --git a/src/adapters/filesystem_exact_record_tests.rs b/src/adapters/filesystem_exact_record_tests.rs index fa1dc3d0..0e78133f 100644 --- a/src/adapters/filesystem_exact_record_tests.rs +++ b/src/adapters/filesystem_exact_record_tests.rs @@ -3,7 +3,7 @@ use std::error::Error; use std::fs; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{ EntryIdentity, ExactRecordError, ExactRecordRefusal, read_exact_optional, read_exact_regular, @@ -88,7 +88,7 @@ fn verify_named_refuses_different_bytes_and_a_byte_equal_substitute() -> Result< let directory = open(&sandbox)?; let path = sandbox.path().join("record"); fs::write(&path, b"exact")?; - let identity = EntryIdentity::of_file(&directory.open("record")?)?; + let identity = EntryIdentity::of_file(&directory.raw().open("record")?)?; super::verify_named(&directory, "record", b"exact", identity)?; let bytes = super::verify_named(&directory, "record", b"other", identity) diff --git a/src/adapters/filesystem_initialization_namespace.rs b/src/adapters/filesystem_initialization_namespace.rs index e3a23352..8aef8a4d 100644 --- a/src/adapters/filesystem_initialization_namespace.rs +++ b/src/adapters/filesystem_initialization_namespace.rs @@ -6,8 +6,7 @@ use std::io; use super::filesystem_namespace_refusal::{ self as namespace, FilesystemEntryKind, FilesystemNamespaceRefusal, }; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; const LOCK_NAME: &str = "writer.lock"; const STAGING_NAME: &str = "staging"; diff --git a/src/adapters/filesystem_initialization_storage.rs b/src/adapters/filesystem_initialization_storage.rs index 833b47e1..be70590a 100644 --- a/src/adapters/filesystem_initialization_storage.rs +++ b/src/adapters/filesystem_initialization_storage.rs @@ -3,9 +3,9 @@ use std::io; use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(any(test, feature = "repository-tasks"))] use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::filesystem_initialization_namespace; use super::filesystem_platform_profile; diff --git a/src/adapters/filesystem_namespace_refusal.rs b/src/adapters/filesystem_namespace_refusal.rs index 2151da4a..ef039fe2 100644 --- a/src/adapters/filesystem_namespace_refusal.rs +++ b/src/adapters/filesystem_namespace_refusal.rs @@ -1,6 +1,7 @@ //! This module owns demonstrated protocol namespace contradictions. -use cap_std::fs::{Dir, FileType}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; +use cap_std::fs::FileType; use std::{error::Error, fmt, io}; /// No-follow kind observed for one protocol namespace entry. diff --git a/src/adapters/filesystem_platform_profile.rs b/src/adapters/filesystem_platform_profile.rs index 19ac8ebb..efcdb388 100644 --- a/src/adapters/filesystem_platform_profile.rs +++ b/src/adapters/filesystem_platform_profile.rs @@ -2,7 +2,7 @@ use std::{io, path::Path}; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_mount_refusal::FilesystemMountRefusal; use super::filesystem_root_identity::FilesystemRootIdentity; @@ -41,12 +41,12 @@ pub(super) fn admit_locked_root(directory: &Dir) -> io::Result io::Result { - NativeFilesystem.root_identity(directory) + NativeFilesystem.root_identity(directory.raw()) } #[cfg(any(test, feature = "repository-tasks"))] pub(super) fn root_identity_lenient(directory: &Dir) -> io::Result { - NativeFilesystem.repository_identity(directory) + NativeFilesystem.repository_identity(directory.raw()) } fn open_profile( @@ -54,7 +54,7 @@ fn open_profile( store_root: &Path, protocol_directories: &[&str], ) -> io::Result { - let directory = platform.open_root(store_root)?; + let directory = Dir::from_raw(platform.open_root(store_root)?); admit_profile(platform, &directory, protocol_directories)?; Ok(directory) } @@ -64,14 +64,14 @@ fn admit_profile( directory: &Dir, protocol_directories: &[&str], ) -> io::Result<()> { - let root = platform.admit_directory(directory)?; + let root = platform.admit_directory(directory.raw())?; for name in protocol_directories { let child = match open_protocol_directory(directory, name) { Ok(child) => child, Err(source) if source.kind() == io::ErrorKind::NotFound => continue, Err(source) => return Err(source), }; - let child = platform.admit_directory(&child)?; + let child = platform.admit_directory(child.raw())?; admit_child_identity(root, child)?; } directory.try_clone()?.into_std_file().sync_all() diff --git a/src/adapters/filesystem_recovery_inventory_reader.rs b/src/adapters/filesystem_recovery_inventory_reader.rs index 392522fb..6350f471 100644 --- a/src/adapters/filesystem_recovery_inventory_reader.rs +++ b/src/adapters/filesystem_recovery_inventory_reader.rs @@ -3,9 +3,9 @@ use std::io; use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(test)] use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::{ FilesystemRecoveryStageError, RecoveryEntryName, RecoveryInventory, RecoveryInventoryError, @@ -24,7 +24,7 @@ const CATALOGS_NAME: &str = "catalogs"; /// A pinned, read-only view of the four recovery inventory namespaces. /// /// Opening performs no protocol mutation. The production constructor admits -/// the same Linux ext4 profile as store initialization and pins all three child +/// the same platform profile as store initialization and pins all three child /// directories without following symbolic links. #[must_use] pub struct FilesystemRecoveryInventoryReader { diff --git a/src/adapters/filesystem_recovery_inventory_scan.rs b/src/adapters/filesystem_recovery_inventory_scan.rs index 7077be77..bc4272f9 100644 --- a/src/adapters/filesystem_recovery_inventory_scan.rs +++ b/src/adapters/filesystem_recovery_inventory_scan.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{RecoveryEntryName, RecoveryInventoryLimit}; diff --git a/src/adapters/filesystem_recovery_namespace.rs b/src/adapters/filesystem_recovery_namespace.rs index 6ced5dcc..3ace94e0 100644 --- a/src/adapters/filesystem_recovery_namespace.rs +++ b/src/adapters/filesystem_recovery_namespace.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_exact_record::EntryIdentity; diff --git a/src/adapters/filesystem_recovery_next_head_finalizer.rs b/src/adapters/filesystem_recovery_next_head_finalizer.rs index 1f1daeb0..59a6d3ad 100644 --- a/src/adapters/filesystem_recovery_next_head_finalizer.rs +++ b/src/adapters/filesystem_recovery_next_head_finalizer.rs @@ -2,7 +2,7 @@ use std::path::Path; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{ CatalogRestartPolicy, FilesystemRecoveryNextHeadFinalizationOpenError, diff --git a/src/adapters/filesystem_recovery_stage.rs b/src/adapters/filesystem_recovery_stage.rs index 3359dab7..95ca9e36 100644 --- a/src/adapters/filesystem_recovery_stage.rs +++ b/src/adapters/filesystem_recovery_stage.rs @@ -2,8 +2,9 @@ use std::io::{Seek, SeekFrom}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, MetadataExt, OpenOptionsFollowExt, OpenOptionsSyncExt}; -use cap_std::fs::{Dir, File, Metadata, OpenOptions}; +use cap_std::fs::{File, Metadata, OpenOptions}; use super::{ FilesystemRecoveryStageError, RecoveryStage, RecoveryStageEvidence, RecoveryStageLength, diff --git a/src/adapters/filesystem_recovery_stage_completion_pool.rs b/src/adapters/filesystem_recovery_stage_completion_pool.rs index a6884d95..ca26d7ca 100644 --- a/src/adapters/filesystem_recovery_stage_completion_pool.rs +++ b/src/adapters/filesystem_recovery_stage_completion_pool.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{ FilesystemRecoveryInventoryReader, FilesystemRecoveryStageCompleter, diff --git a/src/adapters/filesystem_recovery_stage_discard_storage.rs b/src/adapters/filesystem_recovery_stage_discard_storage.rs index 398507ed..4ffdaeb2 100644 --- a/src/adapters/filesystem_recovery_stage_discard_storage.rs +++ b/src/adapters/filesystem_recovery_stage_discard_storage.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_recovery_stage::ObservedRecoveryStage; use super::{ diff --git a/src/adapters/filesystem_recovery_stage_discarder.rs b/src/adapters/filesystem_recovery_stage_discarder.rs index e29cb928..960635f8 100644 --- a/src/adapters/filesystem_recovery_stage_discarder.rs +++ b/src/adapters/filesystem_recovery_stage_discarder.rs @@ -2,9 +2,9 @@ use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(any(test, feature = "repository-tasks"))] use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::{ FilesystemRecoveryInventoryReader, FilesystemRecoveryStageDiscardOpenError, diff --git a/src/adapters/filesystem_recovery_stage_sync.rs b/src/adapters/filesystem_recovery_stage_sync.rs index f5bd57b7..afcf8375 100644 --- a/src/adapters/filesystem_recovery_stage_sync.rs +++ b/src/adapters/filesystem_recovery_stage_sync.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{ FilesystemRecoveryStageError, RecoveryStageCompletionStorageError, RecoveryStageEvidence, diff --git a/src/adapters/filesystem_root_identity.rs b/src/adapters/filesystem_root_identity.rs index 36af8bba..4d2dcb09 100644 --- a/src/adapters/filesystem_root_identity.rs +++ b/src/adapters/filesystem_root_identity.rs @@ -10,7 +10,8 @@ pub(super) struct FilesystemRootIdentity { impl FilesystemRootIdentity { #[cfg(any( target_os = "linux", - all(not(target_os = "linux"), any(test, feature = "repository-tasks")) + target_os = "macos", + any(test, feature = "repository-tasks") ))] pub(super) const fn new(device: u64, mount: u64, file: u64) -> Self { Self { diff --git a/src/adapters/filesystem_store_initializer.rs b/src/adapters/filesystem_store_initializer.rs index ff2fb3a6..b120438e 100644 --- a/src/adapters/filesystem_store_initializer.rs +++ b/src/adapters/filesystem_store_initializer.rs @@ -3,9 +3,9 @@ use std::path::Path; #[cfg(test)] -use cap_std::ambient_authority; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(test)] -use cap_std::fs::Dir; +use cap_std::ambient_authority; use super::filesystem_initialization_storage::FilesystemInitializationStorage; use super::{ @@ -17,8 +17,8 @@ use super::{ impl FilesystemPlatformAdmission { /// Initializes or resumes one admitted filesystem store root. /// - /// The production adapter currently admits only a writable, - /// case-sensitive Linux ext4 root opened without symbolic links. It + /// The production adapter admits only a writable, case-sensitive Linux ext4 + /// root or a writable local macOS APFS root, opened without symbolic links. It /// validates an empty or partial canonical namespace before mutation, /// retains the exclusive writer lock, and returns only after synchronizing /// the complete root namespace. The call is synchronous, allocates no @@ -97,14 +97,16 @@ fn initialize_storage( } fn reopen_root( - root: cap_std::fs::Dir, + root: crate::adapters::filesystem_directory::FilesystemDirectory, ) -> Result { admit_reopened(root, filesystem_initialization_namespace::admit_published) } fn admit_reopened( - root: cap_std::fs::Dir, - admit_namespace: fn(&cap_std::fs::Dir) -> std::io::Result<()>, + root: crate::adapters::filesystem_directory::FilesystemDirectory, + admit_namespace: fn( + &crate::adapters::filesystem_directory::FilesystemDirectory, + ) -> std::io::Result<()>, ) -> Result { let lock = FilesystemWriterLock::try_acquire_in(root) .map_err(|source| FilesystemPlatformAdmissionError::WriterLock { source })?; diff --git a/src/adapters/filesystem_store_initializer_tests.rs b/src/adapters/filesystem_store_initializer_tests.rs index 96847878..518360f6 100644 --- a/src/adapters/filesystem_store_initializer_tests.rs +++ b/src/adapters/filesystem_store_initializer_tests.rs @@ -29,7 +29,7 @@ fn empty_namespace_is_admitted_only_with_the_complete_root_shape() -> Result<(), Ok(()) } -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[test] fn linux_initializer_synchronizes_an_opath_root() -> Result<(), Box> { let sandbox = TestDirectory::create("store-initialization-opath-sync")?; diff --git a/src/adapters/filesystem_version_two_admission.rs b/src/adapters/filesystem_version_two_admission.rs index 3ef32471..41ebbda9 100644 --- a/src/adapters/filesystem_version_two_admission.rs +++ b/src/adapters/filesystem_version_two_admission.rs @@ -2,10 +2,9 @@ use std::path::Path; -use cap_fs_ext::DirExt; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(any(test, feature = "repository-tasks"))] use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::filesystem_root_identity::FilesystemRootIdentity; pub(super) use super::filesystem_version_two_records::BoundRootIdentity; diff --git a/src/adapters/filesystem_version_two_records.rs b/src/adapters/filesystem_version_two_records.rs index acb2e1f7..d20723e6 100644 --- a/src/adapters/filesystem_version_two_records.rs +++ b/src/adapters/filesystem_version_two_records.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::VersionTwoRecordRefusal as Refusal; use super::filesystem_exact_record::{self as exact_record, ExactRecordError, ExactRecordRefusal}; diff --git a/src/adapters/filesystem_writer_lock.rs b/src/adapters/filesystem_writer_lock.rs index 3b43f6a1..8ed2053c 100644 --- a/src/adapters/filesystem_writer_lock.rs +++ b/src/adapters/filesystem_writer_lock.rs @@ -4,9 +4,10 @@ use std::fs::{File, TryLockError}; use std::io; use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, MetadataExt, OpenOptionsFollowExt, OpenOptionsSyncExt}; use cap_std::ambient_authority; -use cap_std::fs::{Dir, Metadata, OpenOptions}; +use cap_std::fs::{Metadata, OpenOptions}; use rustix::fs::{FlockOperation, Mode, OFlags, flock, openat}; use rustix::io::Errno; diff --git a/src/adapters/filesystem_writer_lock_tests.rs b/src/adapters/filesystem_writer_lock_tests.rs index ff6e371d..edaef39f 100644 --- a/src/adapters/filesystem_writer_lock_tests.rs +++ b/src/adapters/filesystem_writer_lock_tests.rs @@ -3,8 +3,8 @@ use std::error::Error; use std::fs; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::{FilesystemWriterLock, LOCK_FILE_NAME, acquire_root, open_existing}; use crate::adapters::filesystem_test_sandbox::TestDirectory; diff --git a/src/adapters/mod.rs b/src/adapters/mod.rs index 47dc6630..296d26cd 100644 --- a/src/adapters/mod.rs +++ b/src/adapters/mod.rs @@ -74,6 +74,8 @@ mod checksummed_catalog; mod checksummed_publication_head; mod checksummed_segment_record; mod closed_segment; +#[cfg(target_os = "macos")] +mod darwin_filesystem; mod decoded_catalog_entry; mod digest_hex; mod exports; @@ -88,6 +90,8 @@ mod filesystem_catalog_publisher_tests; mod filesystem_catalog_segment; mod filesystem_catalog_snapshot; mod filesystem_catalog_storage; +mod filesystem_directory; +mod filesystem_entry_spelling; mod filesystem_exact_record; mod filesystem_initialization_namespace; mod filesystem_initialization_storage; @@ -251,7 +255,7 @@ mod sync_capable_directory; #[cfg(test)] #[path = "../../tests/support/mod.rs"] mod test_support; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] mod unsupported_filesystem; mod verification_admission; mod verification_decode_error; diff --git a/src/adapters/native_filesystem.rs b/src/adapters/native_filesystem.rs index 9076eb28..07f68cdb 100644 --- a/src/adapters/native_filesystem.rs +++ b/src/adapters/native_filesystem.rs @@ -3,5 +3,8 @@ #[cfg(target_os = "linux")] pub(super) use super::linux_filesystem::LinuxFilesystem as NativeFilesystem; -#[cfg(not(target_os = "linux"))] +#[cfg(target_os = "macos")] +pub(super) use super::darwin_filesystem::DarwinFilesystem as NativeFilesystem; + +#[cfg(not(any(target_os = "linux", target_os = "macos")))] pub(super) use super::unsupported_filesystem::UnsupportedFilesystem as NativeFilesystem; diff --git a/src/adapters/retention.rs b/src/adapters/retention.rs index 7dcc6450..b6fa76b9 100644 --- a/src/adapters/retention.rs +++ b/src/adapters/retention.rs @@ -198,6 +198,8 @@ mod verified_closure; mod filesystem_retention_forward_error_tests; mod reader_attempt_limit; mod reader_fence; +#[cfg(test)] +mod reader_platform_authority_tests; #[cfg(all(test, target_os = "linux"))] mod reader_platform_law_tests; mod recovery_evidence; diff --git a/src/adapters/retention/filesystem_retention_attempt.rs b/src/adapters/retention/filesystem_retention_attempt.rs index 1edd7f69..be77328c 100644 --- a/src/adapters/retention/filesystem_retention_attempt.rs +++ b/src/adapters/retention/filesystem_retention_attempt.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_pool_name as pool_name; use super::filesystem_retention_stage::{FilesystemRetentionStage, invalid_data}; diff --git a/src/adapters/retention/filesystem_retention_authority.rs b/src/adapters/retention/filesystem_retention_authority.rs index 36d61efa..c2c5b4d5 100644 --- a/src/adapters/retention/filesystem_retention_authority.rs +++ b/src/adapters/retention/filesystem_retention_authority.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_attempt::PublicationAttempt; use super::filesystem_retention_authority_error::{ diff --git a/src/adapters/retention/filesystem_retention_catalog.rs b/src/adapters/retention/filesystem_retention_catalog.rs index a85c82b6..eb35fe02 100644 --- a/src/adapters/retention/filesystem_retention_catalog.rs +++ b/src/adapters/retention/filesystem_retention_catalog.rs @@ -2,9 +2,7 @@ use std::io; -use cap_std::fs::Dir; - -use cap_fs_ext::DirExt; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_current::read_exact_optional; use super::{RetentionCurrentStateRefusal, RetentionPublicationPreparation}; diff --git a/src/adapters/retention/filesystem_retention_closure_admission.rs b/src/adapters/retention/filesystem_retention_closure_admission.rs index d6e208e4..a579a6c0 100644 --- a/src/adapters/retention/filesystem_retention_closure_admission.rs +++ b/src/adapters/retention/filesystem_retention_closure_admission.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::{ RetentionRecoveryEvidence, RetentionStageAssessment, VerifiedRetentionClosure, diff --git a/src/adapters/retention/filesystem_retention_current.rs b/src/adapters/retention/filesystem_retention_current.rs index e12a79ce..77e30d4d 100644 --- a/src/adapters/retention/filesystem_retention_current.rs +++ b/src/adapters/retention/filesystem_retention_current.rs @@ -2,8 +2,7 @@ use std::io; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_pool_name as pool_name; use super::root_header_decoder; diff --git a/src/adapters/retention/filesystem_retention_namespace.rs b/src/adapters/retention/filesystem_retention_namespace.rs index 75b31617..9ad1d52e 100644 --- a/src/adapters/retention/filesystem_retention_namespace.rs +++ b/src/adapters/retention/filesystem_retention_namespace.rs @@ -2,8 +2,7 @@ use std::io; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::AdmittedRetentionRoot; use super::RetentionCurrentStateRefusal as Refusal; diff --git a/src/adapters/retention/filesystem_retention_recovery.rs b/src/adapters/retention/filesystem_retention_recovery.rs index 60e843d3..e6769622 100644 --- a/src/adapters/retention/filesystem_retention_recovery.rs +++ b/src/adapters/retention/filesystem_retention_recovery.rs @@ -7,8 +7,7 @@ use super::{ }; use std::io; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_authority::FilesystemRetentionPublicationAuthority; use super::filesystem_retention_pool_name as pool_name; diff --git a/src/adapters/retention/filesystem_retention_recovery_observation.rs b/src/adapters/retention/filesystem_retention_recovery_observation.rs index b756f1fd..9a3dcb87 100644 --- a/src/adapters/retention/filesystem_retention_recovery_observation.rs +++ b/src/adapters/retention/filesystem_retention_recovery_observation.rs @@ -2,8 +2,7 @@ use std::io::{self, Read}; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_current::{self, ObservedRetentionState}; use super::filesystem_retention_pool_name as pool_name; diff --git a/src/adapters/retention/filesystem_retention_recovery_roots.rs b/src/adapters/retention/filesystem_retention_recovery_roots.rs index ec949365..e396fc80 100644 --- a/src/adapters/retention/filesystem_retention_recovery_roots.rs +++ b/src/adapters/retention/filesystem_retention_recovery_roots.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_current::{verify_committed, verify_predecessor}; use super::{RetentionRecoveryEvidence, RetentionStageAssessment}; diff --git a/src/adapters/retention/filesystem_retention_snapshot.rs b/src/adapters/retention/filesystem_retention_snapshot.rs index 0cd2e9c1..8bc35f6e 100644 --- a/src/adapters/retention/filesystem_retention_snapshot.rs +++ b/src/adapters/retention/filesystem_retention_snapshot.rs @@ -3,8 +3,7 @@ use std::io; use std::path::Path; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_current::{self, ObservedRetentionState}; use super::filesystem_retention_pool_name as pool_name; @@ -116,8 +115,8 @@ impl FilesystemRetentionSnapshot { /// Platform admission synchronizes the opened root directory; failure is /// returned as `Admission` with the original I/O cause. It may also block /// while collection holds the fence exclusively. - /// Platform admission requires the existing local writable, case-sensitive - /// Linux ext4 profile across every present version-two protocol directory. + /// Platform admission requires an admitted local writable Linux ext4 or + /// macOS APFS profile across every present version-two protocol directory. /// The same opened root capability is retained through namespace, migration /// identity, fence, and coordinate admission; the ambient path is not reopened. /// diff --git a/src/adapters/retention/filesystem_retention_snapshot_coordinate_tests.rs b/src/adapters/retention/filesystem_retention_snapshot_coordinate_tests.rs index 35ff35cd..18b65fab 100644 --- a/src/adapters/retention/filesystem_retention_snapshot_coordinate_tests.rs +++ b/src/adapters/retention/filesystem_retention_snapshot_coordinate_tests.rs @@ -6,6 +6,7 @@ use std::io; use std::num::NonZeroU32; use super::{Source, collect_retention_view, pool_name}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::retention::filesystem_retention_test_fixture::migrated_store; use crate::adapters::retention::{ ReaderAttemptLimit, ReaderFence, RetentionViewCoordinates, RetentionViewError, @@ -15,8 +16,6 @@ use crate::adapters::{ CatalogRestartByteLimit, CatalogRestartPolicy, ChecksummedPublicationHead, SegmentReadPolicy, publication_head_decoder, }; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; struct RewriteHead { source: Source, diff --git a/src/adapters/retention/filesystem_retention_snapshot_moving_error_tests.rs b/src/adapters/retention/filesystem_retention_snapshot_moving_error_tests.rs index c9239865..0577c826 100644 --- a/src/adapters/retention/filesystem_retention_snapshot_moving_error_tests.rs +++ b/src/adapters/retention/filesystem_retention_snapshot_moving_error_tests.rs @@ -5,6 +5,7 @@ use std::fs; use std::io; use super::{Source, collect_retention_view, pool_name}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::retention::filesystem_retention_test_fixture::{ CATALOG_NAME, ROOT_HEX, fixture, initial_preparation, open_authority, }; @@ -13,8 +14,6 @@ use crate::adapters::retention::{ }; use crate::adapters::{CatalogRestartByteLimit, CatalogRestartPolicy, SegmentReadPolicy}; use crate::execute_retention_publication; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; struct AfterLoad { source: Source, diff --git a/src/adapters/retention/filesystem_retention_snapshot_pinning_tests.rs b/src/adapters/retention/filesystem_retention_snapshot_pinning_tests.rs index 7f5a8a58..eae3bbc4 100644 --- a/src/adapters/retention/filesystem_retention_snapshot_pinning_tests.rs +++ b/src/adapters/retention/filesystem_retention_snapshot_pinning_tests.rs @@ -4,12 +4,11 @@ use std::error::Error; use std::fs; use super::{Source, collect_retention_view, pool_name}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::filesystem_test_sandbox::TestDirectory; use crate::adapters::retention::filesystem_retention_test_fixture::migrated_store; use crate::adapters::retention::{ReaderAttemptLimit, RetentionViewSource}; use crate::adapters::{CatalogRestartByteLimit, CatalogRestartPolicy, SegmentReadPolicy}; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; // Size: medium (owned filesystem). Oracle: a pinned reader's catalog must // match its original validated HEAD despite replacement of the ambient path. diff --git a/src/adapters/retention/filesystem_retention_snapshot_tests.rs b/src/adapters/retention/filesystem_retention_snapshot_tests.rs index abbff4ac..50000cb9 100644 --- a/src/adapters/retention/filesystem_retention_snapshot_tests.rs +++ b/src/adapters/retention/filesystem_retention_snapshot_tests.rs @@ -3,7 +3,7 @@ use std::error::Error; use std::fs; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use rustix::fs::{FlockOperation, flock}; use rustix::io::Errno; diff --git a/src/adapters/retention/filesystem_retention_stage.rs b/src/adapters/retention/filesystem_retention_stage.rs index 259c21e9..255535bc 100644 --- a/src/adapters/retention/filesystem_retention_stage.rs +++ b/src/adapters/retention/filesystem_retention_stage.rs @@ -7,7 +7,8 @@ use super::{ use super::{RetentionRecordRefusal, RetentionStorageError}; use std::io::{self, Write}; -use cap_std::fs::{Dir, File}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; +use cap_std::fs::File; use crate::adapters::filesystem_catalog_artifact; use crate::adapters::filesystem_exact_record::{ diff --git a/src/adapters/retention/filesystem_retention_storage.rs b/src/adapters/retention/filesystem_retention_storage.rs index a9f756c1..a44f807c 100644 --- a/src/adapters/retention/filesystem_retention_storage.rs +++ b/src/adapters/retention/filesystem_retention_storage.rs @@ -2,8 +2,7 @@ use std::io; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_retention_attempt::{self as attempt, PublicationAttempt}; use super::filesystem_retention_authority::FilesystemRetentionPublicationAuthority; diff --git a/src/adapters/retention/filesystem_version_two_admission_tests.rs b/src/adapters/retention/filesystem_version_two_admission_tests.rs index 98fe3a28..6fa956bd 100644 --- a/src/adapters/retention/filesystem_version_two_admission_tests.rs +++ b/src/adapters/retention/filesystem_version_two_admission_tests.rs @@ -95,7 +95,7 @@ fn version_two_reopen_admits_exact_migration_records() -> Result<(), Box Result<(), Box> { @@ -116,7 +116,7 @@ fn production_version_two_reopen_refuses_an_aliased_protocol_directory() Ok(()) } -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[test] fn production_version_two_reopen_admits_an_exact_migrated_store() -> Result<(), Box> { let sandbox = migrated_store("version-two-admission-production-exact")?; diff --git a/src/adapters/retention/reader_fence.rs b/src/adapters/retention/reader_fence.rs index 9218aff3..1df281ca 100644 --- a/src/adapters/retention/reader_fence.rs +++ b/src/adapters/retention/reader_fence.rs @@ -2,8 +2,9 @@ use std::io; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::MetadataExt; -use cap_std::fs::{Dir, File, Metadata}; +use cap_std::fs::{File, Metadata}; use rustix::fs::{FlockOperation, flock}; use crate::adapters::filesystem_exact_record; diff --git a/src/adapters/retention/reader_fence_tests.rs b/src/adapters/retention/reader_fence_tests.rs index b89fbfb8..f8857cce 100644 --- a/src/adapters/retention/reader_fence_tests.rs +++ b/src/adapters/retention/reader_fence_tests.rs @@ -5,8 +5,8 @@ use std::fs; use std::io; use super::ReaderFence; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::retention::filesystem_retention_test_fixture::migrated_store; -use cap_std::fs::Dir; // Size: medium (owned filesystem). Oracle: a fence must lock the same inode // its directory entry names after acquisition, even when both files are empty. diff --git a/src/adapters/retention/reader_platform_authority_tests.rs b/src/adapters/retention/reader_platform_authority_tests.rs new file mode 100644 index 00000000..dd234738 --- /dev/null +++ b/src/adapters/retention/reader_platform_authority_tests.rs @@ -0,0 +1,33 @@ +//! Reader admission on every admitted platform leaves writer authority alone. +//! +//! Size: medium. Oracle: a reader admits its catalog while another authority +//! holds the writer lock. Delete when stronger public reader laws subsume it. + +use std::error::Error; + +use super::filesystem_retention_test_fixture::migrated_store; +use crate::{ + CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemRetentionSnapshot, + FilesystemWriterLock, ReaderAttemptLimit, SegmentReadPolicy, +}; + +#[test] +fn reader_platform_admission_does_not_acquire_writer_authority() -> Result<(), Box> { + let store = migrated_store("reader-platform-existing-writer")?; + let writer = FilesystemWriterLock::try_acquire(store.path())?; + let policy = CatalogRestartPolicy::new( + SegmentReadPolicy::MAXIMUM, + CatalogRestartByteLimit::new(1_048_576)?, + ); + + let snapshot = + FilesystemRetentionSnapshot::load(store.path(), policy, ReaderAttemptLimit::DEFAULT)?; + + assert_eq!( + snapshot.catalog().generation().get(), + 1, + "reader must admit its catalog while another authority holds the writer lock" + ); + drop(writer); + Ok(()) +} diff --git a/src/adapters/retention/reader_platform_law_tests.rs b/src/adapters/retention/reader_platform_law_tests.rs index 3f31c1c7..3e4146b5 100644 --- a/src/adapters/retention/reader_platform_law_tests.rs +++ b/src/adapters/retention/reader_platform_law_tests.rs @@ -50,23 +50,6 @@ fn a_canonical_tmpfs_store_refuses_durable_snapshot_admission() -> Result<(), Bo Ok(()) } -#[test] -fn reader_platform_admission_does_not_acquire_writer_authority() -> Result<(), Box> { - let store = migrated_store("reader-platform-existing-writer")?; - let writer = FilesystemWriterLock::try_acquire(store.path())?; - - let snapshot = - FilesystemRetentionSnapshot::load(store.path(), policy()?, ReaderAttemptLimit::DEFAULT)?; - - assert_eq!( - snapshot.catalog().generation().get(), - 1, - "reader must admit its catalog while another authority holds the writer lock" - ); - drop(writer); - Ok(()) -} - struct TmpfsStore(PathBuf); impl TmpfsStore { diff --git a/src/adapters/store_migration/filesystem_inventory_catalogs.rs b/src/adapters/store_migration/filesystem_inventory_catalogs.rs index ebed9d8c..0da28216 100644 --- a/src/adapters/store_migration/filesystem_inventory_catalogs.rs +++ b/src/adapters/store_migration/filesystem_inventory_catalogs.rs @@ -2,7 +2,7 @@ use std::collections::TryReserveError; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::StoreMigrationInventoryEntry; use super::filesystem_inventory_catalog_errors; diff --git a/src/adapters/store_migration/filesystem_inventory_catalogs_test_fixture.rs b/src/adapters/store_migration/filesystem_inventory_catalogs_test_fixture.rs index c9a45c3b..07bde1eb 100644 --- a/src/adapters/store_migration/filesystem_inventory_catalogs_test_fixture.rs +++ b/src/adapters/store_migration/filesystem_inventory_catalogs_test_fixture.rs @@ -4,8 +4,8 @@ use std::error::Error; use std::fs; use std::path::{Path, PathBuf}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_std::ambient_authority; -use cap_std::fs::Dir; use crate::LayoutEntryLimit; use crate::adapters::filesystem_test_sandbox::TestDirectory; diff --git a/src/adapters/store_migration/filesystem_inventory_directory.rs b/src/adapters/store_migration/filesystem_inventory_directory.rs index e48f836f..966e146a 100644 --- a/src/adapters/store_migration/filesystem_inventory_directory.rs +++ b/src/adapters/store_migration/filesystem_inventory_directory.rs @@ -1,6 +1,6 @@ //! This module owns pinned migration pool-directory identity. -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::filesystem_exact_record::EntryIdentity; diff --git a/src/adapters/store_migration/filesystem_inventory_file.rs b/src/adapters/store_migration/filesystem_inventory_file.rs index 70080165..38e18581 100644 --- a/src/adapters/store_migration/filesystem_inventory_file.rs +++ b/src/adapters/store_migration/filesystem_inventory_file.rs @@ -1,7 +1,8 @@ //! This module owns identity-stable migration artifact reads. +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::MetadataExt; -use cap_std::fs::{Dir, File, Metadata}; +use cap_std::fs::{File, Metadata}; use crate::adapters::{ CatalogRestartArtifact, CatalogRestartError, CatalogRestartPhase, catalog_restart_io, diff --git a/src/adapters/store_migration/filesystem_inventory_file_tests.rs b/src/adapters/store_migration/filesystem_inventory_file_tests.rs index 57359501..e3202beb 100644 --- a/src/adapters/store_migration/filesystem_inventory_file_tests.rs +++ b/src/adapters/store_migration/filesystem_inventory_file_tests.rs @@ -4,8 +4,8 @@ use std::error::Error; use std::fs; use std::io; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::filesystem_inventory_file::{ self, FilesystemInventoryFileError, FilesystemInventoryFilePolicy, diff --git a/src/adapters/store_migration/filesystem_inventory_names.rs b/src/adapters/store_migration/filesystem_inventory_names.rs index ce76dcdc..6a326cd7 100644 --- a/src/adapters/store_migration/filesystem_inventory_names.rs +++ b/src/adapters/store_migration/filesystem_inventory_names.rs @@ -1,6 +1,6 @@ //! This module owns deterministic bounded migration pool-name scans. -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_inventory_error::{ FilesystemMigrationInventoryError, FilesystemMigrationInventoryOperation, diff --git a/src/adapters/store_migration/filesystem_inventory_names_tests.rs b/src/adapters/store_migration/filesystem_inventory_names_tests.rs index 729b0cc0..5a618594 100644 --- a/src/adapters/store_migration/filesystem_inventory_names_tests.rs +++ b/src/adapters/store_migration/filesystem_inventory_names_tests.rs @@ -3,8 +3,8 @@ use std::error::Error; use std::fs; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_std::ambient_authority; -use cap_std::fs::Dir; use super::filesystem_inventory_error::{ FilesystemMigrationInventoryError, MigrationInventoryPool, diff --git a/src/adapters/store_migration/filesystem_inventory_reader.rs b/src/adapters/store_migration/filesystem_inventory_reader.rs index 2519a9ce..f5013ebc 100644 --- a/src/adapters/store_migration/filesystem_inventory_reader.rs +++ b/src/adapters/store_migration/filesystem_inventory_reader.rs @@ -1,6 +1,6 @@ //! This module owns writer-locked filesystem migration pool inventory. -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_inventory_catalogs; use super::filesystem_inventory_catalogs::FilesystemMigrationCatalogInventory; diff --git a/src/adapters/store_migration/filesystem_inventory_segments.rs b/src/adapters/store_migration/filesystem_inventory_segments.rs index 9aa77226..ac93819c 100644 --- a/src/adapters/store_migration/filesystem_inventory_segments.rs +++ b/src/adapters/store_migration/filesystem_inventory_segments.rs @@ -2,7 +2,7 @@ use std::collections::TryReserveError; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::StoreMigrationInventoryEntry; use super::filesystem_inventory_error::{ diff --git a/src/adapters/store_migration/filesystem_inventory_segments_test_fixture.rs b/src/adapters/store_migration/filesystem_inventory_segments_test_fixture.rs index 7a399046..da90b5c8 100644 --- a/src/adapters/store_migration/filesystem_inventory_segments_test_fixture.rs +++ b/src/adapters/store_migration/filesystem_inventory_segments_test_fixture.rs @@ -4,8 +4,8 @@ use std::error::Error; use std::fs; use std::path::Path; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_std::ambient_authority; -use cap_std::fs::Dir; use crate::LayoutEntryLimit; use crate::adapters::filesystem_test_sandbox::TestDirectory; diff --git a/src/adapters/store_migration/filesystem_migration_authority.rs b/src/adapters/store_migration/filesystem_migration_authority.rs index 44221757..70ff1dbb 100644 --- a/src/adapters/store_migration/filesystem_migration_authority.rs +++ b/src/adapters/store_migration/filesystem_migration_authority.rs @@ -1,7 +1,5 @@ //! This module owns exact writer-locked filesystem migration authority. -use cap_fs_ext::DirExt; - use super::filesystem_inventory_file::{self, FilesystemInventoryFilePolicy}; use super::filesystem_migration_authority_error::{ FilesystemMigrationAuthorityArtifact as Artifact, FilesystemMigrationAuthorityError as Error, @@ -221,7 +219,7 @@ impl FilesystemStoreMigrationAuthority { verify_catalog(head, catalog) } - pub(super) const fn root(&self) -> &cap_std::fs::Dir { + pub(super) const fn root(&self) -> &crate::adapters::filesystem_directory::FilesystemDirectory { self.inventory.root() } } diff --git a/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs b/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs index 5a9f52b7..ff1726b0 100644 --- a/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs +++ b/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs @@ -2,7 +2,8 @@ use std::io::{self, Write}; -use cap_std::fs::{Dir, File}; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; +use cap_std::fs::File; use super::filesystem_migration_recovery_refusal::{ FilesystemMigrationRecoveryRefusal as Refusal, invalid, diff --git a/src/adapters/store_migration/filesystem_migration_namespace.rs b/src/adapters/store_migration/filesystem_migration_namespace.rs index 0343d15a..d8b9be93 100644 --- a/src/adapters/store_migration/filesystem_migration_namespace.rs +++ b/src/adapters/store_migration/filesystem_migration_namespace.rs @@ -2,7 +2,7 @@ use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_migration_namespace_directory::{ PinnedMigrationDirectory, ambiguous, exact_membership, optional_directory, diff --git a/src/adapters/store_migration/filesystem_migration_namespace_directory.rs b/src/adapters/store_migration/filesystem_migration_namespace_directory.rs index 297dd829..eb3a7a04 100644 --- a/src/adapters/store_migration/filesystem_migration_namespace_directory.rs +++ b/src/adapters/store_migration/filesystem_migration_namespace_directory.rs @@ -3,7 +3,7 @@ use std::ffi::OsStr; use std::io; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use crate::adapters::filesystem_exact_record::EntryIdentity; diff --git a/src/adapters/store_migration/filesystem_migration_reader_fence.rs b/src/adapters/store_migration/filesystem_migration_reader_fence.rs index 7465fb12..52b37279 100644 --- a/src/adapters/store_migration/filesystem_migration_reader_fence.rs +++ b/src/adapters/store_migration/filesystem_migration_reader_fence.rs @@ -2,8 +2,9 @@ use std::io; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{FollowSymlinks, MetadataExt, OpenOptionsFollowExt, OpenOptionsSyncExt}; -use cap_std::fs::{Dir, File, Metadata, OpenOptions}; +use cap_std::fs::{File, Metadata, OpenOptions}; use super::filesystem_migration_namespace_directory::ambiguous; diff --git a/src/adapters/store_migration/filesystem_migration_recovery.rs b/src/adapters/store_migration/filesystem_migration_recovery.rs index 23151f43..f0e91799 100644 --- a/src/adapters/store_migration/filesystem_migration_recovery.rs +++ b/src/adapters/store_migration/filesystem_migration_recovery.rs @@ -3,7 +3,7 @@ use std::io; use std::path::Path; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::FilesystemMigrationRecoveryRefusal as Refusal; use super::filesystem_migration_authority::MigrationNamespacePolicy; diff --git a/src/adapters/store_migration/filesystem_migration_repository_tasks.rs b/src/adapters/store_migration/filesystem_migration_repository_tasks.rs index e7579f3e..18567220 100644 --- a/src/adapters/store_migration/filesystem_migration_repository_tasks.rs +++ b/src/adapters/store_migration/filesystem_migration_repository_tasks.rs @@ -9,7 +9,7 @@ use std::io; use std::path::Path; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::filesystem_migration_authority::MigrationNamespacePolicy; use super::filesystem_migration_authority_error::FilesystemMigrationAuthorityError as Error; diff --git a/src/adapters/store_migration/filesystem_migration_residue.rs b/src/adapters/store_migration/filesystem_migration_residue.rs index 0552d5c4..0af404af 100644 --- a/src/adapters/store_migration/filesystem_migration_residue.rs +++ b/src/adapters/store_migration/filesystem_migration_residue.rs @@ -2,8 +2,7 @@ use std::io; -use cap_fs_ext::DirExt; -use cap_std::fs::Dir; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::FilesystemMigrationRecoveryRefusal as Refusal; use super::filesystem_migration_recovery_refusal::{invalid, kind}; diff --git a/src/adapters/sync_capable_directory.rs b/src/adapters/sync_capable_directory.rs index 0951a11c..76729b9f 100644 --- a/src/adapters/sync_capable_directory.rs +++ b/src/adapters/sync_capable_directory.rs @@ -2,10 +2,11 @@ use std::io; +use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use cap_fs_ext::{ FollowSymlinks, OpenOptionsFollowExt, OpenOptionsMaybeDirExt, OpenOptionsSyncExt, }; -use cap_std::fs::{Dir, OpenOptions}; +use cap_std::fs::OpenOptions; pub(super) fn open(parent: &Dir, name: &str) -> io::Result { let mut options = OpenOptions::new(); diff --git a/src/adapters/unsupported_filesystem.rs b/src/adapters/unsupported_filesystem.rs index 2b471e0a..9a869f69 100644 --- a/src/adapters/unsupported_filesystem.rs +++ b/src/adapters/unsupported_filesystem.rs @@ -45,6 +45,6 @@ impl PlatformFilesystem for UnsupportedFilesystem { fn unsupported() -> io::Error { io::Error::new( io::ErrorKind::Unsupported, - "filesystem initialization currently requires the admitted Linux ext4 profile", + "filesystem initialization requires an admitted Linux ext4 or macOS APFS profile", ) } diff --git a/src/lib.rs b/src/lib.rs index 74ef5b55..818039b8 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -14,7 +14,7 @@ //! immutable-segment writing and verified reading, canonical catalog //! generations, platform-gated filesystem publication mechanics, bounded //! immutable restart snapshots, typed store-initialization orchestration, and -//! production initialization for the admitted Linux ext4 profile. Recovery +//! production initialization for the admitted Linux ext4 and macOS APFS profiles. Recovery //! inventory, name classification, bounded stage fingerprinting, exact //! truncated-stage discard, and complete-stage valid-orphan recovery are //! explicit. Exact next-head finalization now has a storage-independent diff --git a/tests/catalog_platform_admission.rs b/tests/catalog_platform_admission.rs index 3e6b35fd..73ffc6f2 100644 --- a/tests/catalog_platform_admission.rs +++ b/tests/catalog_platform_admission.rs @@ -1,24 +1,32 @@ //! Public platform-admission laws; medium Linux filesystem evidence for #150. //! Oracle: KEEP-CATALOG-007 requires platform admission independently of lock ownership. -#![cfg(all(target_os = "linux", feature = "repository-tasks"))] +#![cfg(all( + any(target_os = "linux", target_os = "macos"), + feature = "repository-tasks" +))] #[path = "segment_filesystem_stage/sandbox.rs"] pub mod admitted_directory; +#[cfg(target_os = "linux")] #[path = "catalog_platform_admission/unsupported_directory.rs"] pub mod unsupported_directory; use std::error::Error; use std::fs; +#[cfg(target_os = "linux")] use std::io::ErrorKind; use std::io::Write; use keep::{ CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemCatalogPublisher, FilesystemPlatformAdmission, FilesystemWriterLock, LayoutEntryLimit, SegmentReadPolicy, - SegmentRecordLimit, StoreInitializationError, StoreInitializationPhase, + SegmentRecordLimit, }; +#[cfg(target_os = "linux")] +use keep::{StoreInitializationError, StoreInitializationPhase}; +#[cfg(target_os = "linux")] #[test] fn a_writer_lock_cannot_mint_publication_authority_on_a_refused_platform() -> Result<(), Box> { diff --git a/tests/darwin_fifo_refusal.rs b/tests/darwin_fifo_refusal.rs new file mode 100644 index 00000000..b2725a6a --- /dev/null +++ b/tests/darwin_fifo_refusal.rs @@ -0,0 +1,50 @@ +//! A FIFO at a protocol name refuses instead of blocking the reader on Darwin. +//! +//! Size: medium. Oracle: the Linux FIFO laws create FIFOs with `mknodat`, which +//! Rustix omits on Apple targets. Spawning `mkfifo(1)` can leak flock holders to +//! concurrent laws until exec, so this law owns its own test executable. +//! Delete when a safe in-process FIFO constructor reaches the pinned toolchain. + +#![cfg(all(feature = "repository-tasks", target_os = "macos"))] + +#[allow(dead_code, reason = "this law uses only the migrated-store fixture")] +#[path = "reader_fence_process/fixture.rs"] +pub mod fixture; +#[allow(dead_code, reason = "this law uses Drop cleanup")] +#[path = "segment_filesystem_stage/sandbox.rs"] +pub mod sandbox; + +use std::{error::Error, fs, process::Command, sync::mpsc, thread, time::Duration}; + +use keep::{ + CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemRetentionSnapshot, ReaderAttemptLimit, + SegmentReadPolicy, +}; + +const DEADLINE: Duration = Duration::from_secs(20); + +#[test] +fn a_fifo_at_the_head_refuses_without_blocking_the_reader() -> Result<(), Box> { + let (store, writer) = fixture::migrated("darwin-fifo-head")?; + drop(writer); + let head = store.path().join("HEAD"); + fs::remove_file(&head)?; + assert!( + Command::new("/usr/bin/mkfifo") + .arg(&head) + .status()? + .success() + ); + let root = store.path().to_owned(); + let (sender, receiver) = mpsc::channel(); + let _reader = thread::spawn(move || { + let policy = CatalogRestartByteLimit::new(1_048_576) + .map(|limit| CatalogRestartPolicy::new(SegmentReadPolicy::MAXIMUM, limit)); + let refused = policy.is_ok_and(|policy| { + FilesystemRetentionSnapshot::load(&root, policy, ReaderAttemptLimit::DEFAULT).is_err() + }); + let _ = sender.send(refused); + }); + assert!(receiver.recv_timeout(DEADLINE)?, "a FIFO head was admitted"); + Ok(()) +} diff --git a/tests/darwin_store_spelling.rs b/tests/darwin_store_spelling.rs new file mode 100644 index 00000000..2e938430 --- /dev/null +++ b/tests/darwin_store_spelling.rs @@ -0,0 +1,63 @@ +//! A respelled protocol name never admits a complete store on either APFS variant. +//! +//! Size: medium. Oracle: a case-only rename leaves every byte intact. Namespace +//! admission refuses the foreign entry from its directory listing, independently +//! of the per-lookup spelling check, which `darwin_filesystem` tests pin directly. +//! Delete when public laws over a shared platform corpus subsume it. + +#![cfg(all(feature = "repository-tasks", target_os = "macos"))] + +#[allow(dead_code, reason = "these laws use only the migrated-store fixture")] +#[path = "reader_fence_process/fixture.rs"] +pub mod fixture; +#[allow(dead_code, reason = "these laws use Drop cleanup")] +#[path = "segment_filesystem_stage/sandbox.rs"] +pub mod sandbox; + +use std::{error::Error, fs, path::Path}; + +use keep::{ + CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemRetentionSnapshot, ReaderAttemptLimit, + SegmentReadPolicy, +}; + +fn load(root: &Path) -> Result> { + let policy = CatalogRestartPolicy::new( + SegmentReadPolicy::MAXIMUM, + CatalogRestartByteLimit::new(1_048_576)?, + ); + Ok(FilesystemRetentionSnapshot::load( + root, + policy, + ReaderAttemptLimit::DEFAULT, + )?) +} + +fn assert_respelling_refuses(law: &str, stored: &str, foreign: &str) -> Result<(), Box> { + let (store, writer) = fixture::migrated(law)?; + drop(writer); + drop(load(store.path())?); + fs::rename(store.path().join(stored), store.path().join(foreign))?; + assert!( + load(store.path()).is_err(), + "{foreign} answered for {stored}" + ); + fs::rename(store.path().join(foreign), store.path().join(stored))?; + drop(load(store.path())?); + Ok(()) +} + +#[test] +fn a_respelled_head_refuses_until_its_exact_name_returns() -> Result<(), Box> { + assert_respelling_refuses("darwin-respelled-head", "HEAD", "head") +} + +#[test] +fn a_respelled_format_marker_refuses_until_its_exact_name_returns() -> Result<(), Box> { + assert_respelling_refuses("darwin-respelled-format", "FORMAT", "format") +} + +#[test] +fn a_respelled_segment_pool_refuses_until_its_exact_name_returns() -> Result<(), Box> { + assert_respelling_refuses("darwin-respelled-pool", "segments", "Segments") +} diff --git a/tests/durable_locator.rs b/tests/durable_locator.rs index cadff7c3..1e4652dd 100644 --- a/tests/durable_locator.rs +++ b/tests/durable_locator.rs @@ -5,7 +5,7 @@ //! executables have separate descriptor tables even when Cargo runs in parallel. //! The locator parent owns no store; each admitted child runs one law serially. -#![cfg(target_os = "linux")] +#![cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_locator/suite.rs"] mod suite; diff --git a/tests/golden_file_worldline.rs b/tests/golden_file_worldline.rs index 6f374afe..4fb01dc4 100644 --- a/tests/golden_file_worldline.rs +++ b/tests/golden_file_worldline.rs @@ -5,6 +5,6 @@ pub(crate) mod support; #[path = "golden_file_worldline/suite.rs"] mod suite; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "layout_mutations/support.rs"] pub(crate) mod layout_mutation_support; diff --git a/tests/golden_file_worldline/durable_locator_laws.rs b/tests/golden_file_worldline/durable_locator_laws.rs index f435f326..a3c5b0a4 100644 --- a/tests/golden_file_worldline/durable_locator_laws.rs +++ b/tests/golden_file_worldline/durable_locator_laws.rs @@ -43,9 +43,7 @@ fn run_isolated(law: &str, operation: fn() -> TestResult) -> Result<(), Box TestResult) -> Result<(), Box Result> { + let mut command = Command::new("timeout"); + command.arg("20s").arg(std::env::current_exe()?); + Ok(command) +} + +#[cfg(target_os = "macos")] +fn watchdog_command() -> Result> { + let mut command = Command::new("/bin/sh"); + command + .args(["-c", "ulimit -t 20; exec \"$@\"", "locator-law"]) + .arg(std::env::current_exe()?); + Ok(command) +} + fn deleted_current_directory() -> Result<(), Box> { let sandbox = TestDirectory::create("durable-locator-deleted-cwd")?; let restore = WorkingDirectory(std::env::current_dir()?); diff --git a/tests/golden_file_worldline/suite.rs b/tests/golden_file_worldline/suite.rs index 7519b499..8451aef6 100644 --- a/tests/golden_file_worldline/suite.rs +++ b/tests/golden_file_worldline/suite.rs @@ -15,37 +15,37 @@ mod scenario_corpus; #[path = "storage_assertions.rs"] mod storage_assertions; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_assertions.rs"] mod durable_assertions; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_corruption_laws.rs"] mod durable_corruption_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_diagnostic_laws.rs"] mod durable_diagnostic_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_fixture.rs"] mod durable_fixture; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_layout_laws.rs"] mod durable_layout_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_namespace_laws.rs"] mod durable_namespace_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_output_laws.rs"] mod durable_output_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_range_properties.rs"] mod durable_range_properties; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_read_memory.rs"] mod durable_read_memory; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_refusal_laws.rs"] mod durable_refusal_laws; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "../segment_filesystem_stage/sandbox.rs"] mod durable_sandbox; @@ -251,10 +251,10 @@ fn reader_failures_preserve_precise_boundaries_and_sources() -> TestResult { Ok(()) } -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_writer_failures.rs"] mod durable_writer_failures; -#[cfg(target_os = "linux")] +#[cfg(any(target_os = "linux", target_os = "macos"))] #[path = "durable_closure_refusal.rs"] mod durable_closure_refusal; diff --git a/tests/migration_descriptor_exhaustion.rs b/tests/migration_descriptor_exhaustion.rs index fb1af637..c01de31e 100644 --- a/tests/migration_descriptor_exhaustion.rs +++ b/tests/migration_descriptor_exhaustion.rs @@ -1,6 +1,9 @@ //! This module owns migration admission diagnostics under descriptor exhaustion. -#![cfg(all(feature = "repository-tasks", target_os = "linux"))] +#![cfg(all( + feature = "repository-tasks", + any(target_os = "linux", target_os = "macos") +))] #[path = "segment_filesystem_stage/sandbox.rs"] pub mod sandbox; @@ -25,12 +28,10 @@ fn a_root_clone_failure_reports_the_namespace_boundary() -> Result<(), Box Result<(), Box Command { + let mut command = Command::new("timeout"); + command.args(["20s", "/bin/sh"]); + command +} + +#[cfg(target_os = "macos")] +fn watchdog_shell() -> Command { + Command::new("/bin/sh") +} + fn exhaust_descriptors() -> Result<(), Box> { let sandbox = sandbox::TestDirectory::create("migration-clone-descriptor-exhaustion")?; let mut storage = RepositoryInitializationStorage::admit_unchecked(sandbox.path())?; diff --git a/tests/observed_segment_stage.rs b/tests/observed_segment_stage.rs index 66f0eaef..0f4f2a86 100644 --- a/tests/observed_segment_stage.rs +++ b/tests/observed_segment_stage.rs @@ -1,7 +1,10 @@ //! Medium public filesystem laws for sealed observation; oracle: canonical bytes or typed refusal. //! Delete only when the observation capability is removed or stronger boundary laws replace these. -#![cfg(all(target_os = "linux", feature = "repository-tasks"))] +#![cfg(all( + any(target_os = "linux", target_os = "macos"), + feature = "repository-tasks" +))] #[path = "observed_segment_stage/byte_equivalence.rs"] mod byte_equivalence; diff --git a/tests/reader_fence_process.rs b/tests/reader_fence_process.rs index 2e85ca3b..f26e3688 100644 --- a/tests/reader_fence_process.rs +++ b/tests/reader_fence_process.rs @@ -1,5 +1,8 @@ //! Public snapshot fencing across actual reader process death and collection. -#![cfg(all(feature = "repository-tasks", target_os = "linux"))] +#![cfg(all( + feature = "repository-tasks", + any(target_os = "linux", target_os = "macos") +))] #[path = "reader_fence_process/fixture.rs"] pub mod fixture; diff --git a/tests/reader_fence_process/reader.rs b/tests/reader_fence_process/reader.rs index 60d1dd2b..8ce0c6e5 100644 --- a/tests/reader_fence_process/reader.rs +++ b/tests/reader_fence_process/reader.rs @@ -7,8 +7,9 @@ use keep::{ use std::error::Error; use std::fs::{self, File}; use std::io::{Read, Write}; +#[cfg(target_os = "linux")] +use std::os::unix::fs::MetadataExt; use std::os::unix::{ - fs::MetadataExt, net::{UnixListener, UnixStream}, process::ExitStatusExt, }; @@ -18,6 +19,8 @@ use std::time::{Duration, Instant}; const CHILD_ROOT: &str = "KEEP_FENCE_CHILD_ROOT"; const WATCHDOG: Duration = Duration::from_secs(20); +#[cfg(target_os = "macos")] +const ESCAPE_WINDOW: Duration = Duration::from_secs(2); pub(super) fn is_child() -> bool { std::env::var_os(CHILD_ROOT).is_some() @@ -89,6 +92,7 @@ impl Reader { Ok(()) } + #[cfg(target_os = "linux")] pub(super) fn await_kernel_wait(&mut self, fence: &File) -> Result<(), Box> { let pid = self.child.id().to_string(); let inode = format!(":{}", fence.metadata()?.ino()); @@ -116,6 +120,22 @@ impl Reader { } } + /// Darwin exposes no queued-lock table, so this cannot prove the child reached + /// its flock wait. It can only require that no snapshot escapes the fence for a + /// bounded window: a late child weakens the law but cannot falsely fail it. + #[cfg(target_os = "macos")] + pub(super) fn await_kernel_wait(&mut self, _fence: &File) -> Result<(), Box> { + self.channel.set_read_timeout(Some(ESCAPE_WINDOW))?; + let mut byte = [0]; + let observed = self.channel.read(&mut byte); + self.channel.set_read_timeout(Some(WATCHDOG))?; + match observed { + Ok(1) => Err("snapshot escaped the exclusive collector fence".into()), + Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => Ok(()), + result => Err(format!("reader failed during the fence window: {result:?}").into()), + } + } + pub(super) fn kill(&mut self) -> Result<(), Box> { self.child.kill()?; let status = self.child.wait()?; @@ -158,7 +178,11 @@ fn connect(listener: &UnixListener, child: &mut Child) -> Result return Ok(channel), + Ok((channel, _address)) => { + // BSD sockets inherit the listener's nonblocking mode on accept. + channel.set_nonblocking(false)?; + return Ok(channel); + } Err(error) if error.kind() == std::io::ErrorKind::WouldBlock => {} Err(error) => return Err(error.into()), } @@ -177,6 +201,7 @@ fn require_live(child: &mut Child, started: Instant) -> Result<(), Box bool { let mut fields = line.split_whitespace(); let _row = fields.next(); diff --git a/tests/recovery_inventory.rs b/tests/recovery_inventory.rs index e6aa6844..470d4196 100644 --- a/tests/recovery_inventory.rs +++ b/tests/recovery_inventory.rs @@ -4,19 +4,19 @@ pub mod inventory_double; #[path = "recovery_inventory/refusal_laws.rs"] mod refusal_laws; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] #[path = "segment_filesystem_stage/sandbox.rs"] pub mod sandbox; use std::error::Error; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] use std::fs; use inventory_double::{InventoryCall, InventoryDouble}; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] use keep::{FilesystemRecoveryInventoryReader, RecoveryInventoryError, RecoveryInventoryOperation}; use keep::{RecoveryEntryName, RecoveryInventoryLimit, RecoveryNamespace, read_recovery_inventory}; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] use sandbox::TestDirectory; #[test] @@ -61,7 +61,7 @@ fn every_count_precedes_name_retention_in_fixed_namespace_order() -> Result<(), Ok(()) } -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] #[test] fn unsupported_platform_refuses_before_filesystem_inventory_mutation() -> Result<(), Box> { diff --git a/tests/segment_filesystem_stage.rs b/tests/segment_filesystem_stage.rs index 8d00d182..a305a852 100644 --- a/tests/segment_filesystem_stage.rs +++ b/tests/segment_filesystem_stage.rs @@ -2,7 +2,7 @@ //! Oracle: specified exclusive creation and explicit sealing; independent v1 golden vectors. //! Retire only if this public stage contract is removed or stronger public evidence replaces it. -#![cfg(target_os = "linux")] +#![cfg(any(target_os = "linux", target_os = "macos"))] #[path = "segment_filesystem_stage/sandbox.rs"] pub mod sandbox; diff --git a/tests/store_initialization.rs b/tests/store_initialization.rs index e594a954..bb61b8fb 100644 --- a/tests/store_initialization.rs +++ b/tests/store_initialization.rs @@ -3,14 +3,14 @@ use std::error::Error; use std::io; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] use keep::FilesystemPlatformAdmission; use keep::{ StoreInitializationError, StoreInitializationPhase, StoreInitializationStorage, initialize_store, }; -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] #[path = "segment_filesystem_stage/sandbox.rs"] pub mod sandbox; @@ -58,7 +58,7 @@ fn initialization_stops_at_and_preserves_every_exact_failure_phase() -> Result<( Ok(()) } -#[cfg(not(target_os = "linux"))] +#[cfg(not(any(target_os = "linux", target_os = "macos")))] #[test] fn unsupported_production_platform_refuses_before_namespace_mutation() -> Result<(), Box> { From 342cbbb5f122dcf156dddc1315be5d258e042358 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 22:17:30 -0800 Subject: [PATCH 4/7] Repair macOS review findings Compare only restart-stable coordinates after a volume reattach, run tests on the APFS variant the runner selected, restore a host-independent child-refusal law, satisfy the crash-matrix workflow contract, and fix Linux-only test code. --- .github/workflows/ci.yml | 28 +++++- docs/adr/filesystem-platform-adapters.md | 9 +- docs/architecture/filesystem-platforms.md | 4 + .../filesystem-platform-adapters.md | 42 +++++++-- scripts/native/check-worker.sh | 3 + scripts/native/check.sh | 4 +- src/adapters/darwin_filesystem.rs | 6 +- src/adapters/darwin_filesystem_tests.rs | 23 ++++- src/adapters/filesystem_catalog_artifact.rs | 4 +- .../filesystem_mount_admission_tests.rs | 43 ++++++++- src/adapters/filesystem_root_identity.rs | 19 ++++ .../filesystem_version_two_admission.rs | 14 ++- .../filesystem_version_two_admission_tests.rs | 8 +- .../retention/reader_platform_law_tests.rs | 6 +- .../filesystem_migration_remount_tests.rs | 11 ++- tests/darwin_store_spelling.rs | 17 ++++ tests/darwin_volume_reattach.rs | 91 +++++++++++++++++++ tests/segment_filesystem_stage/sandbox.rs | 11 ++- 18 files changed, 303 insertions(+), 40 deletions(-) create mode 100644 tests/darwin_volume_reattach.rs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fa9151d4..5d123114 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -107,8 +107,17 @@ jobs: - name: Lint all features run: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings - - name: Run process-death crash matrix on default APFS - run: cargo xtask durability-crash-matrix + - name: Select default APFS for tests + run: | + mkdir "$RUNNER_TEMP/keep-default" + echo "KEEP_TEST_SCRATCH_ROOT=$RUNNER_TEMP/keep-default" >> "$GITHUB_ENV" + echo "KEEP_TEST_EXPECT_APFS_VARIANT=default" >> "$GITHUB_ENV" + + - name: Run process-death crash matrix on default APFS (debug) + run: cargo run --quiet --locked --package xtask -- durability-crash-matrix + + - name: Run process-death crash matrix on default APFS (optimized) + run: cargo run --quiet --profile release --locked --package xtask -- durability-crash-matrix - name: Test debug profile on default APFS run: cargo test --all-features --locked @@ -123,10 +132,21 @@ jobs: hdiutil attach -nobrowse -mountpoint "$RUNNER_TEMP/keep-cs" "$RUNNER_TEMP/keep-cs.sparseimage" - name: Test debug profile on case-sensitive APFS - run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo test --all-features --locked + run: KEEP_TEST_SCRATCH_ROOT="$RUNNER_TEMP/keep-cs" KEEP_TEST_EXPECT_APFS_VARIANT=case-sensitive cargo test --all-features --locked + + - name: Reopen a migrated store after reattaching its volume + run: | + hdiutil create -size 16m -type SPARSE -fs APFS -volname keep-occupier "$RUNNER_TEMP/keep-occupier.sparseimage" + mkdir "$RUNNER_TEMP/keep-occupier" + KEEP_TEST_SCRATCH_ROOT="$RUNNER_TEMP/keep-cs" \ + KEEP_TEST_REATTACH_IMAGE="$RUNNER_TEMP/keep-cs.sparseimage" \ + KEEP_TEST_REATTACH_MOUNT="$RUNNER_TEMP/keep-cs" \ + KEEP_TEST_REATTACH_OCCUPIER="$RUNNER_TEMP/keep-occupier.sparseimage" \ + KEEP_TEST_REATTACH_OCCUPIER_MOUNT="$RUNNER_TEMP/keep-occupier" \ + cargo test --all-features --locked --test darwin_volume_reattach - name: Run process-death crash matrix on case-sensitive APFS - run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo xtask durability-crash-matrix + run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo run --quiet --locked --package xtask -- durability-crash-matrix documentation: name: Documentation and workflow integrity diff --git a/docs/adr/filesystem-platform-adapters.md b/docs/adr/filesystem-platform-adapters.md index 4956002e..f748ed8c 100644 --- a/docs/adr/filesystem-platform-adapters.md +++ b/docs/adr/filesystem-platform-adapters.md @@ -80,8 +80,13 @@ durability protocol. Filesystem support stays explicit and fail-closed. Existing Linux wire bytes remain unchanged. Darwin recovery coordinates set the top device bit and store `st_dev` as both device and mount, because each mounted -APFS volume has its own `st_dev`; a record written on one OS can never match a -root on the other. Logical content identity does not incorporate native metadata. +APFS volume has its own `st_dev`. Darwin renumbers `st_dev` when a volume is +reattached, so its restart check compares only that top bit and the inode: a record +from another OS still refuses, but a store moved to another volume at the same inode +number is admitted. Logical content identity does not incorporate native metadata. + +Finder metadata such as `.DS_Store` in a store root refuses admission, like any +other entry the namespace cannot account for. On macOS, opening an existing entry costs one streaming scan of its parent directory. A rename that replaces a name during that scan could in principle hide diff --git a/docs/architecture/filesystem-platforms.md b/docs/architecture/filesystem-platforms.md index e7e0e760..7fb8488d 100644 --- a/docs/architecture/filesystem-platforms.md +++ b/docs/architecture/filesystem-platforms.md @@ -156,6 +156,8 @@ is independently observed mount identity. Symbolic links in the selected path remain a refusal. The documentation must explain that `/tmp` on macOS is a symlink and use the actual selected directory instead of silently resolving it. +Finder can write `.DS_Store` into a folder it displays. That entry refuses admission +until removed, so operators should not browse a store root in Finder. Case-insensitive APFS needs an explicit naming contract. Keep-generated names are canonical ASCII; their grammar must be injective under APFS name matching. The actual names observed during namespace admission must match the grammar @@ -194,6 +196,8 @@ The Darwin mapping sets the top bit of the device coordinate over the checked 32-bit `st_dev`, uses that value for both device and mount, and stores the inode unchanged. A negative `st_dev` refuses instead of being truncated. The top bit keeps Darwin coordinates apart from Linux ones without changing the wire grammar. +Darwin renumbers `st_dev` on reattach, so the restart check compares only that bit +and the inode. Live same-process checks still compare the full coordinate. Cross-OS movement of a physical store is not ordinary reopen. A store whose recovery evidence binds it to another physical root must refuse. Logical diff --git a/docs/testing-evidence/filesystem-platform-adapters.md b/docs/testing-evidence/filesystem-platform-adapters.md index e4b215ac..f969f428 100644 --- a/docs/testing-evidence/filesystem-platform-adapters.md +++ b/docs/testing-evidence/filesystem-platform-adapters.md @@ -19,16 +19,20 @@ Darwin 27.0.0, arm64, pinned `rustc 1.96.0`, under `scripts/native/check.sh`. | Check | Default APFS | Case-sensitive APFS | | --- | --- | --- | -| `cargo test --all-features` (keep package, debug) | 959 passed, 0 failed | 959 passed, 0 failed | +| `cargo test --all-features` (keep package, debug) | failed passed, 0 failed | failed passed, 0 failed | | `cargo xtask durability-crash-matrix` | exit 0 | exit 0 | The volumes are sparse images: default APFS `data` and case-sensitive APFS -`data-cs`, 256 MiB each, selected with `KEEP_NATIVE_DATA_IMAGE`. Release-profile -tests, `xtask` tests, and lint run in the `Native macOS storage` CI job. +`data-cs`, 256 MiB each, selected with `KEEP_NATIVE_DATA_IMAGE`. The worker points +`KEEP_TEST_SCRATCH_ROOT` at the selected image and sets +`KEEP_TEST_EXPECT_APFS_VARIANT`, and the spelling laws fail if the volume is the +other variant. Release-profile tests and the optimized crash matrix run in the +`Native macOS storage` CI job. `xtask` tests do not run on macOS. -An earlier set of native runs used Homebrew's Rust 1.98 ahead of rustup on `PATH` -and is not counted. The worker now selects the pinned toolchain and asserts its -version before running. +Two earlier sets of native runs are not counted. One used Homebrew's Rust 1.98 +ahead of rustup on `PATH`; the worker now selects the pinned toolchain and asserts +its version. Another claimed both variants, but the test sandbox ignored `TMPDIR`, +so both runs used the default-APFS build image. Independent review found this. ## Spelling oracle @@ -45,6 +49,18 @@ Calibration with the check disabled, on default APFS: - `darwin_store_spelling` laws **still pass**. Namespace admission independently refuses the foreign entry, so those laws do not witness the per-lookup check. +## Volume reattachment + +Darwin assigns `st_dev` per mount and reuses a freed number. Mounting another +volume in the gap moved a volume from 16777251 to 16777255, with its root inode +unchanged. Comparing the full device coordinate therefore refused a migrated store +for good, with `RootIdentityChanged`. `tests/darwin_volume_reattach.rs` reproduces +this: it fails when the full coordinate is compared and passes with the fix. + +The restart check on Darwin now compares only the coordinate domain and the inode. +This is weaker than on Linux: a migrated store moved to another volume at the same +inode number is admitted. Reboot itself has not been exercised. + ## Laws that remain Linux-only, and why - tmpfs refusal (`reader_platform_law_tests`, `a_writer_lock_cannot_mint_...`): @@ -63,6 +79,20 @@ Calibration with the check disabled, on default APFS: child can weaken it but cannot falsely fail it. - Isolated laws use `ulimit -t` where Linux uses wall-clock `timeout`. +## Known refusal: Finder metadata + +Finder can write `.DS_Store` into a folder it displays. The namespace admits no +entry it cannot account for, so the store then refuses until that file is removed. +`a_finder_metadata_file_in_the_root_refuses_until_it_is_removed` pins this. It +fails closed and loses nothing, but an operator browsing a store in Finder will +hit it, and whether to tolerate the name is an owner decision. + +## Crash matrix scope + +The matrix uses repository entry points that skip the production platform probe, +as it does on Linux. It exercises the protocols and the spelling boundary on APFS, +and separate public laws exercise production admission. + ## Not measured - The cost of the parent scan per existing-entry lookup. It is linear in the diff --git a/scripts/native/check-worker.sh b/scripts/native/check-worker.sh index 91661c89..32d1e307 100644 --- a/scripts/native/check-worker.sh +++ b/scripts/native/check-worker.sh @@ -7,6 +7,7 @@ readonly scratch_dir="$3" readonly log_path="$4" readonly sandbox_profile="$5" readonly check_scope="$6" +readonly data_variant="$7" export CARGO_HOME="$build_dir/cargo-home" export CARGO_TARGET_DIR="$build_dir/target" @@ -17,6 +18,8 @@ export CARGO_PROFILE_DEV_DEBUG=0 export CARGO_PROFILE_TEST_DEBUG=0 export TMPDIR="$scratch_dir/" export CARGO_TARGET_TMPDIR="$scratch_dir/test" +export KEEP_TEST_SCRATCH_ROOT="${scratch_dir%/}/test" +export KEEP_TEST_EXPECT_APFS_VARIANT="$data_variant" unset RUSTC_WRAPPER RUSTC_WORKSPACE_WRAPPER cd "$source_dir" diff --git a/scripts/native/check.sh b/scripts/native/check.sh index 4527f2dc..2b941a04 100644 --- a/scripts/native/check.sh +++ b/scripts/native/check.sh @@ -10,6 +10,8 @@ readonly build_dir="$native_root/build" # data is default case-insensitive APFS; data-cs is the case-sensitive variant. readonly data_image="${KEEP_NATIVE_DATA_IMAGE:-data}" case "$data_image" in data|data-cs) ;; *) exit 2 ;; esac +if [[ "$data_image" == data ]]; then data_variant=default; else data_variant=case-sensitive; fi +readonly data_variant readonly scratch_dir="$native_root/$data_image/tmp" readonly logs_dir="$native_root/logs" readonly profile="$native_root/compiler.sb" @@ -56,7 +58,7 @@ trap stop_child EXIT INT TERM /usr/bin/perl -MPOSIX=setsid -e \ 'setsid() >= 0 or die "setsid failed"; exec @ARGV or die "exec failed";' \ /bin/bash "$worker" "$source_dir" "$build_dir" "$scratch_dir" \ - "$log_path" "$profile" "$check_scope" & + "$log_path" "$profile" "$check_scope" "$data_variant" & task_pid=$! readonly task_started=$SECONDS failure='' diff --git a/src/adapters/darwin_filesystem.rs b/src/adapters/darwin_filesystem.rs index 78eb855c..36ea1f37 100644 --- a/src/adapters/darwin_filesystem.rs +++ b/src/adapters/darwin_filesystem.rs @@ -10,13 +10,9 @@ use cap_fs_ext::MetadataExt; use cap_std::fs::Dir; use rustix::fs::{CWD, Mode, OFlags, fstatfs, openat}; -use super::filesystem_root_identity::FilesystemRootIdentity; +use super::filesystem_root_identity::{DARWIN_COORDINATE_DOMAIN, FilesystemRootIdentity}; use super::platform_filesystem::PlatformFilesystem; -/// Separates Darwin device coordinates from Linux ones in persisted records, so -/// a record written on one OS can never match a live root on the other. -const DARWIN_COORDINATE_DOMAIN: u64 = 0x8000_0000_0000_0000; - pub(super) struct DarwinFilesystem; impl PlatformFilesystem for DarwinFilesystem { diff --git a/src/adapters/darwin_filesystem_tests.rs b/src/adapters/darwin_filesystem_tests.rs index 5aedae94..3d6cb628 100644 --- a/src/adapters/darwin_filesystem_tests.rs +++ b/src/adapters/darwin_filesystem_tests.rs @@ -5,9 +5,10 @@ use std::{error::Error, fs, io, path::Path, thread}; use cap_fs_ext::MetadataExt; use cap_std::{ambient_authority, fs::OpenOptions}; -use super::{DARWIN_COORDINATE_DOMAIN, DarwinFilesystem}; +use super::DarwinFilesystem; use crate::adapters::filesystem_directory::FilesystemDirectory; use crate::adapters::filesystem_entry_spelling::EntrySpellingMismatch; +use crate::adapters::filesystem_root_identity::{DARWIN_COORDINATE_DOMAIN, restart_stable_device}; use crate::adapters::filesystem_test_sandbox::TestDirectory; use crate::adapters::platform_filesystem::PlatformFilesystem; @@ -21,11 +22,17 @@ fn read_options() -> OpenOptions { options } -/// Observes this run's volume, so one law states both APFS naming variants. +/// Observes this run's volume, so one law states both APFS naming variants. A +/// runner that names the variant it selected fails here if it got the other one. fn folds_case(root: &Path) -> io::Result { fs::write(root.join("variant-probe"), [])?; let folds = root.join("VARIANT-PROBE").try_exists()?; fs::remove_file(root.join("variant-probe"))?; + match std::env::var("KEEP_TEST_EXPECT_APFS_VARIANT").as_deref() { + Ok("default") => assert!(folds, "expected default APFS"), + Ok("case-sensitive") => assert!(!folds, "expected case-sensitive APFS"), + _ => {} + } Ok(folds) } @@ -187,3 +194,15 @@ fn read_only_apfs_volumes_are_refused() -> Result<(), Box> { assert!(matches!(refusal, Err(source) if source.kind() == io::ErrorKind::Unsupported)); Ok(()) } + +#[test] +fn a_renumbered_volume_keeps_its_restart_coordinate_but_another_os_does_not() { + let first = DARWIN_COORDINATE_DOMAIN | 0x0100_0023; + let reattached = DARWIN_COORDINATE_DOMAIN | 0x0100_0027; + let linux = 0x0100_0023; + assert_eq!( + restart_stable_device(first), + restart_stable_device(reattached) + ); + assert_ne!(restart_stable_device(first), restart_stable_device(linux)); +} diff --git a/src/adapters/filesystem_catalog_artifact.rs b/src/adapters/filesystem_catalog_artifact.rs index 8339d3a9..05fab66d 100644 --- a/src/adapters/filesystem_catalog_artifact.rs +++ b/src/adapters/filesystem_catalog_artifact.rs @@ -146,7 +146,9 @@ fn require_exact_bytes( mod tests { use std::error::Error; - use cap_std::{ambient_authority, fs::Dir}; + use cap_std::ambient_authority; + + use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; use super::super::filesystem_test_sandbox::TestDirectory; diff --git a/src/adapters/filesystem_mount_admission_tests.rs b/src/adapters/filesystem_mount_admission_tests.rs index 6a609ce5..ca9f6226 100644 --- a/src/adapters/filesystem_mount_admission_tests.rs +++ b/src/adapters/filesystem_mount_admission_tests.rs @@ -3,11 +3,15 @@ //! The oracle is the platform contract's single-device/single-mount rule; //! child inode identity is deliberately not required to equal the root inode. -use std::{error::Error, io}; +use std::{error::Error, fs, io, path::Path}; -use super::admit_child_identity; +use cap_std::fs::Dir; + +use super::{admit_child_identity, open_profile}; use crate::adapters::filesystem_mount_refusal::FilesystemMountRefusal; use crate::adapters::filesystem_root_identity::FilesystemRootIdentity; +use crate::adapters::filesystem_test_sandbox::TestDirectory; +use crate::adapters::platform_filesystem::PlatformFilesystem; #[test] fn a_distinct_child_inode_on_the_same_mount_is_admitted() -> Result<(), Box> { @@ -49,3 +53,38 @@ fn assert_refusal(result: io::Result<()>, expected: FilesystemMountRefusal) { && source.get_ref().and_then(|error| error.downcast_ref::()) == Some(&expected))); } + +/// Admits the root and refuses the first child, as a casefolded, read-only, or +/// foreign child would be refused by a native implementation. +struct RefusesChildren(std::cell::Cell); + +impl PlatformFilesystem for RefusesChildren { + fn open_root(&self, path: &Path) -> io::Result { + Dir::open_ambient_dir(path, cap_std::ambient_authority()) + } + + fn admit_directory(&self, directory: &Dir) -> io::Result { + if self.0.replace(true) { + return Err(io::Error::from(io::ErrorKind::Unsupported)); + } + self.root_identity(directory) + } + + fn root_identity(&self, _directory: &Dir) -> io::Result { + Ok(FilesystemRootIdentity::new(8, 41, 73)) + } + + fn repository_identity(&self, directory: &Dir) -> io::Result { + self.root_identity(directory) + } +} + +#[test] +fn every_present_protocol_child_must_satisfy_the_platform_profile() -> Result<(), Box> { + let sandbox = TestDirectory::create("platform-child-refusal")?; + fs::create_dir(sandbox.path().join("segments"))?; + let platform = RefusesChildren(std::cell::Cell::new(false)); + let refusal = open_profile(&platform, sandbox.path(), &["segments"]); + assert!(matches!(refusal, Err(source) if source.kind() == io::ErrorKind::Unsupported)); + Ok(()) +} diff --git a/src/adapters/filesystem_root_identity.rs b/src/adapters/filesystem_root_identity.rs index 4d2dcb09..57de658a 100644 --- a/src/adapters/filesystem_root_identity.rs +++ b/src/adapters/filesystem_root_identity.rs @@ -33,3 +33,22 @@ impl FilesystemRootIdentity { self.file } } + +/// Separates Darwin device coordinates from Linux ones in persisted records. +#[cfg(target_os = "macos")] +pub(super) const DARWIN_COORDINATE_DOMAIN: u64 = 0x8000_0000_0000_0000; + +/// Returns the part of a device coordinate that survives a remount or reboot. +/// +/// Darwin assigns `st_dev` per mount, so reattaching a volume renumbers it and only +/// the coordinate domain remains comparable across a restart. +#[cfg(target_os = "macos")] +pub(super) const fn restart_stable_device(device: u64) -> u64 { + device & DARWIN_COORDINATE_DOMAIN +} + +/// Returns the part of a device coordinate that survives a remount or reboot. +#[cfg(not(target_os = "macos"))] +pub(super) const fn restart_stable_device(device: u64) -> u64 { + device +} diff --git a/src/adapters/filesystem_version_two_admission.rs b/src/adapters/filesystem_version_two_admission.rs index 41ebbda9..fc80c52c 100644 --- a/src/adapters/filesystem_version_two_admission.rs +++ b/src/adapters/filesystem_version_two_admission.rs @@ -6,7 +6,7 @@ use crate::adapters::filesystem_directory::FilesystemDirectory as Dir; #[cfg(any(test, feature = "repository-tasks"))] use cap_std::ambient_authority; -use super::filesystem_root_identity::FilesystemRootIdentity; +use super::filesystem_root_identity::{FilesystemRootIdentity, restart_stable_device}; pub(super) use super::filesystem_version_two_records::BoundRootIdentity; use super::{ FilesystemPlatformAdmissionError, FilesystemWriterLock, StoreRootIdentityCoordinate, @@ -111,8 +111,8 @@ fn pin(parent: &Dir, name: &str) -> Result Result<(), FilesystemPlatformAdmissionError> { - for (coordinate, expected, actual) in [ + let device_is_stable = + restart_stable_device(bound.device()) == restart_stable_device(observed.device()); + for (coordinate, expected, actual, stable) in [ ( StoreRootIdentityCoordinate::Device, bound.device(), observed.device(), + device_is_stable, ), ( StoreRootIdentityCoordinate::File, bound.file(), observed.file(), + bound.file() == observed.file(), ), ] { - if expected != actual { + if !stable { return Err(FilesystemPlatformAdmissionError::RootIdentityChanged { coordinate, expected, diff --git a/src/adapters/retention/filesystem_version_two_admission_tests.rs b/src/adapters/retention/filesystem_version_two_admission_tests.rs index 6fa956bd..053b7a63 100644 --- a/src/adapters/retention/filesystem_version_two_admission_tests.rs +++ b/src/adapters/retention/filesystem_version_two_admission_tests.rs @@ -147,13 +147,15 @@ fn reopen_admits_a_remounted_root_and_refuses_a_moved_one() { observed: 4, }) )); + // Only the top bit of a device coordinate survives a restart on Darwin. + let foreign_device = 1 | (1 << 63); assert!(matches!( - require_root_identity(bound, FilesystemRootIdentity::new(9, 2, 3)), + require_root_identity(bound, FilesystemRootIdentity::new(foreign_device, 2, 3)), Err(FilesystemPlatformAdmissionError::RootIdentityChanged { coordinate: StoreRootIdentityCoordinate::Device, expected: 1, - observed: 9, - }) + observed, + }) if observed == foreign_device )); } diff --git a/src/adapters/retention/reader_platform_law_tests.rs b/src/adapters/retention/reader_platform_law_tests.rs index 3e4146b5..23be86fd 100644 --- a/src/adapters/retention/reader_platform_law_tests.rs +++ b/src/adapters/retention/reader_platform_law_tests.rs @@ -10,13 +10,11 @@ use std::fs; use std::io; use std::path::PathBuf; -use super::filesystem_retention_test_fixture::{ - CATALOG_NAME, SEGMENT_NAME, fixture, migrated_store, -}; +use super::filesystem_retention_test_fixture::{CATALOG_NAME, SEGMENT_NAME, fixture}; use crate::{ CatalogRestartByteLimit, CatalogRestartPolicy, DurableSnapshot, DurableStoreError, FilesystemPlatformAdmission, FilesystemRetentionSnapshot, FilesystemRetentionSnapshotError, - FilesystemStoreMigrationAuthority, FilesystemWriterLock, ReaderAttemptLimit, SegmentReadPolicy, + FilesystemStoreMigrationAuthority, ReaderAttemptLimit, SegmentReadPolicy, execute_store_migration, }; diff --git a/src/adapters/store_migration/filesystem_migration_remount_tests.rs b/src/adapters/store_migration/filesystem_migration_remount_tests.rs index 3a9a3860..89f6582e 100644 --- a/src/adapters/store_migration/filesystem_migration_remount_tests.rs +++ b/src/adapters/store_migration/filesystem_migration_remount_tests.rs @@ -76,9 +76,14 @@ fn substitute_coordinate( StoreRootIdentityCoordinate::Mount => original.root_mount_identity().get(), StoreRootIdentityCoordinate::File => original.root_file_identity().get(), }; - let expected = observed - .checked_add(1) - .ok_or("fixture coordinate overflow")?; + // Only the top bit of a device coordinate survives a restart on Darwin. + let expected = if coordinate == StoreRootIdentityCoordinate::Device { + observed ^ (1 << 63) + } else { + observed + .checked_add(1) + .ok_or("fixture coordinate overflow")? + }; let changed = |selected, value| { if coordinate == selected { expected diff --git a/tests/darwin_store_spelling.rs b/tests/darwin_store_spelling.rs index 2e938430..a7f87830 100644 --- a/tests/darwin_store_spelling.rs +++ b/tests/darwin_store_spelling.rs @@ -61,3 +61,20 @@ fn a_respelled_format_marker_refuses_until_its_exact_name_returns() -> Result<() fn a_respelled_segment_pool_refuses_until_its_exact_name_returns() -> Result<(), Box> { assert_respelling_refuses("darwin-respelled-pool", "segments", "Segments") } + +// Finder can write this into any folder it displays. Refusal is deliberate until +// an owner decides otherwise: the namespace admits no entry it cannot account for. +#[test] +fn a_finder_metadata_file_in_the_root_refuses_until_it_is_removed() -> Result<(), Box> { + let (store, writer) = fixture::migrated("darwin-finder-metadata")?; + drop(writer); + let metadata = store.path().join(".DS_Store"); + fs::write(&metadata, [])?; + assert!( + load(store.path()).is_err(), + "a foreign root entry was admitted" + ); + fs::remove_file(&metadata)?; + drop(load(store.path())?); + Ok(()) +} diff --git a/tests/darwin_volume_reattach.rs b/tests/darwin_volume_reattach.rs new file mode 100644 index 00000000..8b2fcb8f --- /dev/null +++ b/tests/darwin_volume_reattach.rs @@ -0,0 +1,91 @@ +//! A migrated store survives its volume being detached and reattached. +//! +//! Size: large. Oracle: Darwin assigns `st_dev` per mount and reuses a freed number, +//! so the law mounts a second volume in the gap to force a new number, while the +//! root inode persists. It needs runner-owned disk images: one holding the scratch +//! root, named by `KEEP_TEST_REATTACH_IMAGE` and `KEEP_TEST_REATTACH_MOUNT`, and one +//! named by `KEEP_TEST_REATTACH_OCCUPIER` and `KEEP_TEST_REATTACH_OCCUPIER_MOUNT`. +//! It is inert otherwise. Delete when the persisted root binding is removed. + +#![cfg(all(feature = "repository-tasks", target_os = "macos"))] + +#[allow(dead_code, reason = "this law uses only the migrated-store fixture")] +#[path = "reader_fence_process/fixture.rs"] +pub mod fixture; +#[allow(dead_code, reason = "this law uses Drop cleanup")] +#[path = "segment_filesystem_stage/sandbox.rs"] +pub mod sandbox; + +use std::{ + env, error::Error, ffi::OsStr, fs, os::unix::fs::MetadataExt, path::Path, process::Command, +}; + +use keep::{ + CatalogRestartByteLimit, CatalogRestartPolicy, FilesystemRetentionSnapshot, ReaderAttemptLimit, + SegmentReadPolicy, +}; + +fn hdiutil(arguments: &[&OsStr]) -> Result<(), Box> { + if Command::new("/usr/bin/hdiutil") + .args(arguments) + .status()? + .success() + { + return Ok(()); + } + Err("hdiutil failed".into()) +} + +fn attach(image: &OsStr, mount: &OsStr) -> Result<(), Box> { + hdiutil(&[ + "attach".as_ref(), + "-nobrowse".as_ref(), + "-mountpoint".as_ref(), + mount, + image, + ]) +} + +fn load(root: &Path) -> Result<(), Box> { + let policy = CatalogRestartPolicy::new( + SegmentReadPolicy::MAXIMUM, + CatalogRestartByteLimit::new(1_048_576)?, + ); + drop(FilesystemRetentionSnapshot::load( + root, + policy, + ReaderAttemptLimit::DEFAULT, + )?); + Ok(()) +} + +#[test] +fn a_reattached_volume_reopens_its_migrated_store() -> Result<(), Box> { + let (Some(image), Some(mount), Some(occupier), Some(occupier_mount)) = ( + env::var_os("KEEP_TEST_REATTACH_IMAGE"), + env::var_os("KEEP_TEST_REATTACH_MOUNT"), + env::var_os("KEEP_TEST_REATTACH_OCCUPIER"), + env::var_os("KEEP_TEST_REATTACH_OCCUPIER_MOUNT"), + ) else { + return Ok(()); + }; + let (store, writer) = fixture::migrated("darwin-reattach")?; + drop(writer); + load(store.path())?; + let before = fs::metadata(store.path())?; + + hdiutil(&["detach".as_ref(), mount.as_ref()])?; + attach(&occupier, &occupier_mount)?; + attach(&image, &mount)?; + + let after = fs::metadata(store.path())?; + assert_ne!( + before.dev(), + after.dev(), + "the volume kept its device number" + ); + assert_eq!(before.ino(), after.ino()); + let reopened = load(store.path()); + hdiutil(&["detach".as_ref(), occupier_mount.as_ref()])?; + reopened +} diff --git a/tests/segment_filesystem_stage/sandbox.rs b/tests/segment_filesystem_stage/sandbox.rs index 126b5331..8dcb56c3 100644 --- a/tests/segment_filesystem_stage/sandbox.rs +++ b/tests/segment_filesystem_stage/sandbox.rs @@ -17,8 +17,15 @@ impl TestDirectory { /// Returns the exact filesystem failure from scratch-root creation, /// removal of same-process stale test evidence, or sandbox creation. pub(super) fn create(name: &str) -> io::Result { - let root = option_env!("CARGO_TARGET_TMPDIR").map_or_else( - || Path::new(env!("CARGO_MANIFEST_DIR")).join("target/tmp"), + // A runner selects a specific filesystem here; Cargo's scratch sits with the + // build output, on whatever volume holds that. + let root = std::env::var_os("KEEP_TEST_SCRATCH_ROOT").map_or_else( + || { + option_env!("CARGO_TARGET_TMPDIR").map_or_else( + || Path::new(env!("CARGO_MANIFEST_DIR")).join("target/tmp"), + PathBuf::from, + ) + }, PathBuf::from, ); fs::create_dir_all(&root)?; From b7f8636bc43340bd88b1772da5667898d761cf35 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 22:17:42 -0800 Subject: [PATCH 5/7] Docs: correct the macOS evidence counts --- docs/testing-evidence/filesystem-platform-adapters.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/testing-evidence/filesystem-platform-adapters.md b/docs/testing-evidence/filesystem-platform-adapters.md index f969f428..d36ca09e 100644 --- a/docs/testing-evidence/filesystem-platform-adapters.md +++ b/docs/testing-evidence/filesystem-platform-adapters.md @@ -19,7 +19,7 @@ Darwin 27.0.0, arm64, pinned `rustc 1.96.0`, under `scripts/native/check.sh`. | Check | Default APFS | Case-sensitive APFS | | --- | --- | --- | -| `cargo test --all-features` (keep package, debug) | failed passed, 0 failed | failed passed, 0 failed | +| `cargo test --all-features` (keep package, debug) | 963 passed, 0 failed | 963 passed, 0 failed | | `cargo xtask durability-crash-matrix` | exit 0 | exit 0 | The volumes are sparse images: default APFS `data` and case-sensitive APFS @@ -103,9 +103,9 @@ and separate public laws exercise production admission. ## Runner bounds -Compilation and test output stays on the reusable 1.5 GiB `build` image, 349 MiB -used after a clean run. Each data image stayed under 4 MiB used. Logs total -1.1 MiB. The sandbox denies network except local Unix sockets, which the process +Compilation and test output stays on the reusable 1.5 GiB `build` image, 353 MiB +used after a clean run. Each data image stayed under 7 MiB used, and logs total +1.9 MiB. The sandbox denies network except local Unix sockets, which the process laws need, and denies writes outside the build and selected data images. The runner watches only its own process group, so a law that starts its own group From 8240be7c1dfbfdd0d8177ad7ca201a857b3f4c7c Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 22:39:16 -0800 Subject: [PATCH 6/7] Allow the infallible spelling stub to mirror its Darwin signature --- src/adapters/filesystem_entry_spelling.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/adapters/filesystem_entry_spelling.rs b/src/adapters/filesystem_entry_spelling.rs index a5262dd6..72816a19 100644 --- a/src/adapters/filesystem_entry_spelling.rs +++ b/src/adapters/filesystem_entry_spelling.rs @@ -37,6 +37,11 @@ pub(super) fn require_exact(directory: &Dir, name: &OsStr) -> io::Result<()> { // Production Linux admission excludes casefold directories. Other platforms // have no production admission; their repository fixtures retain old behavior. #[cfg(not(target_os = "macos"))] +#[allow( + clippy::missing_const_for_fn, + clippy::unnecessary_wraps, + reason = "this mirrors the fallible Darwin signature its callers share" +)] pub(super) fn require_exact(_directory: &Dir, _name: &OsStr) -> io::Result<()> { Ok(()) } From 17fcbbed3cb012c0a7bb5c92751bdee5543dc157 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 3 Nov 2026 22:59:21 -0800 Subject: [PATCH 7/7] Docs: record the macOS parent-scan cost --- .../filesystem-platform-adapters.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/docs/testing-evidence/filesystem-platform-adapters.md b/docs/testing-evidence/filesystem-platform-adapters.md index d36ca09e..b7327e2d 100644 --- a/docs/testing-evidence/filesystem-platform-adapters.md +++ b/docs/testing-evidence/filesystem-platform-adapters.md @@ -93,10 +93,22 @@ The matrix uses repository entry points that skip the production platform probe, as it does on Linux. It exercises the protocols and the spelling boundary on APFS, and separate public laws exercise production admission. +## Parent scan cost + +One release-mode run on the default APFS image, 200 opens of one existing file, using +an unrepeated throwaway test. The cost grows with the directory, as expected. + +| Directory entries | Plain open | With spelling check | +| --- | --- | --- | +| 10 | 25 µs | 74 µs | +| 1,000 | 9 µs | 282 µs | +| 10,000 | 11 µs | 475 µs | +| 50,000 | 17 µs | 3.4 ms | + +This is the per-open overhead only, not a store workload. + ## Not measured -- The cost of the parent scan per existing-entry lookup. It is linear in the - directory size and matters most for a large segment pool. - `F_FULLFSYNC` durability. A successful call is not proof that hardware honored it. - A real non-APFS or network volume at the public boundary.