Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,67 @@ jobs:
cargo check --manifest-path fuzz/Cargo.toml --bins --locked
cargo clippy --manifest-path fuzz/Cargo.toml --bins --locked -- -D warnings

macos:
name: Native macOS storage
runs-on: macos-15
timeout-minutes: 90
steps:
- name: Check out repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Install pinned toolchain
run: rustup show

# The default macOS TMPDIR traverses the /var symlink, which admission refuses.
- name: Select an admitted scratch directory
run: echo "TMPDIR=$RUNNER_TEMP/" >> "$GITHUB_ENV"

- name: Lint all features
run: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings

- name: Select default APFS for tests
run: |
mkdir "$RUNNER_TEMP/keep-default"
echo "KEEP_TEST_SCRATCH_ROOT=$RUNNER_TEMP/keep-default" >> "$GITHUB_ENV"
echo "KEEP_TEST_EXPECT_APFS_VARIANT=default" >> "$GITHUB_ENV"

- name: Run process-death crash matrix on default APFS (debug)
run: cargo run --quiet --locked --package xtask -- durability-crash-matrix

- name: Run process-death crash matrix on default APFS (optimized)
run: cargo run --quiet --profile release --locked --package xtask -- durability-crash-matrix

- name: Test debug profile on default APFS
run: cargo test --all-features --locked

- name: Test release profile on default APFS
run: cargo test --all-features --release --locked

- name: Mount case-sensitive APFS
run: |
hdiutil create -size 1g -type SPARSE -fs "Case-sensitive APFS" -volname keep-cs "$RUNNER_TEMP/keep-cs.sparseimage"
mkdir "$RUNNER_TEMP/keep-cs"
hdiutil attach -nobrowse -mountpoint "$RUNNER_TEMP/keep-cs" "$RUNNER_TEMP/keep-cs.sparseimage"

- name: Test debug profile on case-sensitive APFS
run: KEEP_TEST_SCRATCH_ROOT="$RUNNER_TEMP/keep-cs" KEEP_TEST_EXPECT_APFS_VARIANT=case-sensitive cargo test --all-features --locked

- name: Reopen a migrated store after reattaching its volume
run: |
hdiutil create -size 16m -type SPARSE -fs APFS -volname keep-occupier "$RUNNER_TEMP/keep-occupier.sparseimage"
mkdir "$RUNNER_TEMP/keep-occupier"
KEEP_TEST_SCRATCH_ROOT="$RUNNER_TEMP/keep-cs" \
KEEP_TEST_REATTACH_IMAGE="$RUNNER_TEMP/keep-cs.sparseimage" \
KEEP_TEST_REATTACH_MOUNT="$RUNNER_TEMP/keep-cs" \
KEEP_TEST_REATTACH_OCCUPIER="$RUNNER_TEMP/keep-occupier.sparseimage" \
KEEP_TEST_REATTACH_OCCUPIER_MOUNT="$RUNNER_TEMP/keep-occupier" \
cargo test --all-features --locked --test darwin_volume_reattach

- name: Run process-death crash matrix on case-sensitive APFS
run: TMPDIR="$RUNNER_TEMP/keep-cs/" cargo run --quiet --locked --package xtask -- durability-crash-matrix

documentation:
name: Documentation and workflow integrity
runs-on: ubuntu-latest
Expand Down
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ after its public API and format compatibility policies are established.

## [Unreleased]

- Admit macOS APFS as a production storage platform, on both default and case-sensitive volumes. Keep requires every existing protocol name byte-for-byte in a listing of its parent, so `head` never answers for `HEAD`. Adds a native macOS CI job; no format, dependency, or unsafe change. The crash matrix passes natively, but no power-loss experiment has been run on APFS.

- Update the development-only Markdown validation graph and exact admission policy to markdownlint-cli2 0.24.0. On the 2026-11-03 audit this removes the markdown-it and smol-toml reports and adds the high-severity re2js advisory GHSA-gg6h-894w-4m9c through markdownlint's exact pin. This is a currency update, not a demonstrated security gain: markdown-it was not loaded on Keep's path, re2js now is, and the admission records why its affected path appears unreachable. MD011 also reports slightly less, and #209 tracks removing re2js (#188).

- Update Rustix to 1.1.6, which fixes the high-severity GHSA-jh2m-fhpj-3vr7 extended-attribute undefined behavior with no Keep exposure found, and cap-fs-ext to 4.0.3 in both locked workspaces. Renew their dependency admission and pin the `RDONLY` mount-flag value in a law; selected features and the Rust floor are unchanged (#207, #189).
Expand Down
9 changes: 8 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ Keep is required to refuse all three, before mutating anything.
complete framing, checksum, and identity verification. Immutable segments,
generation-versioned catalogs, and a fixed-width `HEAD` are published
through an ordered protocol whose every step is a named crash point.
Platform admission is Linux ext4, non-casefolded, one writer.
Platform admission is non-casefolded Linux ext4 or local writable macOS APFS,
one writer.
- **Process-death recovery evidence.** The crash matrix kills real writer processes
at 156 before/during/after coordinates (`KEEP-CRASH-001`–`052`) and
verifies the store lands in exactly one documented lawful state each time,
Expand Down Expand Up @@ -170,6 +171,12 @@ The core holds no clock, no caller identity, no paths, and no application
policy. Retention proves a *physical reconstruction* claim only — never what
the content means, who owns it, or whether deleting it is legally safe.

The [filesystem platform design](docs/architecture/filesystem-platforms.md)
defines the shared Linux/macOS adapter boundary. Both APFS naming variants are
admitted; the [evidence record](docs/testing-evidence/filesystem-platform-adapters.md)
states what has been exercised, and process-death evidence is not power-loss
evidence.

## Try it

Keep is `0.0.0` and unpublished; build from source. The in-memory
Expand Down
15 changes: 8 additions & 7 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ names; use those in code, tests, and commits.

- [ ] [F-39 Compression representation codec](#f-39-compression-representation-codec) — Proposed
- [ ] [F-40 Additional chunking profiles and hierarchical layouts](#f-40-additional-chunking-profiles-and-hierarchical-layouts) — Proposed
- [ ] [F-41 Platform adapters beyond Linux ext4](#f-41-platform-adapters-beyond-linux-ext4) — Proposed
- [ ] [F-41 Platform adapters beyond Linux ext4](#f-41-platform-adapters-beyond-linux-ext4) — In progress; macOS APFS done
- [ ] [F-42 Operator surfaces](#f-42-operator-surfaces) — Proposed (CLI and MCP adapters)
- [ ] [F-43 Public release and API stability](#f-43-public-release-and-api-stability) — Proposed
- [ ] [F-44 Multi-writer, replication, and remote tiers](#f-44-multi-writer-replication-and-remote-tiers) — Out of scope
Expand Down Expand Up @@ -582,7 +582,7 @@ in place. One writer holds kernel advisory locks on the store root and
**Status:** Done (issue #17, M3).

Production initialization admits only a writable, non-casefolded Linux
ext4 root on a single local host. Opening is observational; recovery is
ext4 root or a writable local macOS APFS root, on a single local host. Opening is observational; recovery is
explicit and planned against storage ports: inventory, name classification,
stage fingerprint and assessment, then discard, completion, next-head
finalization, or segment resume. `cargo xtask durability-crash-matrix`
Expand Down Expand Up @@ -2396,11 +2396,11 @@ plan or needs bounded plan streaming.

### F-41 Platform adapters beyond Linux ext4

**Status:** Proposed. `recovery.md` defers Windows "until an adapter and
crash harness prove equivalent semantics"; macOS is not admitted for
production; the crash matrix does not simulate power loss.
**Status:** In progress. `recovery.md` defers Windows "until an adapter and
crash harness prove equivalent semantics"; macOS APFS is admitted (T-41.1);
the crash matrix does not simulate power loss.

- [ ] T-41.1 macOS APFS production admission.
- [x] T-41.1 macOS APFS production admission.
- **Requirements:** a platform profile that probes APFS for the same
capabilities (atomic no-clobber link, atomic same-filesystem
replacement, durable `fsync` semantics including `F_FULLFSYNC`,
Expand All @@ -2409,7 +2409,8 @@ production; the crash matrix does not simulate power loss.
- **Acceptance criteria:** `KEEP-RECOVERY-003` gains a macOS row; the
105-case matrix (and the v2 matrices) green on a macOS runner.
- **Scope:** in — `filesystem_platform_profile.rs` sibling. Out —
case-insensitive volumes (refuse), network volumes.
network volumes. Case-insensitive volumes are in, with exact-spelling
checks (see the platform adapter decision).
- **User stories:** Human — a developer on a Mac runs a durable store
locally. API user — same API. MCP user — same. Agent — same.
- **Interface:** none new.
Expand Down
1 change: 1 addition & 0 deletions docs/adr/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,4 @@ encryption, concurrency, or public-API surface it governs.
- [ADR-0007: Terminal signal process-group guard](0007-terminal-signal-process-group-guard.md)
- [ADR-0008: Deadline-bounded reader retirement](0008-deadline-bounded-reader-retirement.md)
- [ADR-0009: Retention roots, release, and GC liveness](0009-retention-roots-release-and-gc-liveness.md)
- [Filesystem platform adapters](filesystem-platform-adapters.md)
98 changes: 98 additions & 0 deletions docs/adr/filesystem-platform-adapters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# Filesystem Platform Adapters

- Status: Accepted; implemented for Linux ext4 and macOS APFS
- Date: 2026-11-02
- Owners: Keep storage implementation and repository owner
- Related issue: none; directly requested platform design and implementation

## Context

Keep's production admission accepts only Linux ext4. Several focused storage
ports already separate protocol ordering from filesystem effects, but their
implementations share concrete Linux assumptions without one native mechanism
boundary. Adding macOS separately to initialization, publication, retention,
migration, and recovery would duplicate authority and durability decisions.

The design baseline is main `3887f2a`, including its existing distinction
between live mount identity and restart device/inode coordinates.

## Decision

Retain the focused semantic storage ports. Implement their filesystem effects
through common Unix protocol adapters over one pinned store authority, backed
by a `PlatformFilesystem` interface. Native implementations supply observations
and OS mechanisms; shared admission policy establishes which profiles are
supported and refuses incomplete evidence.

The [architecture design](../architecture/filesystem-platforms.md) defines
the operations, ownership, Linux profile extension rules, APFS naming and
durability obligations, and acceptance gates. Backend selection happens once.
It must not alter logical identities or duplicate protocol state machines.

Native macOS support admits writable local APFS in both naming variants, with
no format difference between them. Case-insensitive volumes keep exact namespace
spelling: a name the lookup resolves must also appear byte-for-byte in a listing
of its parent, or the operation refuses before it reads, reuses, or mutates the
entry. No unsafe code or Darwin-specific dependency is needed. Other Linux filesystems
are distinct admitted profiles with their own evidence, rather than being
accepted because they provide familiar POSIX operations. Windows and network
filesystems are excluded from this implementation.

Keep existing public admission constructors and protocol entry points during
the refactor. A future convenience facade may construct typed protocol views
from retained authority, but a general object factory is not necessary for OS
selection and must never create independent competing writer authorities.

## Alternatives considered

- **A single broad `StorageInterface`:** conflates initialization, reads,
mutation, and recovery authority; implementations must expose operations
that are invalid in their current state. Keep narrower capability ports.
- **A factory returning unrelated storage objects:** chooses implementations
without sharing their pinned root, observations, or lock lifetime. A
factory can compose the design but cannot replace the platform boundary.
- **A full generic virtual filesystem:** duplicates existing capability file
APIs and spreads type parameters through every protocol. Share Unix file
operations and abstract only the native mechanisms that actually differ.
- **A complete protocol adapter per OS:** duplicates ordered state machines
and lets platform implementations drift on exact-byte and recovery laws.
- **Remove the Linux gate and try ordinary file operations:** turns successful
individual syscalls into an unsupported durability claim. Admission must
remain tied to a named, tested filesystem profile.
- **Require a separate case-sensitive volume without evaluating default
APFS:** avoids a useful native deployment but leaves ordinary Mac storage
unsupported. Exact canonical namespace handling is part of the target.
- **Check spelling with `F_GETPATH`:** it reports a name cached on the vnode.
With hard links, which publication uses, it can name another link of the same
file, so it both refuses valid entries and misreports aliases.
- **Detect the naming variant and scan only on case-insensitive volumes:** needs
an unsafe `fpathconf` binding to save a scan, and adds a branch whose untaken
side no single volume exercises.
- **Claim support from Linux containers on a Mac:** exercises Linux semantics,
not Darwin/APFS. Native acceptance requires native production-path evidence.

## Consequences

The OS port is an infrastructure boundary; semantic domain ports remain free
of descriptors and filesystem-library types. Adding an OS entails one adapter,
one selector entry, and its conformance evidence, rather than changing every
durability protocol. Filesystem support stays explicit and fail-closed.

Existing Linux wire bytes remain unchanged. Darwin recovery coordinates set the
top device bit and store `st_dev` as both device and mount, because each mounted
APFS volume has its own `st_dev`. Darwin renumbers `st_dev` when a volume is
reattached, so its restart check compares only that top bit and the inode: a record
from another OS still refuses, but a store moved to another volume at the same inode
number is admitted. Logical content identity does not incorporate native metadata.

Finder metadata such as `.DS_Store` in a store root refuses admission, like any
other entry the namespace cannot account for.

On macOS, opening an existing entry costs one streaming scan of its parent
directory. A rename that replaces a name during that scan could in principle hide
it and cause a false refusal; a bounded stress law did not observe one, and such
a refusal fails closed.

The native validation environment is separately bounded and serialized within
the same project resource budget. Successful process-death recovery is not
evidence of physical power-loss survival.
Loading
Loading