Integrate backlog step 4: one storage format, macOS port, and expected state on open (#204)

.github/workflows/ci.yml

@@ -134,7 +134,7 @@ jobs:
       - name: Test debug profile on case-sensitive APFS
         run: KEEP_TEST_SCRATCH_ROOT="$RUNNER_TEMP/keep-cs" KEEP_TEST_EXPECT_APFS_VARIANT=case-sensitive cargo test --all-features --locked
 
-      - name: Reopen a migrated store after reattaching its volume
+      - name: Reopen a store after reattaching its volume
         run: |
           hdiutil create -size 16m -type SPARSE -fs APFS -volname keep-occupier "$RUNNER_TEMP/keep-occupier.sparseimage"
           mkdir "$RUNNER_TEMP/keep-occupier"

CHANGELOG.md

@@ -8,6 +8,8 @@ after its public API and format compatibility policies are established.
 
 ## [Unreleased]
 
+- Opening a store requires a typed choice (#198): `FilesystemStoreAdmission::reopen`, `FilesystemCatalogSnapshot::load`, `FilesystemRetentionSnapshot::load`, `DurableSnapshot::open`, and `DurableStore::open` take a `StoreExpectation` with no default. `DurableStore::open` checks nothing until a snapshot is pinned. `Catalog` refuses any other published catalog, or none, with the expected and observed values, and `NoExpectation` accepts any valid store at the path. `ExpectedCatalog` is the caller's claim, not a verified digest: `new` takes the generation and digest a receipt or view reports, `from_recorded` rebuilds it in a later process from a recorded generation and 32 digest bytes, and `digest_bytes` returns the bytes to record. `ObservedCatalog` is the observed side of `StoreExpectationMismatch`. A mismatch from `FilesystemCatalogSnapshot::load` is the new `CatalogRestartError::Expectation`, which exhaustive matches on that error must admit. Keep stores no expectation for the caller. Recovery tools, verification loads, initialization, and the repository-task reopen take none, so their callers do not get this check. Limits: without an expectation, a torn copy may silently be the older state. A `Catalog` expectation misses rollbacks that leave the catalog unchanged. After a crash between a publication and the caller's record of it, the old expectation refuses, and accepting the observed catalog is the caller's own trust decision; every `retained-catalog` crash case (`KEEP-CRASH-074` to `-077`) asserts that refusal once `HEAD` names the successor. Enumerated laws show that every mix of two quiescent published states, and a half-length copy of any one file over the newer state, refuses with a typed absent-or-malformed cause or returns exact bytes. They do not cover a copy that holds a writer's in-flight stage, other truncation lengths, or a tear over the older state.
+
 - Stop naming Cargo workspace members as Dependabot update directories. Dependabot refuses to update from inside a member and the job for the workspace root already updates member manifests, so those entries failed the scheduled run without adding coverage. The dependency policy check now reads the root workspace member list, refuses a member listed as an update directory, and still requires an entry for every standalone manifest.
 
 - Admit macOS APFS as a production storage platform, on both default and case-sensitive volumes. Keep requires every existing protocol name byte-for-byte in a listing of its parent, so `head` never answers for `HEAD`. Adds a native macOS CI job; no format, dependency, or unsafe change. The crash matrix passes natively, but no power-loss experiment has been run on APFS.
@@ -24,7 +26,14 @@ after its public API and format compatibility policies are established.
 
 - Add field-mutation laws for selected version-2 retention root, manifest, and head fields: rows mutate the frozen corpus record, reseal every digest and checksum they did not target, and assert the named refusal variant. KEEP-RETENTION-003 remains in progress (#212). Salvaged from the closed #107 branch; tests and ledger only.
 
-- ADR-0010 commits Keep to one current storage format: direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, opening requires an explicit expected-state choice before the first release, and after it Keep reads released sealed segments while writing only the current format, with explicit metadata upgrades and optional, resumable segment rewriting. Main's behavior is unchanged until the units tracked in #198 land (#198).
+- ADR-0010 commits Keep to one current storage format: direct initialization replaces version-1 initialization plus migration, complete stores become portable while live authority checks remain, opening requires an explicit expected-state choice before the first release, and after it Keep reads released sealed segments while writing only the current format, with explicit metadata upgrades and optional, resumable segment rewriting (#198).
+
+- Keep has one storage format (#198, #125). In the order its units were built: `publish_retained_catalog_generation` refuses a candidate catalog that loses retained content, verifying every current retained closure against the candidate before any catalog mutation and revalidating the retention head before `HEAD` replacement. `FilesystemCatalogPublisher` stages one segment and publishes a catalog successor through that gate, catalog-stage recovery finalizes a staged head only after the same gate admits it, and `KEEP-CRASH-074` to `-077` cover writer process death. `FilesystemStoreAdmission::initialize` creates a store directly through a recoverable protocol with one durable commit point and a 41-case crash matrix (`KEEP-CRASH-081`–`091`); laws witness the order and the failure of the commit-point synchronization at an injected synchronizer, and that the production call reaches stable storage is by inspection, with no power-loss test. Migration and its public API, records, golden fixtures, and crash rows `KEEP-CRASH-053`–`073` are deleted, the persisted root device-and-inode binding is removed so quiescent copies and backups open, and the format-definition digest changes to `f1a0ce10…361d`. Version 1 is deleted: every store open refuses a root without the `FORMAT` marker, `KEEP-CRASH-001`–`030` run on initialized stores, `KEEP-CRASH-031`–`035` are retired, the `VersionTwo*` types take format-neutral names (`FilesystemStoreAdmission`, `FilesystemCatalogPublisher`, `StoreInitialization`, `classify_recovery_names`), and the format pages are one set under `docs/formats/store/`.
+
+- Platform coordinates are live observations only (#198): the internal `FilesystemDirectoryCoordinate` carries a directory's device and mount for comparison within one operation, and the object coordinate and the Darwin coordinate domain, which existed for the persisted root binding, are removed. Nothing persists a coordinate.
+
+- Not yet implemented: `KEEP-CRASH-079` and `-080`, process death in recovery's stage-completion link and in gated finalization's `HEAD` replacement, are reserved. The plan in #198 placed them before version 1's deletion. The owner accepted on 2026-11-05 that they follow it; #221 tracks them and blocks the first release together with #205.
+
 - Update the repository-only serde_json parser to 1.0.154 in both locked workspaces and refresh its dependency admission; the resolved transitive graph, features, license and MSRV are unchanged (#186).
 
 - Update the in-process BLAKE3 dependency to 1.8.7 in both locked workspaces, remove the unused arrayref license exception and renew its admission while retaining the external b3sum oracle and version-one identity contract (#94).

README.md

@@ -45,42 +45,48 @@ Keep is required to refuse all three, before mutating anything.
   single byte reaches the caller. It also provides
   authenticated exact byte-range reads that load only the overlapping chunks
   and state their narrower claim explicitly.
-- **Durable version-1 segment store.** `StagedSegment` writes only
+- **Durable segment store.** `StagedSegment` writes only
   content-admitted records; `AdmittedSegment` exposes payloads only after
   complete framing, checksum, and identity verification. Immutable segments,
   generation-versioned catalogs, and a fixed-width `HEAD` are published
   through an ordered protocol whose every step is a named crash point.
   Platform admission is non-casefolded Linux ext4 or local writable macOS APFS,
   one writer.
 - **Process-death recovery evidence.** The crash matrix kills real writer processes
-  at 156 before/during/after coordinates (`KEEP-CRASH-001`–`052`) and
-  verifies the store lands in exactly one documented lawful state each time,
-  for version-1 publication and for version-2 retention publication.
-- **Version-2 retention and migration, forward path.** Explicit retention
-  roots, deterministic closure verification, a one-way 21-phase migration,
-  and a 17-phase retention publication — all with production filesystem
-  writers, all preserving every version-1 byte. Reopening a migrated store
-  jointly admits its marker, intent, and receipt, binds the root's device and
-  inode identity to the intent, and pins the directories it admitted. Mount
-  identity remains a check within the live migration process. Publication
+  at 194 before/during/after coordinates over 62 boundaries and verifies the
+  store lands in exactly one documented lawful state each time: 141 for
+  segment, catalog and head publication, recovery discard, and retention
+  publication (the 47 live rows of `KEEP-CRASH-001`–`052`), 12 for catalog
+  publication beside a retained root (`KEEP-CRASH-074`–`077`), and 41 for
+  initialization (`KEEP-CRASH-081`–`091`, with one case per directory). Process death is not power loss,
+  which nothing here tests.
+- **One store format, created directly.** `FilesystemStoreAdmission::initialize`
+  creates a version-2 store through a recoverable state machine with one
+  durable commit point; an interrupted call resumes, and any other nonempty
+  directory refuses before it is touched. A 41-case process-death matrix
+  (`KEEP-CRASH-081`–`091`) covers every boundary. Nothing persisted binds a
+  store to its directory: a quiescent copy or restored backup opens wherever
+  it is placed, as a second, independent store. Nothing in a valid store
+  shows that it is the wrong one or an older state, so every normal open
+  takes a `StoreExpectation`: the exact catalog the caller recorded, or an
+  explicit statement that it has none. There is no migration; see the
+  [retirement ledger](docs/formats/store/retired-identifiers.md).
+- **Version-2 retention, forward path.** Explicit retention roots,
+  deterministic closure verification, the gated segment and catalog writer,
+  and a 17-phase retention publication, all with production filesystem
+  writers. Reopening a store admits its namespace and marker and pins the
+  directories it admitted. Publication
   binds this store's own catalog head and the catalog it selects, and refuses
   retained stages, superseded candidates, substituted
   files, replaced protocol directories, and every namespace or capacity
   violation before it writes anything. Each refusal is a typed value, not a
   string.
-- **Migration restart recovery.** Recovery verifies current authority,
-  classifies the observed prefix, and resumes an exact persisted migration
-  intent through the remaining phases. An incomplete pre-effect intent stage
-  is discarded and rebuilt from freshly verified current intent. Its crash
-  matrix kills real writer processes
-  at 68 before/during/after coordinates (`KEEP-CRASH-053`–`073`), preserving
-  every version-1 byte. The [restart matrix](docs/testing-evidence/migration-restart-matrix.md) records additional hostile-prefix evidence and its limits.
 
 Version-2 complete-stage retention recovery, fenced retention snapshots and model-based transitions are implemented on main through [PR #99](https://github.com/flyingrobots/keep/pull/99).
 
 The retention process-death sequence checks the recovered head generation and exact selected-root bytes before retry; its [evidence receipt](docs/testing-evidence/retention-crash-reader-oracle.md) bounds that claim to the declared initial-publication crash coordinates.
 
-Recovery failures retain their typed cause and report known effects separately from uncertain effects or durability under the [approved recovery contract](docs/formats/segment-store-v2/retention-recovery.md).
+Recovery failures retain their typed cause and report known effects separately from uncertain effects or durability under the [approved recovery contract](docs/formats/store/retention-recovery.md).
 
 Writer authority coordinates cooperating writers in a managed namespace; it does not isolate arbitrary concurrent out-of-band filesystem mutation.
 
@@ -96,9 +102,7 @@ Incomplete retention stages are preserved and block publication pending explicit
 
 Complete orphans remain recovery-protected until explicit disposition lands with garbage collection (#21).
 
-Migration restart recovery is implemented and does not grant retention authority.
-
-A version-1 store stays admitted until its owner migrates it.
+Version-1 admission is deleted ([#198](https://github.com/flyingrobots/keep/issues/198)); every open path refuses a root without the `FORMAT` marker, and no conversion from version 1 exists.
 
 | Gap | Tracked |
 | --- | --- |
@@ -204,25 +208,25 @@ assert_eq!(output, b"exact bytes, or nothing");
 # Ok::<(), Box<dyn std::error::Error>>(())
 ```
 
-On Linux with the admitted ext4 profile, read an existing migrated version-two store whose retention roots anchor the requested blob. The following example is also compiled in the `DurableStore` API documentation. Its 16 MiB limit applies to aggregate catalog-selected segment bytes; catalog bytes and decoded metadata allocate separately.
+On an admitted Linux ext4 or macOS APFS volume, read an initialized store whose retention roots anchor the requested blob. The following example is also compiled in the `DurableStore` API documentation. Its 16 MiB limit applies to aggregate catalog-selected segment bytes; catalog bytes and decoded metadata allocate separately. It passes `StoreExpectation::NoExpectation`, which accepts any valid store at the path, including a different store or an older state of the intended one. That is the right choice for a first open. A caller that has recorded a catalog from a publication receipt or a view passes `StoreExpectation::Catalog` instead, rebuilt with `ExpectedCatalog::from_recorded`, and the open refuses with the expected and observed catalogs when they differ.
 
 ```rust
-#[cfg(target_os = "linux")]
+#[cfg(any(target_os = "linux", target_os = "macos"))]
 fn copy_retained_blob(
     root: &std::path::Path,
     target: keep::BlobId,
     output: &mut impl std::io::Write,
 ) -> Result<keep::DurableReconstructionReceipt, Box<dyn std::error::Error>> {
     use keep::{
         CatalogRestartByteLimit, CatalogRestartPolicy, DurableStore, LayoutEntryLimit,
-        ReaderAttemptLimit, SegmentReadPolicy, SegmentRecordLimit,
+        ReaderAttemptLimit, SegmentReadPolicy, SegmentRecordLimit, StoreExpectation,
     };
     // Limit catalog-selected segment bytes; catalog and metadata allocate separately.
     let policy = CatalogRestartPolicy::new(
         SegmentReadPolicy::new(SegmentRecordLimit::MAXIMUM, LayoutEntryLimit::MAXIMUM),
         CatalogRestartByteLimit::new(16_777_216)?,
     );
-    let store = DurableStore::open(root, policy, ReaderAttemptLimit::DEFAULT)?;
+    let store = DurableStore::open(root, policy, ReaderAttemptLimit::DEFAULT, StoreExpectation::NoExpectation)?;
     let snapshot = store.snapshot()?;
     Ok(snapshot.reconstruct(target, output)?)
 }
@@ -247,9 +251,9 @@ Select a crash campaign with `--sequence NAME` using these exact CLI names:
 | `catalog` | Catalog publication |
 | `head` | Publication-head replacement |
 | `recovery-discard` | Explicit recovery evidence discard |
-| `initialization` | Writer-locked initialization |
 | `retention` | Retention root, manifest, and head publication |
-| `migration` | Version-one to version-two migration |
+| `retained-catalog` | Successor catalog publication beside a retained root, and its stage recovery |
+| `initialization` | Direct initialization and its recovery |
 
 ```bash
 cargo xtask durability-crash-matrix --sequence retention
@@ -288,8 +292,8 @@ has one job; this one is the front door.
 | Understand what is proved and what is not | [`docs/invariants/`](docs/invariants/) |
 | Read the byte-level formats | [`docs/formats/`](docs/formats/) |
 | See the architecture and port boundaries | [`docs/architecture/`](docs/architecture/) |
-| Follow the crash and recovery rules | [`segment-store-v1/recovery.md`](docs/formats/segment-store-v1/recovery.md) · [`segment-store-v2/recovery.md`](docs/formats/segment-store-v2/recovery.md) · [`segment-store-v2/migration-recovery.md`](docs/formats/segment-store-v2/migration-recovery.md) |
-| See how a retention generation is published | [`segment-store-v2/retention-publication.md`](docs/formats/segment-store-v2/retention-publication.md) |
+| Follow the crash and recovery rules | [`store/stage-recovery.md`](docs/formats/store/stage-recovery.md) · [`store/recovery.md`](docs/formats/store/recovery.md) · [`store/initialization.md`](docs/formats/store/initialization.md) |
+| See how a retention generation is published | [`store/retention-publication.md`](docs/formats/store/retention-publication.md) |
 | Check reproducible performance evidence | [`docs/benchmarks/`](docs/benchmarks/) |
 | Run the language-neutral corpora | [`conformance/`](conformance/) |
 | See planned features and their tasks | [`ROADMAP.md`](ROADMAP.md) |
@@ -307,4 +311,4 @@ plaintext content, keys, or sensitive paths in a public issue.
 Licensed under the [Apache License 2.0](LICENSE).
 
 [reconstruction]: docs/invariants/authenticated-reconstruction/README.md
-[ledger]: docs/formats/segment-store-v2/requirements.md
+[ledger]: docs/formats/store/requirements.md