fix(gdpr): Strict mode no longer claims deleted documents are not zeroed (#299)

docs/DPIA_CHECKLIST.md

@@ -33,7 +33,7 @@ Features the engine will not implement are marked **(won't implement)** — the
 |---|---|---|---|---|
 | Unauthorised access to the `.db` file | Medium | High | Encryption at-rest (AES-256-GCM, page-level) | [`roadmap/v5/ENCRYPTION_FIX_PLAN.md`](../roadmap/v5/ENCRYPTION_FIX_PLAN.md) |
 | Exfiltration via backup file | Low | High | Encrypted backups (same key derivation) | [`roadmap/v5/ENCRYPTION_FIX_PLAN.md`](../roadmap/v5/ENCRYPTION_FIX_PLAN.md) |
-| Residual data after `DeleteAsync` | Medium | Medium | Slot-level secure erase via `VacuumAsync` (`SecureErase = true`); per-delete `SecureEraseOnDelete` engine toggle **(won't implement — call `VacuumAsync` after deletion)** | [`roadmap/v5/MISSING_FEATURES.md`](../roadmap/v5/MISSING_FEATURES.md) §3 |
+| Residual data after `DeleteAsync` | Medium | Medium | Every delete zero-fills the document's bytes on its page and frees its overflow pages zeroed; the database file receives the zeroed page at the next checkpoint (`CheckpointAsync`). `VacuumAsync` (`SecureErase = true`, the default) additionally zero-fills the free space of every page | `DocumentCollection.DeleteCore`, `SlottedPageUtils.CompactAndErase`, `PageFile.FreePage` |
 | Compromised audit log | Low | High | `IBLiteAuditSink`; implement an append-only file sink with optional cryptographic signing in the host application **(won't implement in engine)** | [`roadmap/v5/AUDIT_IMPLEMENTATION.md`](../roadmap/v5/AUDIT_IMPLEMENTATION.md) |
 | Encryption key loss | Low | Critical | `IKeyProvider` with external KMS | [`roadmap/v5/ENCRYPTION_PLAN.md`](../roadmap/v5/ENCRYPTION_PLAN.md) |
 | CDC PII leak | Medium | Medium | Field masking via `WatchOptions.ExcludeFields`/`IncludeOnlyFields` | [`roadmap/v5/GDPR_PLAN.md`](../roadmap/v5/GDPR_PLAN.md) §4.4 |
@@ -51,7 +51,7 @@ The following settings **must** be enabled for any deployment that processes hig
 
 - [ ] **Encryption configured** — supply a `CryptoOptions` instance (passphrase or `IKeyProvider`) when opening the engine. Verify `DatabaseInspectionReport.IsEncryptionEnabled == true` after open.
 - [ ] **Audit sink registered** — implement and register `IBLiteAuditSink` via `BLiteEngine.ConfigureAudit(...)`. All subject-export and inspection operations emit audit events through this sink.
-- [ ] **Secure erase on vacuum** — call `VacuumAsync(new VacuumOptions { SecureErase = true })` as part of your maintenance schedule and after bulk deletions of personal data to overwrite freed page bytes. A per-delete engine toggle will not be provided; the host application must call `VacuumAsync` explicitly as part of its erasure workflow.
+- [ ] **Checkpoint after erasure** — every `DeleteAsync` zero-fills the deleted document's bytes on its page, and the zeroed page reaches the database file at the next checkpoint. Call `CheckpointAsync()` after deleting personal data (a `DocumentDbContext` checkpoints in `DisposeAsync()`, not in `Dispose()`; a `BLiteEngine` does not). `VacuumAsync(new VacuumOptions { SecureErase = true })` additionally zero-fills the free space of every page and reclaims space; run it as part of your maintenance schedule.
 - [ ] **Retention policy on `[PersonalData]` collections** — configure `HasRetentionPolicy(...)` on every `EntityTypeBuilder<T>` that carries `[PersonalData]` fields so that data is not retained beyond its lawful purpose.
 - [ ] **CDC payload masking** — set `WatchOptions.ExcludeFields` or `WatchOptions.IncludeOnlyFields` on every `Watch(...)` call that may observe personal-data collections, or leave `CapturePayload = false` (the default) for streams that do not require payload visibility.
 - [ ] **`GdprMode = Strict` enabled** — set `DefaultGdprMode = GdprMode.Strict` in `BLiteKvOptions`, or call `.HasGdprMode(GdprMode.Strict)` per collection via `EntityTypeBuilder<T>`. Strict mode validates that encryption, audit, and retention are configured at engine-open time and logs actionable warnings for any gap.
@@ -65,7 +65,7 @@ The following responsibilities belong to the **host application** (the data cont
 - [ ] **External key management** — store encryption keys or passphrases in an external KMS or HSM. Implement `IKeyProvider` to supply keys at open time. Never embed raw key material in application source code or configuration files.
 - [ ] **Host-level access control on the `.db` file path** — restrict OS-level file permissions on the database file and its directory so that only the application process (and authorised administrators) can read or write the file. BLite has no independent access-control layer.
 - [ ] **Subject identifier convention** — choose a stable, consistent field name (e.g. `userId`, `subjectId`) as the `SubjectQuery.FieldName` for all `ExportSubjectDataAsync` calls. Annotate the corresponding property with `[PersonalData]` so it appears in `DatabaseInspectionReport`.
-- [ ] **Deletion workflow** — implement a subject-erasure workflow that: (1) calls `DeleteAsync` on all relevant documents, (2) calls `VacuumAsync(new VacuumOptions { SecureErase = true })` to physically overwrite freed pages, and (3) records the erasure in the external audit log. A built-in per-delete secure-erase toggle will not be provided; the explicit `VacuumAsync` call is the supported erasure mechanism.
+- [ ] **Deletion workflow** — implement a subject-erasure workflow that: (1) calls `DeleteAsync` on all relevant documents (each delete zero-fills the document's bytes on its page), (2) calls `CheckpointAsync()` so the zeroed pages are written into the database file, and (3) records the erasure in the external audit log. `VacuumAsync(new VacuumOptions { SecureErase = true })` is a maintenance step, not a prerequisite for erasure. Zero-filling covers the database file only, not the OS page cache, SSD wear-levelling or swap; encryption at rest, which Strict mode requires, covers those.
 - [ ] **Audit log retention outside BLite** — define and enforce a retention policy for the external audit log produced by your `IBLiteAuditSink` implementation. BLite emits events but does not manage the long-term storage or deletion of audit records.
 
 ---
@@ -80,7 +80,7 @@ The following responsibilities belong to the **host application** (the data cont
 | Art. 5(1)(e) | Storage limitation | `HasRetentionPolicy(...)` on `EntityTypeBuilder<T>`; `GdprMode.Strict` warns on missing retention |
 | Art. 5(1)(f) | Integrity and confidentiality | AES-256-GCM page encryption; WAL encryption; `IBLiteAuditSink` audit trail |
 | Art. 15 | Right of access (subject access request) | `ExportSubjectDataAsync` |
-| Art. 17 | Right to erasure | `DeleteAsync` + `VacuumAsync` (`SecureErase = true`) |
+| Art. 17 | Right to erasure | `DeleteAsync` (zero-fills on the page) + `CheckpointAsync` (writes the zeroed page to the file); `VacuumAsync` (`SecureErase = true`) for free space of every page |
 | Art. 20 | Right to data portability | `ExportSubjectDataAsync` (JSON / BSON export formats) |
 | Art. 25 | Data protection by design and by default | `GdprMode.Strict`; `[PersonalData]` compile-time annotation; `CryptoOptions` required in Strict mode |
 | Art. 30 | Records of processing activities | `InspectDatabase` → `DatabaseInspectionReport` (collections, indexes, encryption status, retention policies) |

src/BLite.Core/GDPR/GdprMode.cs

@@ -19,7 +19,7 @@ public enum GdprMode : byte
     ///   <item>Throws <see cref="InvalidOperationException"/> when encryption is absent.</item>
     ///   <item>Warns when no audit sink is registered.</item>
     ///   <item>Warns when a <c>[PersonalData]</c> collection has no retention policy.</item>
-    ///   <item>Warns when <c>SecureEraseOnDelete</c> cannot be enforced.</item>
+    ///   <item>Reminds that deleted documents are zero-filled on their page and reach the database file at the next checkpoint.</item>
     /// </list>
     /// Strict mode never deletes data, never rotates keys, never modifies stored documents.
     /// </summary>

src/BLite.Core/GDPR/GdprStrictValidator.cs

@@ -37,8 +37,11 @@ internal static class GdprStrictValidator
     /// <summary>SecureEraseOnDelete was forced to <see langword="true"/>.</summary>
     internal const int GdprStrictSecureEraseEnabled = 9005;
 
-    /// <summary><c>SecureEraseOnDelete</c> is not yet implemented; noting the gap.</summary>
-    internal const int GdprStrictSecureEraseUnavailable = 9006;
+    /// <summary>
+    /// Deleted documents are zero-filled on their page, but the database file receives the
+    /// zeroed page only at the next checkpoint; reminds the operator to checkpoint after erasure.
+    /// </summary>
+    internal const int GdprStrictSecureEraseCheckpointHint = 9006;
 
     /// <summary>A CDC Watch was configured with <c>CapturePayload = true</c> under Strict.</summary>
     internal const int GdprStrictCdcCapturePayloadWarning = 9007;
@@ -98,13 +101,16 @@ internal static void Apply(
                 "after construction to satisfy Art. 30 logging requirements.");
         }
 
-        // ── 3. Secure erase on delete — not yet a BLiteKvOptions setting ─────
-        // SecureEraseOnDelete as an engine-level toggle has not been implemented.
-        // Emit a single warning so the operator is aware of the gap.
-        TraceWarning(GdprStrictSecureEraseUnavailable,
-            "GdprMode.Strict: SecureEraseOnDelete is not yet available as an engine-level setting. " +
-            "Deleted document slots are not currently zeroed at the page layer. " +
-            "Track MISSING_FEATURES.md §3 for the planned implementation.");
+        // ── 3. Secure erase on delete — the checkpoint caveat ────────────────
+        // Every delete zero-fills the bytes the document occupied on its page
+        // (DocumentCollection.DeleteCore, SlottedPageUtils.CompactAndErase) and frees its
+        // overflow pages with a zeroed buffer (PageFile.FreePage), since #96. Like every write,
+        // the zeroed page reaches the database file at the next checkpoint; until then the file
+        // can still hold the old bytes. Emit a single reminder of that timing.
+        TraceWarning(GdprStrictSecureEraseCheckpointHint,
+            "GdprMode.Strict: deleted documents are zero-filled on their page, but the database file " +
+            "receives the zeroed page at the next checkpoint. Call CheckpointAsync() after deleting " +
+            "personal data; VacuumAsync(SecureErase = true) additionally zero-fills the free space of every page.");
 
         // ── 4. Retention — warn per [PersonalData] collection without policy ──
         try

tests/BLite.Tests/Gdpr/GdprStrictTests.cs

@@ -138,23 +138,69 @@ public void GdprStrict_EncryptionNoAudit_EmitsAuditMissingWarning()
         Assert.Equal(1, CountWarnings(GdprStrictValidator.GdprStrictAuditMissing));
     }
 
-    // ── Test 5: Strict forces SecureErase warning (not yet implemented) ────────
+    // ── Test 5: Strict reminds to checkpoint after erasure (#299) ──────────────
 
     /// <summary>
-    /// Strict mode warns that SecureEraseOnDelete is not yet available
-    /// (EventId <c>GdprStrictSecureEraseUnavailable</c>, 9006).
+    /// Strict mode emits the checkpoint reminder (EventId
+    /// <c>GdprStrictSecureEraseCheckpointHint</c>, 9006) and no longer claims that deleted
+    /// documents are not zeroed: every delete zero-fills them (see the test below).
     /// </summary>
     [Fact]
-    public void GdprStrict_SecureErase_EmitsUnavailableWarning()
+    public void GdprStrict_SecureErase_EmitsCheckpointHint_NotAnUnavailableWarning()
     {
         var path = TempDb();
         var crypto = new CryptoOptions("gdpr-strict-test-passphrase");
         var kvOpts = new BLiteKvOptions { DefaultGdprMode = GdprMode.Strict };
 
         using var engine = new BLiteEngine(path, crypto, kvOpts);
 
-        Assert.True(HasWarning(GdprStrictValidator.GdprStrictSecureEraseUnavailable),
-            "Expected GdprStrictSecureEraseUnavailable (9006) warning when Strict mode is active.");
+        var message = Assert.Single(_listener.Messages, m => m.Contains($"EventId:{GdprStrictValidator.GdprStrictSecureEraseCheckpointHint}"));
+        Assert.Contains("CheckpointAsync", message);
+        Assert.Contains("zero-filled", message);
+        Assert.DoesNotContain("not currently zeroed", message);
+        Assert.DoesNotContain("not yet available", message);
+        Assert.DoesNotContain("MISSING_FEATURES", message);
+    }
+
+    /// <summary>
+    /// What the reminder describes: after a delete and a checkpoint, no byte of the deleted
+    /// document is left in the database file or the WAL. Plaintext, so the bytes are visible.
+    /// </summary>
+    [Fact]
+    public async Task Delete_ThenCheckpoint_LeavesNoBytesOfTheDocumentInTheFiles()
+    {
+        var path = TempDb();
+        var walPath = Path.ChangeExtension(path, ".wal");
+        var marker = System.Text.Encoding.UTF8.GetBytes("ZZ-PERSONAL-MARKER-7f3a9c-ZZ");
+
+        BLite.Bson.BsonId id;
+        using (var engine = new BLiteEngine(path))
+        {
+            var col = engine.GetOrCreateCollection("people");
+            var doc = engine.CreateDocument(["name"], b => b.AddString("name", "ZZ-PERSONAL-MARKER-7f3a9c-ZZ"));
+            id = await col.InsertAsync(doc);
+            await engine.CheckpointAsync();
+        }
+        Assert.True(CountOccurrences(File.ReadAllBytes(path), marker) >= 1, "the marker must be in the file before the delete");
+
+        using (var engine = new BLiteEngine(path))
+        {
+            var col = engine.GetOrCreateCollection("people");
+            Assert.True(await col.DeleteAsync(id));
+            await engine.CheckpointAsync();
+        }
+
+        Assert.Equal(0, CountOccurrences(File.ReadAllBytes(path), marker));
+        if (File.Exists(walPath))
+            Assert.Equal(0, CountOccurrences(File.ReadAllBytes(walPath), marker));
+    }
+
+    private static int CountOccurrences(byte[] haystack, byte[] needle)
+    {
+        int count = 0;
+        for (int i = 0; i <= haystack.Length - needle.Length; i++)
+            if (haystack.AsSpan(i, needle.Length).SequenceEqual(needle)) count++;
+        return count;
     }
 
     // ── Test 6: GdprMode enum values are correct ─────────────────────────────