diff --git a/docs/DPIA_CHECKLIST.md b/docs/DPIA_CHECKLIST.md index 28aab98f..ba8e0bef 100644 --- a/docs/DPIA_CHECKLIST.md +++ b/docs/DPIA_CHECKLIST.md @@ -33,7 +33,7 @@ Features the engine will not implement are marked **(won't implement)** — the |---|---|---|---|---| | Unauthorised access to the `.db` file | Medium | High | Encryption at-rest (AES-256-GCM, page-level) | [`roadmap/v5/ENCRYPTION_FIX_PLAN.md`](../roadmap/v5/ENCRYPTION_FIX_PLAN.md) | | Exfiltration via backup file | Low | High | Encrypted backups (same key derivation) | [`roadmap/v5/ENCRYPTION_FIX_PLAN.md`](../roadmap/v5/ENCRYPTION_FIX_PLAN.md) | -| Residual data after `DeleteAsync` | Medium | Medium | Slot-level secure erase via `VacuumAsync` (`SecureErase = true`); per-delete `SecureEraseOnDelete` engine toggle **(won't implement — call `VacuumAsync` after deletion)** | [`roadmap/v5/MISSING_FEATURES.md`](../roadmap/v5/MISSING_FEATURES.md) §3 | +| Residual data after `DeleteAsync` | Medium | Medium | Every delete zero-fills the document's bytes on its page and frees its overflow pages zeroed; the database file receives the zeroed page at the next checkpoint (`CheckpointAsync`). `VacuumAsync` (`SecureErase = true`, the default) additionally zero-fills the free space of every page | `DocumentCollection.DeleteCore`, `SlottedPageUtils.CompactAndErase`, `PageFile.FreePage` | | Compromised audit log | Low | High | `IBLiteAuditSink`; implement an append-only file sink with optional cryptographic signing in the host application **(won't implement in engine)** | [`roadmap/v5/AUDIT_IMPLEMENTATION.md`](../roadmap/v5/AUDIT_IMPLEMENTATION.md) | | Encryption key loss | Low | Critical | `IKeyProvider` with external KMS | [`roadmap/v5/ENCRYPTION_PLAN.md`](../roadmap/v5/ENCRYPTION_PLAN.md) | | CDC PII leak | Medium | Medium | Field masking via `WatchOptions.ExcludeFields`/`IncludeOnlyFields` | [`roadmap/v5/GDPR_PLAN.md`](../roadmap/v5/GDPR_PLAN.md) §4.4 | @@ -51,7 +51,7 @@ The following settings **must** be enabled for any deployment that processes hig - [ ] **Encryption configured** — supply a `CryptoOptions` instance (passphrase or `IKeyProvider`) when opening the engine. Verify `DatabaseInspectionReport.IsEncryptionEnabled == true` after open. - [ ] **Audit sink registered** — implement and register `IBLiteAuditSink` via `BLiteEngine.ConfigureAudit(...)`. All subject-export and inspection operations emit audit events through this sink. -- [ ] **Secure erase on vacuum** — call `VacuumAsync(new VacuumOptions { SecureErase = true })` as part of your maintenance schedule and after bulk deletions of personal data to overwrite freed page bytes. A per-delete engine toggle will not be provided; the host application must call `VacuumAsync` explicitly as part of its erasure workflow. +- [ ] **Checkpoint after erasure** — every `DeleteAsync` zero-fills the deleted document's bytes on its page, and the zeroed page reaches the database file at the next checkpoint. Call `CheckpointAsync()` after deleting personal data (a `DocumentDbContext` checkpoints in `DisposeAsync()`, not in `Dispose()`; a `BLiteEngine` does not). `VacuumAsync(new VacuumOptions { SecureErase = true })` additionally zero-fills the free space of every page and reclaims space; run it as part of your maintenance schedule. - [ ] **Retention policy on `[PersonalData]` collections** — configure `HasRetentionPolicy(...)` on every `EntityTypeBuilder` that carries `[PersonalData]` fields so that data is not retained beyond its lawful purpose. - [ ] **CDC payload masking** — set `WatchOptions.ExcludeFields` or `WatchOptions.IncludeOnlyFields` on every `Watch(...)` call that may observe personal-data collections, or leave `CapturePayload = false` (the default) for streams that do not require payload visibility. - [ ] **`GdprMode = Strict` enabled** — set `DefaultGdprMode = GdprMode.Strict` in `BLiteKvOptions`, or call `.HasGdprMode(GdprMode.Strict)` per collection via `EntityTypeBuilder`. Strict mode validates that encryption, audit, and retention are configured at engine-open time and logs actionable warnings for any gap. @@ -65,7 +65,7 @@ The following responsibilities belong to the **host application** (the data cont - [ ] **External key management** — store encryption keys or passphrases in an external KMS or HSM. Implement `IKeyProvider` to supply keys at open time. Never embed raw key material in application source code or configuration files. - [ ] **Host-level access control on the `.db` file path** — restrict OS-level file permissions on the database file and its directory so that only the application process (and authorised administrators) can read or write the file. BLite has no independent access-control layer. - [ ] **Subject identifier convention** — choose a stable, consistent field name (e.g. `userId`, `subjectId`) as the `SubjectQuery.FieldName` for all `ExportSubjectDataAsync` calls. Annotate the corresponding property with `[PersonalData]` so it appears in `DatabaseInspectionReport`. -- [ ] **Deletion workflow** — implement a subject-erasure workflow that: (1) calls `DeleteAsync` on all relevant documents, (2) calls `VacuumAsync(new VacuumOptions { SecureErase = true })` to physically overwrite freed pages, and (3) records the erasure in the external audit log. A built-in per-delete secure-erase toggle will not be provided; the explicit `VacuumAsync` call is the supported erasure mechanism. +- [ ] **Deletion workflow** — implement a subject-erasure workflow that: (1) calls `DeleteAsync` on all relevant documents (each delete zero-fills the document's bytes on its page), (2) calls `CheckpointAsync()` so the zeroed pages are written into the database file, and (3) records the erasure in the external audit log. `VacuumAsync(new VacuumOptions { SecureErase = true })` is a maintenance step, not a prerequisite for erasure. Zero-filling covers the database file only, not the OS page cache, SSD wear-levelling or swap; encryption at rest, which Strict mode requires, covers those. - [ ] **Audit log retention outside BLite** — define and enforce a retention policy for the external audit log produced by your `IBLiteAuditSink` implementation. BLite emits events but does not manage the long-term storage or deletion of audit records. --- @@ -80,7 +80,7 @@ The following responsibilities belong to the **host application** (the data cont | Art. 5(1)(e) | Storage limitation | `HasRetentionPolicy(...)` on `EntityTypeBuilder`; `GdprMode.Strict` warns on missing retention | | Art. 5(1)(f) | Integrity and confidentiality | AES-256-GCM page encryption; WAL encryption; `IBLiteAuditSink` audit trail | | Art. 15 | Right of access (subject access request) | `ExportSubjectDataAsync` | -| Art. 17 | Right to erasure | `DeleteAsync` + `VacuumAsync` (`SecureErase = true`) | +| Art. 17 | Right to erasure | `DeleteAsync` (zero-fills on the page) + `CheckpointAsync` (writes the zeroed page to the file); `VacuumAsync` (`SecureErase = true`) for free space of every page | | Art. 20 | Right to data portability | `ExportSubjectDataAsync` (JSON / BSON export formats) | | Art. 25 | Data protection by design and by default | `GdprMode.Strict`; `[PersonalData]` compile-time annotation; `CryptoOptions` required in Strict mode | | Art. 30 | Records of processing activities | `InspectDatabase` → `DatabaseInspectionReport` (collections, indexes, encryption status, retention policies) | diff --git a/src/BLite.Core/GDPR/GdprMode.cs b/src/BLite.Core/GDPR/GdprMode.cs index 521eaa7f..c940fc0c 100644 --- a/src/BLite.Core/GDPR/GdprMode.cs +++ b/src/BLite.Core/GDPR/GdprMode.cs @@ -19,7 +19,7 @@ public enum GdprMode : byte /// Throws when encryption is absent. /// Warns when no audit sink is registered. /// Warns when a [PersonalData] collection has no retention policy. - /// Warns when SecureEraseOnDelete cannot be enforced. + /// Reminds that deleted documents are zero-filled on their page and reach the database file at the next checkpoint. /// /// Strict mode never deletes data, never rotates keys, never modifies stored documents. /// diff --git a/src/BLite.Core/GDPR/GdprStrictValidator.cs b/src/BLite.Core/GDPR/GdprStrictValidator.cs index f498c0c2..57c9977b 100644 --- a/src/BLite.Core/GDPR/GdprStrictValidator.cs +++ b/src/BLite.Core/GDPR/GdprStrictValidator.cs @@ -37,8 +37,11 @@ internal static class GdprStrictValidator /// SecureEraseOnDelete was forced to . internal const int GdprStrictSecureEraseEnabled = 9005; - /// SecureEraseOnDelete is not yet implemented; noting the gap. - internal const int GdprStrictSecureEraseUnavailable = 9006; + /// + /// Deleted documents are zero-filled on their page, but the database file receives the + /// zeroed page only at the next checkpoint; reminds the operator to checkpoint after erasure. + /// + internal const int GdprStrictSecureEraseCheckpointHint = 9006; /// A CDC Watch was configured with CapturePayload = true under Strict. internal const int GdprStrictCdcCapturePayloadWarning = 9007; @@ -98,13 +101,16 @@ internal static void Apply( "after construction to satisfy Art. 30 logging requirements."); } - // ── 3. Secure erase on delete — not yet a BLiteKvOptions setting ───── - // SecureEraseOnDelete as an engine-level toggle has not been implemented. - // Emit a single warning so the operator is aware of the gap. - TraceWarning(GdprStrictSecureEraseUnavailable, - "GdprMode.Strict: SecureEraseOnDelete is not yet available as an engine-level setting. " + - "Deleted document slots are not currently zeroed at the page layer. " + - "Track MISSING_FEATURES.md §3 for the planned implementation."); + // ── 3. Secure erase on delete — the checkpoint caveat ──────────────── + // Every delete zero-fills the bytes the document occupied on its page + // (DocumentCollection.DeleteCore, SlottedPageUtils.CompactAndErase) and frees its + // overflow pages with a zeroed buffer (PageFile.FreePage), since #96. Like every write, + // the zeroed page reaches the database file at the next checkpoint; until then the file + // can still hold the old bytes. Emit a single reminder of that timing. + TraceWarning(GdprStrictSecureEraseCheckpointHint, + "GdprMode.Strict: deleted documents are zero-filled on their page, but the database file " + + "receives the zeroed page at the next checkpoint. Call CheckpointAsync() after deleting " + + "personal data; VacuumAsync(SecureErase = true) additionally zero-fills the free space of every page."); // ── 4. Retention — warn per [PersonalData] collection without policy ── try diff --git a/tests/BLite.Tests/Gdpr/GdprStrictTests.cs b/tests/BLite.Tests/Gdpr/GdprStrictTests.cs index fbe32637..24a9567f 100644 --- a/tests/BLite.Tests/Gdpr/GdprStrictTests.cs +++ b/tests/BLite.Tests/Gdpr/GdprStrictTests.cs @@ -138,14 +138,15 @@ public void GdprStrict_EncryptionNoAudit_EmitsAuditMissingWarning() Assert.Equal(1, CountWarnings(GdprStrictValidator.GdprStrictAuditMissing)); } - // ── Test 5: Strict forces SecureErase warning (not yet implemented) ──────── + // ── Test 5: Strict reminds to checkpoint after erasure (#299) ────────────── /// - /// Strict mode warns that SecureEraseOnDelete is not yet available - /// (EventId GdprStrictSecureEraseUnavailable, 9006). + /// Strict mode emits the checkpoint reminder (EventId + /// GdprStrictSecureEraseCheckpointHint, 9006) and no longer claims that deleted + /// documents are not zeroed: every delete zero-fills them (see the test below). /// [Fact] - public void GdprStrict_SecureErase_EmitsUnavailableWarning() + public void GdprStrict_SecureErase_EmitsCheckpointHint_NotAnUnavailableWarning() { var path = TempDb(); var crypto = new CryptoOptions("gdpr-strict-test-passphrase"); @@ -153,8 +154,53 @@ public void GdprStrict_SecureErase_EmitsUnavailableWarning() using var engine = new BLiteEngine(path, crypto, kvOpts); - Assert.True(HasWarning(GdprStrictValidator.GdprStrictSecureEraseUnavailable), - "Expected GdprStrictSecureEraseUnavailable (9006) warning when Strict mode is active."); + var message = Assert.Single(_listener.Messages, m => m.Contains($"EventId:{GdprStrictValidator.GdprStrictSecureEraseCheckpointHint}")); + Assert.Contains("CheckpointAsync", message); + Assert.Contains("zero-filled", message); + Assert.DoesNotContain("not currently zeroed", message); + Assert.DoesNotContain("not yet available", message); + Assert.DoesNotContain("MISSING_FEATURES", message); + } + + /// + /// What the reminder describes: after a delete and a checkpoint, no byte of the deleted + /// document is left in the database file or the WAL. Plaintext, so the bytes are visible. + /// + [Fact] + public async Task Delete_ThenCheckpoint_LeavesNoBytesOfTheDocumentInTheFiles() + { + var path = TempDb(); + var walPath = Path.ChangeExtension(path, ".wal"); + var marker = System.Text.Encoding.UTF8.GetBytes("ZZ-PERSONAL-MARKER-7f3a9c-ZZ"); + + BLite.Bson.BsonId id; + using (var engine = new BLiteEngine(path)) + { + var col = engine.GetOrCreateCollection("people"); + var doc = engine.CreateDocument(["name"], b => b.AddString("name", "ZZ-PERSONAL-MARKER-7f3a9c-ZZ")); + id = await col.InsertAsync(doc); + await engine.CheckpointAsync(); + } + Assert.True(CountOccurrences(File.ReadAllBytes(path), marker) >= 1, "the marker must be in the file before the delete"); + + using (var engine = new BLiteEngine(path)) + { + var col = engine.GetOrCreateCollection("people"); + Assert.True(await col.DeleteAsync(id)); + await engine.CheckpointAsync(); + } + + Assert.Equal(0, CountOccurrences(File.ReadAllBytes(path), marker)); + if (File.Exists(walPath)) + Assert.Equal(0, CountOccurrences(File.ReadAllBytes(walPath), marker)); + } + + private static int CountOccurrences(byte[] haystack, byte[] needle) + { + int count = 0; + for (int i = 0; i <= haystack.Length - needle.Length; i++) + if (haystack.AsSpan(i, needle.Length).SequenceEqual(needle)) count++; + return count; } // ── Test 6: GdprMode enum values are correct ─────────────────────────────