From 71bd2636b99939a00f85ad334abf4e8090867f96 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 14:37:46 -0500 Subject: [PATCH 1/5] Document database-wide-only transaction log deletion on RocksDB delete_transaction_logs_before now rejects table-scoped requests on RocksDB (HarperFast/harper#2049), and the deprecated delete_audit_logs_before always errors there since it requires table. Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 7af4f925..1d45faad 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -116,12 +116,14 @@ The `original_records` field contains the record state before the operation was #### `delete_audit_logs_before` -Deletes audit log entries older than the specified timestamp. +Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before). — Audit log cleanup improved to reduce resource consumption during scheduled cleanups — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold + — This operation requires `table`, but on the RocksDB storage engine (the default) history cannot be deleted for a single table because all tables in a database share one transaction log. On RocksDB this operation now always returns an error directing you to `delete_transaction_logs_before`; it remains usable on LMDB. + ```json { "operation": "delete_audit_logs_before", @@ -131,6 +133,24 @@ Deletes audit log entries older than the specified timestamp. } ``` +#### `delete_transaction_logs_before` + +Deletes transaction log entries older than the specified timestamp. + + + +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. + + — On RocksDB, a request that includes `table` is now rejected with a `400` error, and a `table` that does not exist returns a `404`. Previously the `table` scope was silently ignored and the entire database's transaction log was purged. On LMDB, `table` scopes the deletion to that table's history, unchanged. + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "timestamp": 1598290282817 +} +``` + --- ## Enabling Audit Log Per Table From 220040284a88fd6857abc99c6ef059241c2989c3 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 15:35:21 -0500 Subject: [PATCH 2/5] Add Transaction Log Operations section header Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 1d45faad..8ecbdb95 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -133,6 +133,8 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo } ``` +### Transaction Log Operations + #### `delete_transaction_logs_before` Deletes transaction log entries older than the specified timestamp. From f3d66404c2b9d507527793509ef79c279fd242f3 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Mon, 3 Aug 2026 16:01:35 -0500 Subject: [PATCH 3/5] Clarify engine scope of the 404 change and document cleanup_deleted_records Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 8ecbdb95..abf882fc 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -143,7 +143,9 @@ Deletes transaction log entries older than the specified timestamp. On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. - — On RocksDB, a request that includes `table` is now rejected with a `400` error, and a `table` that does not exist returns a `404`. Previously the `table` scope was silently ignored and the entire database's transaction log was purged. On LMDB, `table` scopes the deletion to that table's history, unchanged. + — On RocksDB, a request that includes `table` is now rejected with a `400` error; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now returns a `404` (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + +On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additionally removes leftover tombstone entries for records deleted before the timestamp — a repair step for tombstones that normal audit log cleanup should already have removed. It is ignored on RocksDB. ```json { From ffe30c4463a26ed38b483b970cc1c9936e75e912 Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 4 Aug 2026 00:25:22 -0500 Subject: [PATCH 4/5] Address review: job semantics, storage-model consistency, badge placement, type style Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 43 +++++++++++++++++++------------ 1 file changed, 27 insertions(+), 16 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index abf882fc..4c4b8d45 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -11,17 +11,17 @@ title: Transaction Logging # Transaction Logging -Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases. - -| Feature | Audit Log | Transaction Log | -| ----------------------------- | --------------------------------- | ------------------------------ | -| Storage | Standard Harper table (per-table) | Clustering streams (per-table) | -| Requires clustering | No | Yes | -| Available since | v4.1.0 | v4.1.0 | -| Stores original record values | Yes | No | -| Query by username | Yes | No | -| Query by primary key | Yes | No | -| Used for real-time messaging | Yes (required) | No | +Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log. + +| Feature | Audit Log | Transaction Log | +| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | +| Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | +| Requires clustering | No | Yes | +| Available since | v4.1.0 | v4.1.0 | +| Stores original record values | Yes | No | +| Query by username | Yes | No | +| Query by primary key | Yes | No | +| Used for real-time messaging | Yes (required) | No | ## Audit Log @@ -122,7 +122,7 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold - — This operation requires `table`, but on the RocksDB storage engine (the default) history cannot be deleted for a single table because all tables in a database share one transaction log. On RocksDB this operation now always returns an error directing you to `delete_transaction_logs_before`; it remains usable on LMDB. + — This operation is unsupported on the RocksDB storage engine (the default): it requires `table`, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to `delete_transaction_logs_before`; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB. ```json { @@ -137,15 +137,17 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo #### `delete_transaction_logs_before` -Deletes transaction log entries older than the specified timestamp. - + — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + +Deletes transaction log entries older than the specified timestamp. + On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. - — On RocksDB, a request that includes `table` is now rejected with a `400` error; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now returns a `404` (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. +`cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. -On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additionally removes leftover tombstone entries for records deleted before the timestamp — a repair step for tombstones that normal audit log cleanup should already have removed. It is ignored on RocksDB. +The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). ```json { @@ -155,6 +157,15 @@ On LMDB only, the optional `cleanup_deleted_records` (boolean) parameter additio } ``` +Response: + +```json +{ + "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69", + "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69" +} +``` + --- ## Enabling Audit Log Per Table From 5cc6c517e32ccc7fcd488b7169bb10be70abc0dc Mon Sep 17 00:00:00 2001 From: Chris Barber Date: Tue, 4 Aug 2026 12:28:10 -0500 Subject: [PATCH 5/5] Address review round 2: LMDB example gap, WAL clustering row, data-safety admonition, params/results, get_job error example, 5.2 release note Co-Authored-By: Claude Fable 5 --- reference/database/transaction.md | 45 +++++++++++++++++++++++++++---- release-notes/v5-lincoln/5.2.md | 10 +++++++ 2 files changed, 50 insertions(+), 5 deletions(-) diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 4c4b8d45..c8328366 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -16,7 +16,7 @@ Harper provides two complementary mechanisms for recording a history of data cha | Feature | Audit Log | Transaction Log | | ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | | Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | -| Requires clustering | No | Yes | +| Requires clustering | No | No (RocksDB, native WAL); Yes (LMDB, clustering streams) | | Available since | v4.1.0 | v4.1.0 | | Stores original record values | Yes | No | | Query by username | Yes | No | @@ -139,15 +139,26 @@ Deletes audit log entries older than the specified timestamp. Deprecated in favo - — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously a typo'd `table` fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. Deletes transaction log entries older than the specified timestamp. -On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. +:::warning Database-wide and irreversible +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This deletes whole log files, permanently removing [`read_audit_log`](#read_audit_log) history below the timestamp for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +::: -`cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. +Parameters: -The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). +- `database` (or the deprecated `schema` alias): `string` (required) — a request naming neither fails validation before a job starts; a `database` that does not exist fails the job with a not-found error. +- `timestamp`: `number` (required) — epoch milliseconds; entries older than this are deleted. +- `table`: `string` (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above). +- `cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. + +On LMDB, the table-scoped deletion scans the database's full audit history (and `cleanup_deleted_records: true` adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history. + +The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md#get-job) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). + +**RocksDB (database-wide — omit `table`):** ```json { @@ -157,6 +168,17 @@ The operation runs as a background job: the request itself returns `200` with a } ``` +**LMDB (`table` is required — omitting it deletes nothing and reports `entries_deleted: 0` with job status `COMPLETE`):** + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "table": "dog", + "timestamp": 1598290282817 +} +``` + Response: ```json @@ -166,6 +188,19 @@ Response: } ``` +`get_job` reports the outcome. A successful job's `result` carries `log_files_deleted` and `entries_deleted` (plus `start_timestamp`/`end_timestamp` on LMDB) — the record of how much was deleted. A rejected request looks like: + +```json +[ + { + "id": "2fe25039-566e-4670-8bb3-2db3d4e07e69", + "type": "delete_transaction_logs_before", + "status": "ERROR", + "message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'" + } +] +``` + --- ## Enabling Audit Log Per Table diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 1097504a..c28d4ea1 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -14,6 +14,16 @@ New managed-backup operations for RocksDB databases: `create_backup`, `list_back `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. +## Transaction Log Deletion + +`delete_transaction_logs_before` no longer accepts a `table` scope on RocksDB (the default storage engine): all tables in a database share one transaction log, and the previous behavior silently ignored `table` and purged the **entire database's** log while reporting success ([harper#2049](https://github.com/HarperFast/harper/issues/2049)). Three behavior changes may affect scheduled retention jobs on upgrade: + +- On RocksDB, a request naming a `table` now fails with an error directing you to the database-wide form (`database` + `timestamp` only). +- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now always fails with the same guidance; it remains usable on LMDB. +- On either engine, a `table` that does not exist now fails with a not-found error (previously a silent no-op on LMDB, and the database-wide purge on RocksDB). + +See [Transaction Logging](/reference/v5/database/transaction) for the updated operation reference. + ## Querying ### Filtered Vector Search (Predicate-Aware HNSW Traversal)