diff --git a/reference/database/transaction.md b/reference/database/transaction.md index 7af4f925..c8328366 100644 --- a/reference/database/transaction.md +++ b/reference/database/transaction.md @@ -11,17 +11,17 @@ title: Transaction Logging # Transaction Logging -Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases. - -| Feature | Audit Log | Transaction Log | -| ----------------------------- | --------------------------------- | ------------------------------ | -| Storage | Standard Harper table (per-table) | Clustering streams (per-table) | -| Requires clustering | No | Yes | -| Available since | v4.1.0 | v4.1.0 | -| Stores original record values | Yes | No | -| Query by username | Yes | No | -| Query by primary key | Yes | No | -| Used for real-time messaging | Yes (required) | No | +Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log. + +| Feature | Audit Log | Transaction Log | +| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- | +| Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) | +| Requires clustering | No | No (RocksDB, native WAL); Yes (LMDB, clustering streams) | +| Available since | v4.1.0 | v4.1.0 | +| Stores original record values | Yes | No | +| Query by username | Yes | No | +| Query by primary key | Yes | No | +| Used for real-time messaging | Yes (required) | No | ## Audit Log @@ -116,12 +116,14 @@ The `original_records` field contains the record state before the operation was #### `delete_audit_logs_before` -Deletes audit log entries older than the specified timestamp. +Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before). — Audit log cleanup improved to reduce resource consumption during scheduled cleanups — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold + — This operation is unsupported on the RocksDB storage engine (the default): it requires `table`, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to `delete_transaction_logs_before`; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB. + ```json { "operation": "delete_audit_logs_before", @@ -131,6 +133,74 @@ Deletes audit log entries older than the specified timestamp. } ``` +### Transaction Log Operations + +#### `delete_transaction_logs_before` + + + + — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously a typo'd `table` fell through to the database-wide purge on RocksDB, and was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged. + +Deletes transaction log entries older than the specified timestamp. + +:::warning Database-wide and irreversible +On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`. This deletes whole log files, permanently removing [`read_audit_log`](#read_audit_log) history below the timestamp for **every table in the database** — there is no per-table survivor and no undo. The only recovery route is a [backup](../backups/overview.md), which restores the transaction log alongside the data. Purging below a lagging replica's catch-up position does not lose data on that replica — the sender detects that the requested start predates its retained history and forces a full base copy instead of incremental catch-up — but that full resync is far more expensive than incremental replication, so avoid purging below your slowest replica's position. +::: + +Parameters: + +- `database` (or the deprecated `schema` alias): `string` (required) — a request naming neither fails validation before a job starts; a `database` that does not exist fails the job with a not-found error. +- `timestamp`: `number` (required) — epoch milliseconds; entries older than this are deleted. +- `table`: `string` (LMDB only) — scopes deletion to that table's history. On RocksDB the job fails (see the warning above). +- `cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB. + +On LMDB, the table-scoped deletion scans the database's full audit history (and `cleanup_deleted_records: true` adds a second full scan of the table's records), so the cost grows with total history depth — schedule accordingly on databases with deep audit history. + +The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md#get-job) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection). + +**RocksDB (database-wide — omit `table`):** + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "timestamp": 1598290282817 +} +``` + +**LMDB (`table` is required — omitting it deletes nothing and reports `entries_deleted: 0` with job status `COMPLETE`):** + +```json +{ + "operation": "delete_transaction_logs_before", + "database": "dev", + "table": "dog", + "timestamp": 1598290282817 +} +``` + +Response: + +```json +{ + "message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69", + "job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69" +} +``` + +`get_job` reports the outcome. A successful job's `result` carries `log_files_deleted` and `entries_deleted` (plus `start_timestamp`/`end_timestamp` on LMDB) — the record of how much was deleted. A rejected request looks like: + +```json +[ + { + "id": "2fe25039-566e-4670-8bb3-2db3d4e07e69", + "type": "delete_transaction_logs_before", + "status": "ERROR", + "message": "There was an error running deleteTransactionLogsBefore job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69 - Table-level transaction log deletion is not supported for RocksDB tables because all tables in a database share one transaction log; to delete the transaction logs for the entire 'dev' database, use delete_transaction_logs_before with only 'database' and 'timestamp'" + } +] +``` + --- ## Enabling Audit Log Per Table diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 1097504a..c28d4ea1 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -14,6 +14,16 @@ New managed-backup operations for RocksDB databases: `create_backup`, `list_back `get_backup` now understands RocksDB: it streams a full-snapshot `tar` of the database — including any file-backed blobs (pass `exclude_blobs: true` to omit them), gzipped by default (pass `gzip: false` for a plain `tar`) — instead of failing. The LMDB behavior (streaming the `.mdb`, with `table`/`tables`/`include_audit`) is unchanged. +## Transaction Log Deletion + +`delete_transaction_logs_before` no longer accepts a `table` scope on RocksDB (the default storage engine): all tables in a database share one transaction log, and the previous behavior silently ignored `table` and purged the **entire database's** log while reporting success ([harper#2049](https://github.com/HarperFast/harper/issues/2049)). Three behavior changes may affect scheduled retention jobs on upgrade: + +- On RocksDB, a request naming a `table` now fails with an error directing you to the database-wide form (`database` + `timestamp` only). +- The deprecated `delete_audit_logs_before` operation requires `table`, so on RocksDB it now always fails with the same guidance; it remains usable on LMDB. +- On either engine, a `table` that does not exist now fails with a not-found error (previously a silent no-op on LMDB, and the database-wide purge on RocksDB). + +See [Transaction Logging](/reference/v5/database/transaction) for the updated operation reference. + ## Querying ### Filtered Vector Search (Predicate-Aware HNSW Traversal)