Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 47 additions & 12 deletions reference/database/transaction.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,17 @@ title: Transaction Logging

# Transaction Logging

Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both are available at the table level and serve different use cases.

| Feature | Audit Log | Transaction Log |
| ----------------------------- | --------------------------------- | ------------------------------ |
| Storage | Standard Harper table (per-table) | Clustering streams (per-table) |
| Requires clustering | No | Yes |
| Available since | v4.1.0 | v4.1.0 |
| Stores original record values | Yes | No |
| Query by username | Yes | No |
| Query by primary key | Yes | No |
| Used for real-time messaging | Yes (required) | No |
Harper provides two complementary mechanisms for recording a history of data changes on a table: the **audit log** and the **transaction log**. Both record and expose history for individual tables, but their underlying storage differs: on RocksDB (the default storage engine) the transaction log is physically a single database-wide log shared by all tables — history is read per table, while deletion operates on the whole database's log.

| Feature | Audit Log | Transaction Log |
| ----------------------------- | --------------------------------- | ----------------------------------------------------------------------- |
| Storage | Standard Harper table (per-table) | Shared per-database log (RocksDB); clustering streams, per-table (LMDB) |
| Requires clustering | No | Yes |
| Available since | v4.1.0 | v4.1.0 |
| Stores original record values | Yes | No |
| Query by username | Yes | No |
| Query by primary key | Yes | No |
| Used for real-time messaging | Yes (required) | No |

## Audit Log

Expand Down Expand Up @@ -116,12 +116,14 @@ The `original_records` field contains the record state before the operation was

#### `delete_audit_logs_before`

Deletes audit log entries older than the specified timestamp.
Deletes audit log entries older than the specified timestamp. Deprecated in favor of [`delete_transaction_logs_before`](#delete_transaction_logs_before).

<VersionBadge type="changed" version="v4.3.0" /> — Audit log cleanup improved to reduce resource consumption during scheduled cleanups

<VersionBadge type="changed" version="v4.5.0" /> — Storage reclamation: Harper automatically evicts older audit log entries when free storage drops below a configurable threshold

<VersionBadge type="changed" version="v5.2.0" /> — This operation is unsupported on the RocksDB storage engine (the default): it requires `table`, but history cannot be deleted for a single table because all tables in a database share one transaction log. For an existing table the job fails with an error directing you to `delete_transaction_logs_before`; for a nonexistent table the job fails with a not-found error. The operation remains usable on LMDB.

```json
{
"operation": "delete_audit_logs_before",
Expand All @@ -131,6 +133,39 @@ Deletes audit log entries older than the specified timestamp.
}
```

Comment thread
cb1kenobi marked this conversation as resolved.
### Transaction Log Operations

#### `delete_transaction_logs_before`

<EngineBadge engines="RocksDB, LMDB" />

<VersionBadge type="changed" version="v5.2.0" /> — On RocksDB, a request that includes `table` now fails; previously the `table` scope was silently ignored and the entire database's transaction log was purged. On either engine, a `table` that does not exist now fails with a not-found error (previously this was a silent no-op on LMDB). On LMDB, a valid `table` continues to scope the deletion to that table's history, unchanged.

Deletes transaction log entries older than the specified timestamp.

On RocksDB (the default storage engine), deletion is database-wide: all tables in a database share one transaction log, so omit `table` and pass only `database` (or `schema`) and `timestamp`.
Comment thread
cb1kenobi marked this conversation as resolved.

`cleanup_deleted_records`: `boolean` (optional) — LMDB only; additionally removes leftover tombstone entries for records deleted before the timestamp, a repair step for tombstones that normal audit log cleanup should already have removed. Ignored on RocksDB.

The operation runs as a background job: the request itself returns `200` with a job ID, and any rejection surfaces when the job runs. Poll [`get_job`](jobs.md) to observe the outcome — a rejected request ends with job status `ERROR` and a message describing the failure (for example, the RocksDB table-scope rejection).

```json
{
"operation": "delete_transaction_logs_before",
Comment thread
cb1kenobi marked this conversation as resolved.
"database": "dev",
"timestamp": 1598290282817
}
```

Response:

```json
{
"message": "Starting job with id 2fe25039-566e-4670-8bb3-2db3d4e07e69",
"job_id": "2fe25039-566e-4670-8bb3-2db3d4e07e69"
}
```

---

## Enabling Audit Log Per Table
Expand Down