From 38b14bb989e4abd4d21216361929004a8b361047 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 09:18:03 -0600 Subject: [PATCH 1/8] docs(analytics): document transaction-commit-time metric Add a Storage Metrics section documenting transaction-commit-time (write commit submit->durable duration distribution) and how its upper percentiles serve as the leading indicator for the storage overload (503) rejection. Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/analytics/overview.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6..d9549cc4 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -169,6 +169,25 @@ Harper automatically tracks the following metrics for all services. Applications | `bytes-received` | node.database | `replication` | `blob` | bytes | Bytes received for blob replication | | `replication-latency` | node.database.table | | `ingest` | ms | Time difference from source commit timestamp to local time | +### Storage Metrics + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | + +`transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, +`p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard +uses: when the oldest outstanding commit exceeds `storage.maxTransactionQueueTime` (default 45s), +Harper rejects new write transactions with `Outstanding write transactions have too long of queue, +please try again later` (HTTP 503). + +Because it is measured on that same clock, it is the leading indicator for that rejection: a rising +`p99`/`p999` means commits are taking longer to drain — from write volume, large transactions, or a +saturated storage volume — and is the signal to shed or throttle write load before commits start +timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared +write path rather than per-thread submission concurrency, so alert on the upper percentiles trending +toward `maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. + ### Resource Usage Metrics | `metric` | Key attributes | Other | Unit | Description | From 340817646b104543dda30f97f25a428e65f93eb0 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 10:29:14 -0600 Subject: [PATCH 2/8] fix(analytics): satisfy prettier table formatting --- reference/analytics/overview.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index d9549cc4..69e2bb29 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,8 +171,8 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | --------------------------------------------------- | +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------- | | `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, From daf8ab9d2fd206a66ce09e87ac6ac0ab0eb1a234 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:38:46 -0600 Subject: [PATCH 3/8] docs(analytics): address PR #572 review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Correct transaction-commit-time description: the recorder fires on both fulfilment and rejection (submit -> settle), not only on durable commit — a failed/retried commit still consumes queue time and should not be undercounted. - Restore the storage. prefix on maxTransactionQueueTime for consistency with its earlier mention. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 69e2bb29..fcb982e7 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,9 +171,9 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------- | -| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → durable commit) | +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------------------- | +| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → settle, success or failure) | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, `p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard @@ -186,7 +186,7 @@ Because it is measured on that same clock, it is the leading indicator for that saturated storage volume — and is the signal to shed or throttle write load before commits start timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared write path rather than per-thread submission concurrency, so alert on the upper percentiles trending -toward `maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. +toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics From 1771f0afbe4032c34fa87f4fb8ff0ff34aec9a75 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:49:53 -0600 Subject: [PATCH 4/8] docs(analytics): qualify transaction-commit-time as RocksDB-only, settle-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (codex) flagged the original framing as overstating the metric's reliability as a leading indicator: it is RocksDB-only, records one sample per commit attempt (not per logical transaction — conflict retries each get their own sample), and only records once an attempt settles, so a wedged commit contributes nothing while still tripping the 503 guard. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 35 ++++++++++++++++++++------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index fcb982e7..41eba706 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,22 +171,29 @@ Harper automatically tracks the following metrics for all services. Applications ### Storage Metrics -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ----------------------------------------------------------------------------- | -| `transaction-commit-time` | | | | ms | Time a write transaction is outstanding (submit → settle, success or failure) | + + +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------ | +| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit → settle, per commit attempt | `transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, -`p999`, `count`) per write commit. It measures the same interval the storage engine's overload guard -uses: when the oldest outstanding commit exceeds `storage.maxTransactionQueueTime` (default 45s), -Harper rejects new write transactions with `Outstanding write transactions have too long of queue, -please try again later` (HTTP 503). - -Because it is measured on that same clock, it is the leading indicator for that rejection: a rising -`p99`/`p999` means commits are taking longer to drain — from write volume, large transactions, or a -saturated storage volume — and is the signal to shed or throttle write load before commits start -timing out. Unlike an in-flight transaction count, this reflects transaction size and the shared -write path rather than per-thread submission concurrency, so alert on the upper percentiles trending -toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. +`p999`, `count`) on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each +sample covers one commit attempt, not one logical write transaction — a transient-conflict retry +re-issues the commit and records its own sample, so `count` can exceed the number of logical writes — +and a sample is only recorded once an attempt settles, so a commit that is still outstanding +contributes nothing yet. It measures the same clock the storage engine's overload guard uses: when +the oldest outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default 45s), +Harper rejects new write transactions on that thread with `Outstanding write transactions have too +long of queue, please try again later` (HTTP 503). + +A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large +transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write +load. But because it only records on settle, a wedged commit — the case the overload guard exists to +catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this +distribution as a leading indicator on its own, pair it with queue-depth or stuck-commit observability +before alerting solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the +threshold against a baseline for your workload. ### Resource Usage Metrics From f2fd67a35e960b14620111b1b914807d5490c327 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:57:33 -0600 Subject: [PATCH 5/8] docs(analytics): fix raw/aggregate field split and overload-guard wording - median/p90/p95/p99/p999 are only present on the per-minute aggregate (hdb_analytics); raw per-second entries only carry mean/distribution/count. - The overload guard tracks a commit it happened to arm on, not reliably the oldest outstanding one (retries/chained commits can wedge unarmed). - ASCII "to" instead of the arrow glyph per the content style guide. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 28 +++++++++++++++------------- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 41eba706..01eb938c 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -173,19 +173,21 @@ Harper automatically tracks the following metrics for all services. Applications -| `metric` | `path` | `method` | `type` | Unit | Description | -| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------ | -| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit → settle, per commit attempt | - -`transaction-commit-time` is recorded as a distribution (`mean`, `median`, `p90`, `p95`, `p99`, -`p999`, `count`) on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each -sample covers one commit attempt, not one logical write transaction — a transient-conflict retry -re-issues the commit and records its own sample, so `count` can exceed the number of logical writes — -and a sample is only recorded once an attempt settles, so a commit that is still outstanding -contributes nothing yet. It measures the same clock the storage engine's overload guard uses: when -the oldest outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default 45s), -Harper rejects new write transactions on that thread with `Outstanding write transactions have too -long of queue, please try again later` (HTTP 503). +| `metric` | `path` | `method` | `type` | Unit | Description | +| ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- | +| `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt | + +`transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for +LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a +transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the +number of logical writes — and a sample is only recorded once an attempt settles, so a commit that is +still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, +`distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on +the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table +for percentile-based alerting. It measures the same clock the storage engine's overload guard uses: +when a tracked outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default +45s), Harper rejects new write transactions on that thread with `Outstanding write transactions have +too long of queue, please try again later` (HTTP 503). A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write From c2abfafa3ee2a93dc11df9f8495ccd4b3704f563 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:04:33 -0600 Subject: [PATCH 6/8] docs(analytics): add 5.2 release note, point to the real stuck-commit signal - Add a transaction-commit-time entry to the v5.2 release notes so it's discoverable from the changelog. - Replace the vague "queue-depth or stuck-commit observability" pointer (no such metric is documented yet) with the actual overload-guard log line operators can watch today. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 5 +++-- release-notes/v5-lincoln/5.2.md | 6 ++++++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 01eb938c..43a598b0 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -193,8 +193,9 @@ A rising `p99`/`p999` signals commits are taking longer to drain — from write transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write load. But because it only records on settle, a wedged commit — the case the overload guard exists to catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this -distribution as a leading indicator on its own, pair it with queue-depth or stuck-commit observability -before alerting solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the +distribution as a leading indicator on its own. Watch the server log for the "Rejecting writes on this +thread" error the guard emits once it trips, and don't rely solely on percentiles trending toward +`storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 365ddf38..665e2a09 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -13,3 +13,9 @@ All patch release notes for 5.2.x are available on the [releases page](https://g ### Replicated `set_configuration` The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration). + +## Analytics + +### `transaction-commit-time` metric + +A new `transaction-commit-time` storage metric records the submit-to-settle duration of RocksDB write commits, giving operators a distribution to watch alongside the `storage.maxTransactionQueueTime` overload guard — see [Storage Metrics](/reference/v5/analytics/overview#storage-metrics). From d90edac819c3e7f25b1869c73fbaad2319db6f52 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:45:17 -0600 Subject: [PATCH 7/8] docs(analytics): fix accuracy issues found by independent review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cross-model review (codex + grok + harper-domain adjudication) found the initial framing overstated this metric's reliability as a leading indicator. Verified each claim against harper's DatabaseTransaction.ts and analytics/write.ts before applying: - RocksDB-only; never emitted for LMDB-backed databases. - One sample per commit attempt, not per logical transaction (conflict retries each record their own sample). - Only records once an attempt settles — a still-outstanding commit contributes nothing. - Raw entries (hdb_raw_analytics) carry mean/distribution/count only; percentiles are computed on the per-minute aggregate (hdb_analytics). Fixed the page's own raw-entry JSON example, which showed percentile fields that don't exist on raw rows. - The overload guard rejects record updates and publishes only — deletes and canonical-source writes (replication, caching sources) bypass it. - The guard tracks at most one outstanding commit per thread: a retry issued while the prior attempt still holds that slot is never armed (so a wedge there won't trip the 503); a later backoff-delayed retry re-arms fresh. - Added a v5.2 release-note entry and VersionBadge. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 29 ++++++++++++++++------------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 43a598b0..c9abeeb5 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -55,10 +55,8 @@ Example raw entry: "metric": "bytes-sent", "path": "search_by_conditions", "type": "operation", - "median": 202, "mean": 202, - "p95": 202, - "p90": 202, + "distribution": [202], "count": 1 }, { @@ -180,22 +178,27 @@ Harper automatically tracks the following metrics for all services. Applications `transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the -number of logical writes — and a sample is only recorded once an attempt settles, so a commit that is +number of logical writes. A sample is only recorded once an attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`, `distribution`, and `count`; percentiles (`median`, `p90`, `p95`, `p99`, `p999`) are only available on the per-minute aggregate (`hdb_analytics`) once raw entries are rolled up — query the aggregate table -for percentile-based alerting. It measures the same clock the storage engine's overload guard uses: -when a tracked outstanding commit on a thread exceeds `storage.maxTransactionQueueTime` (default -45s), Harper rejects new write transactions on that thread with `Outstanding write transactions have -too long of queue, please try again later` (HTTP 503). +for percentile-based alerting. It shares a timebase with the RocksDB storage engine's overload guard, +which times only the commit attempt it arms: when a tracked outstanding commit on a thread exceeds +`storage.maxTransactionQueueTime` (default 45s), Harper rejects new record updates and publishes on +that thread with `Outstanding write transactions have too long of queue, please try again later` +(HTTP 503) — deletes and writes applied from a canonical source (e.g. replication or a caching +source) bypass this check. The guard tracks at most one outstanding commit per thread: a retry +issued while the prior attempt still holds that slot (a coordinated retry, or an early backoff +retry) recommits before the slot clears and is never armed, so a wedge there won't trip the 503; a +later, backoff-delayed retry recommits after the slot clears and is tracked like a fresh attempt. A rising `p99`/`p999` signals commits are taking longer to drain — from write volume, large transactions, or a saturated storage volume — and is a useful early warning to shed or throttle write -load. But because it only records on settle, a wedged commit — the case the overload guard exists to -catch — can sit outstanding and about to trip the 503 while contributing no sample; don't treat this -distribution as a leading indicator on its own. Watch the server log for the "Rejecting writes on this -thread" error the guard emits once it trips, and don't rely solely on percentiles trending toward -`storage.maxTransactionQueueTime`. Tune the +load. But the metric shares only a timebase with the guard, not its population: a wedged commit +contributes no sample until it settles, and an early retry that wedges can go untracked by the guard +entirely (see above). Don't treat this distribution as a leading indicator on its own — watch the +server log for the "Rejecting writes on this thread" error the guard emits when it does trip, and +don't rely solely on percentiles trending toward `storage.maxTransactionQueueTime`. Tune the threshold against a baseline for your workload. ### Resource Usage Metrics From a4cca363c762a6f5013972cb0ac32462c7d21457 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 19:55:36 -0600 Subject: [PATCH 8/8] docs(analytics): note the metric is absent during transaction-log replay recordCommitLatency wraps the async transaction.commit() path only; replayLogs.ts's directCommitSync() calls commitSync() directly and isn't instrumented, so replay/startup commits emit no sample either. Co-Authored-By: Claude Opus --- reference/analytics/overview.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c9abeeb5..de585db9 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -175,8 +175,9 @@ Harper automatically tracks the following metrics for all services. Applications | ------------------------- | ------ | -------- | ------ | ---- | ------------------------------------------------------------------- | | `transaction-commit-time` | | | | ms | RocksDB write-commit duration, submit to settle, per commit attempt | -`transaction-commit-time` is recorded on the RocksDB write path only; it is not emitted for -LMDB-backed databases. Each sample covers one commit attempt, not one logical write transaction — a +`transaction-commit-time` is recorded on the RocksDB asynchronous commit path only; it is not +emitted for LMDB-backed databases, and not for the synchronous `commitSync()` path used during +transaction-log replay. Each sample covers one commit attempt, not one logical write transaction — a transient-conflict retry re-issues the commit and records its own sample, so `count` can exceed the number of logical writes. A sample is only recorded once an attempt settles, so a commit that is still outstanding contributes nothing yet. Raw entries (`hdb_raw_analytics`) carry `mean`,