From 169c5d9eb300a31ee1d722f687acfa827a7b1ac7 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Sun, 5 Jul 2026 22:43:12 -0600 Subject: [PATCH 1/7] docs(analytics): document transaction queue depth metrics Add write-transaction-queue-depth and read-transaction-queue-depth to the Resource Usage metrics reference, with a section explaining the depth/maxDepth fields and alerting guidance (alert on the per-period maxDepth peak). Companion to harper#592. Co-Authored-By: Claude Opus 4.8 (1M context) --- reference/analytics/overview.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 5a6309b6..39fad3b0 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -175,10 +175,40 @@ Harper automatically tracks the following metrics for all services. Applications | ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | | `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | | `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | | `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | | `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | + +#### Transaction Queue Depth Metrics + +`write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are +in flight against the storage engine, giving operators a leading indicator before a write-heavy +workload hits the `Outstanding write transactions have too long of queue, please try again later` +(HTTP 503) rejection. + +| Field | Unit | Description | +| ---------- | ----- | -------------------------------------------------------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark observed over the sampling period | + +- **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet + durably committed — the backlog that produces the overload error when it drains too slowly. +- **`read-transaction-queue-depth`** counts open read (snapshot) transactions. Persistently high read + depth indicates long-lived read snapshots, which can hold back compaction. + +Both are gauges sampled per worker thread and summed across threads in the aggregate table (the raw +per-thread entries in `hdb_raw_analytics` retain each thread's own depth). Note that the aggregate +`maxDepth` is the sum of each thread's peak, which can read higher than any true simultaneous global +peak since per-thread spikes need not coincide — treat it as an upper bound, not an exact concurrent +queue length. Because the write queue can fill and drain within a single sampling period, always +alert on `maxDepth` (the per-period peak) rather than `depth` alone — an instantaneous sample will +routinely read low even while short spikes are occurring. A healthy system keeps `write-transaction-queue-depth.maxDepth` +near zero; a sustained non-zero peak that trends upward is the signal to shed or throttle write load +before commits start timing out. Tune the concrete alert threshold against a baseline for your +workload, since absolute depth scales with worker-thread count and per-transaction size. #### `resource-usage` Metric From e99504f6d4b315c4d8710866944d519365f514be Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Tue, 7 Jul 2026 10:29:20 -0600 Subject: [PATCH 2/7] fix(analytics): satisfy prettier table formatting --- reference/analytics/overview.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 39fad3b0..3dc19060 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,16 +171,16 @@ Harper automatically tracks the following metrics for all services. Applications ### Resource Usage Metrics -| `metric` | Key attributes | Other | Unit | Description | -| ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------- | -| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | -| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | -| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | -| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | -| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | -| `table-size` | `size` | `database`, `table` | bytes | Table file size | -| `utilization` | | | % | Percentage of time the worker thread was processing requests | -| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `metric` | Key attributes | Other | Unit | Description | +| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | ----------------------------------------------------------------------------------------------------- | +| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | +| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | +| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | +| `table-size` | `size` | `database`, `table` | bytes | Table file size | +| `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | #### Transaction Queue Depth Metrics @@ -189,10 +189,10 @@ in flight against the storage engine, giving operators a leading indicator befor workload hits the `Outstanding write transactions have too long of queue, please try again later` (HTTP 503) rejection. -| Field | Unit | Description | -| ---------- | ----- | -------------------------------------------------------------------------------------------------- | -| `depth` | count | Instantaneous depth sampled at emit time | -| `maxDepth` | count | High-water mark observed over the sampling period | +| Field | Unit | Description | +| ---------- | ----- | ------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark observed over the sampling period | - **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet durably committed — the backlog that produces the overload error when it drains too slowly. From 8ddf9ff15c7096e7df43a82665fb28c111bcfbd5 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:06:29 -0600 Subject: [PATCH 3/7] fix(analytics): correct transaction queue depth documentation Address review feedback on PR #573: fix hyphenation inconsistency, remove the overclaim that write-transaction-queue-depth predicts the write-queue 503 rejection (that check is duration-based on a single outstanding commit, not a function of concurrent commit count), and add the missing caveat that both queue-depth metrics are RocksDB-only and always read zero on LMDB-backed databases. Co-Authored-By: Claude Sonnet 5 --- reference/analytics/overview.md | 26 ++++++++++++++++++-------- 1 file changed, 18 insertions(+), 8 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 3dc19060..ad01c4fd 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -180,14 +180,17 @@ Harper automatically tracks the following metrics for all services. Applications | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | | `utilization` | | | % | Percentage of time the worker thread was processing requests | -| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write-transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | #### Transaction Queue Depth Metrics `write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are -in flight against the storage engine, giving operators a leading indicator before a write-heavy -workload hits the `Outstanding write transactions have too long of queue, please try again later` -(HTTP 503) rejection. +in flight against the storage engine per worker thread — a concurrency and throughput signal, not a +predictor of the `Outstanding write transactions have too long of queue, please try again later` +(HTTP 503) rejection. That rejection is a duration check on a single outstanding commit +(`storage.maxTransactionQueueTime`, default 45s — see [Storage](../configuration/options.md#storage)), +not a function of concurrent commit count: a thread can carry a high `maxDepth` with fast commits and +never trip it, or sit at `depth` of 1 for the full timeout and trip it while this metric barely moves. | Field | Unit | Description | | ---------- | ----- | ------------------------------------------------- | @@ -195,10 +198,15 @@ workload hits the `Outstanding write transactions have too long of queue, please | `maxDepth` | count | High-water mark observed over the sampling period | - **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet - durably committed — the backlog that produces the overload error when it drains too slowly. + durably committed on this thread — how many commits it's juggling concurrently, not how close any + one of them is to the 503 timeout. - **`read-transaction-queue-depth`** counts open read (snapshot) transactions. Persistently high read depth indicates long-lived read snapshots, which can hold back compaction. +Both metrics are tracked only on the RocksDB write/read path. On an LMDB-backed database +(`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` — indistinguishable +from a healthy, empty queue — regardless of actual read/write load. + Both are gauges sampled per worker thread and summed across threads in the aggregate table (the raw per-thread entries in `hdb_raw_analytics` retain each thread's own depth). Note that the aggregate `maxDepth` is the sum of each thread's peak, which can read higher than any true simultaneous global @@ -206,9 +214,11 @@ peak since per-thread spikes need not coincide — treat it as an upper bound, n queue length. Because the write queue can fill and drain within a single sampling period, always alert on `maxDepth` (the per-period peak) rather than `depth` alone — an instantaneous sample will routinely read low even while short spikes are occurring. A healthy system keeps `write-transaction-queue-depth.maxDepth` -near zero; a sustained non-zero peak that trends upward is the signal to shed or throttle write load -before commits start timing out. Tune the concrete alert threshold against a baseline for your -workload, since absolute depth scales with worker-thread count and per-transaction size. +near zero; a sustained non-zero peak that trends upward indicates growing write concurrency worth +investigating, but treat it as a throughput signal rather than an early warning for the 503 — that +rejection depends on a single commit's duration, not on how many commits are queued. Tune the +concrete alert threshold against a baseline for your workload, since absolute depth scales with +worker-thread count and per-transaction size. #### `resource-usage` Metric From 14ebc9e9d979cab792c9ac154b08ae8efd600c19 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:19:24 -0600 Subject: [PATCH 4/7] fix(analytics): correct aggregation semantics and doc conventions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independent pre-push review (codex + harper-domain) verified against resources/analytics/write.ts:897-956 that the aggregate hdb_analytics maxDepth is a sum of per-thread period averages, not per-thread peaks as previously documented — the doc's spike-detection advice pointed at the wrong table. Also: add the required v5.2.0 version badge and release-notes entry, soften the write/read depth semantics (settled commit promise != durable under storage.writeAsync; open snapshot count can't distinguish long-lived from many short-lived), and link storage.maxTransactionQueueTime to its documented section instead of a generic options list. Co-Authored-By: Claude Sonnet 5 --- reference/analytics/overview.md | 47 +++++++++++++++++++-------------- release-notes/v5-lincoln/5.2.md | 6 +++++ 2 files changed, 33 insertions(+), 20 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index ad01c4fd..c4d503c9 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -184,41 +184,48 @@ Harper automatically tracks the following metrics for all services. Applications #### Transaction Queue Depth Metrics + + `write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are in flight against the storage engine per worker thread — a concurrency and throughput signal, not a predictor of the `Outstanding write transactions have too long of queue, please try again later` (HTTP 503) rejection. That rejection is a duration check on a single outstanding commit -(`storage.maxTransactionQueueTime`, default 45s — see [Storage](../configuration/options.md#storage)), -not a function of concurrent commit count: a thread can carry a high `maxDepth` with fast commits and -never trip it, or sit at `depth` of 1 for the full timeout and trip it while this metric barely moves. +(see [`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime), +default 45s), not a function of concurrent commit count: a thread can carry a high `maxDepth` with +fast commits and never trip it, or sit at `depth` of 1 for the full timeout and trip it while this +metric barely moves. | Field | Unit | Description | | ---------- | ----- | ------------------------------------------------- | | `depth` | count | Instantaneous depth sampled at emit time | | `maxDepth` | count | High-water mark observed over the sampling period | -- **`write-transaction-queue-depth`** counts write commits handed to the storage engine but not yet - durably committed on this thread — how many commits it's juggling concurrently, not how close any - one of them is to the 503 timeout. -- **`read-transaction-queue-depth`** counts open read (snapshot) transactions. Persistently high read - depth indicates long-lived read snapshots, which can hold back compaction. +- **`write-transaction-queue-depth`** counts write commits handed to the storage engine whose commit + promises have not yet settled — how many commits this thread is juggling concurrently, not how + close any one of them is to the 503 timeout. This is in-flight, not durability: under + `storage.writeAsync: true` a settled commit promise does not guarantee the write has been synced + to disk. +- **`read-transaction-queue-depth`** counts concurrently open (snapshot) transactions. A high count + can mean either many short-lived reads or a few long-lived ones — the count alone can't + distinguish them, so use it as a concurrency signal; a metric with duration would be needed to spot + a single snapshot held open long enough to hold back compaction. Both metrics are tracked only on the RocksDB write/read path. On an LMDB-backed database (`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` — indistinguishable from a healthy, empty queue — regardless of actual read/write load. -Both are gauges sampled per worker thread and summed across threads in the aggregate table (the raw -per-thread entries in `hdb_raw_analytics` retain each thread's own depth). Note that the aggregate -`maxDepth` is the sum of each thread's peak, which can read higher than any true simultaneous global -peak since per-thread spikes need not coincide — treat it as an upper bound, not an exact concurrent -queue length. Because the write queue can fill and drain within a single sampling period, always -alert on `maxDepth` (the per-period peak) rather than `depth` alone — an instantaneous sample will -routinely read low even while short spikes are occurring. A healthy system keeps `write-transaction-queue-depth.maxDepth` -near zero; a sustained non-zero peak that trends upward indicates growing write concurrency worth -investigating, but treat it as a throughput signal rather than an early warning for the 503 — that -rejection depends on a single commit's duration, not on how many commits are queued. Tune the -concrete alert threshold against a baseline for your workload, since absolute depth scales with -worker-thread count and per-transaction size. +Both are gauges sampled per worker thread. The raw per-thread entries in `hdb_raw_analytics` retain +each thread's true instantaneous `depth` and per-period `maxDepth`; treat those as the reliable +source for spike detection. The aggregate `hdb_analytics` table is not a sum of per-thread peaks — +each thread's `maxDepth` is first averaged across its raw samples for the period, then those +per-thread averages are summed — so a brief single-thread spike is diluted rather than preserved. +Always alert on `hdb_raw_analytics.maxDepth` (or lower the sampling/aggregation period) rather than +relying on the aggregate table to catch short spikes. A healthy system keeps +`write-transaction-queue-depth.maxDepth` near zero; a sustained non-zero peak that trends upward +indicates growing write concurrency worth investigating, but treat it as a throughput signal rather +than an early warning for the 503 — that rejection depends on a single commit's duration, not on how +many commits are queued. Tune the concrete alert threshold against a baseline for your workload, +since absolute depth scales with worker-thread count and per-transaction size. #### `resource-usage` Metric diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index dde9cb59..88dd3363 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -26,6 +26,12 @@ Components can now declare recurring jobs in their configuration with a new buil The `set_configuration` operation now accepts `"replicated": true` to apply a configuration change to all cluster nodes in a single Operations API call, with per-node outcomes reported in the response's `replicated` array. Only cluster-appropriate parameters should be replicated — see [Configuration Operations](/reference/v5/configuration/operations#set-configuration). +## Analytics + +### Transaction queue depth metrics + +New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread commit and read-snapshot concurrency against the storage engine, as a throughput/concurrency signal — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). + ## CLI ### Explicit Authentication for Operations API Commands From d8a0dc6b8ce2a37c58b289a28fcbc330fc5462ba Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:39:13 -0600 Subject: [PATCH 5/7] fix(analytics): document transaction-commit-time as the 503 leading indicator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Independent review (codex + grok + harper-domain, unanimous across three lenses) flagged that the doc named the 503 rejection four times while omitting the metric that actually predicts it. Verified against resources/analytics/metadata.ts and DatabaseTransaction.ts: enterWriteQueue/ leaveWriteQueue and the outstandingCommit arm/clear share the same commitResolution promise, so a wedged commit pins write-transaction-queue-depth at >=1 for its whole duration — the opposite of what the prior wording claimed. Add the transaction-commit-time metric row and point to it as the direct duration signal; correct the depth/duration relationship; soften the read-depth "(snapshot)" wording (disableSnapshot reads are counted too); and fix storage-tuning.md's maxTransactionQueueTime description, which read as a queue-length threshold rather than the per-commit duration check it is. Co-Authored-By: Claude Sonnet 5 --- reference/analytics/overview.md | 63 ++++++++++++++-------------- reference/database/storage-tuning.md | 2 +- release-notes/v5-lincoln/5.2.md | 2 +- 3 files changed, 34 insertions(+), 33 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index c4d503c9..56579c50 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -171,29 +171,33 @@ Harper automatically tracks the following metrics for all services. Applications ### Resource Usage Metrics -| `metric` | Key attributes | Other | Unit | Description | -| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | ----------------------------------------------------------------------------------------------------- | -| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | -| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | -| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open read (snapshot) transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | -| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | -| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | -| `table-size` | `size` | `database`, `table` | bytes | Table file size | -| `utilization` | | | % | Percentage of time the worker thread was processing requests | -| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `metric` | Key attributes | Other | Unit | Description | +| ------------------------------- | ------------------------------------------------------------------------------------------------ | ------------------- | ------- | --------------------------------------------------------------------------------------------------------------------- | +| `database-size` | `size`, `used`, `free`, `audit` | `database` | bytes | Database file size breakdown | +| `main-thread-utilization` | `idle`, `active`, `taskQueueLatency`, `rss`, `heapTotal`, `heapUsed`, `external`, `arrayBuffers` | `time` | various | Main thread resource usage: idle/active time, queue latency, and memory breakdown | +| `read-transaction-queue-depth` | `depth`, `maxDepth` | | count | Open tracked read transactions (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | +| `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | +| `table-size` | `size` | `database`, `table` | bytes | Table file size | +| `transaction-commit-time` | `median`, `mean`, `p95`, `p90` | | ms | Duration from write commit submission to settlement (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `utilization` | | | % | Percentage of time the worker thread was processing requests | +| `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | #### Transaction Queue Depth Metrics `write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are -in flight against the storage engine per worker thread — a concurrency and throughput signal, not a -predictor of the `Outstanding write transactions have too long of queue, please try again later` -(HTTP 503) rejection. That rejection is a duration check on a single outstanding commit -(see [`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime), -default 45s), not a function of concurrent commit count: a thread can carry a high `maxDepth` with -fast commits and never trip it, or sit at `depth` of 1 for the full timeout and trip it while this -metric barely moves. +in flight against the storage engine per worker thread — a concurrency and throughput signal. +`maxDepth` amplitude alone is not a reliable predictor of the `Outstanding write transactions have too +long of queue, please try again later` (HTTP 503) rejection: a thread can carry a high `maxDepth` with +fast commits and never trip it. But a single commit stuck in flight — the precondition for that +rejection — does show up here as a distinguishing pattern: `depth` pinned at 1 or more across many +consecutive raw samples (rather than a brief blip that clears) means a commit isn't settling. For a +direct duration reading on that specific risk, use `transaction-commit-time`, which records each +commit's submit-to-settle time on the same clock as the +[`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) +check (default 45s) — a rising p99/p999 there is the leading indicator for the 503. | Field | Unit | Description | | ---------- | ----- | ------------------------------------------------- | @@ -201,14 +205,14 @@ metric barely moves. | `maxDepth` | count | High-water mark observed over the sampling period | - **`write-transaction-queue-depth`** counts write commits handed to the storage engine whose commit - promises have not yet settled — how many commits this thread is juggling concurrently, not how - close any one of them is to the 503 timeout. This is in-flight, not durability: under - `storage.writeAsync: true` a settled commit promise does not guarantee the write has been synced - to disk. -- **`read-transaction-queue-depth`** counts concurrently open (snapshot) transactions. A high count - can mean either many short-lived reads or a few long-lived ones — the count alone can't - distinguish them, so use it as a concurrency signal; a metric with duration would be needed to spot - a single snapshot held open long enough to hold back compaction. + promises have not yet settled — how many commits this thread is juggling concurrently. This is + in-flight, not durability: under `storage.writeAsync: true` a settled commit promise does not + guarantee the write has been synced to disk. +- **`read-transaction-queue-depth`** counts concurrently open tracked read transactions, including + ones opened with snapshot disabled. A high count can mean either many short-lived reads or a few + long-lived ones — the count alone can't distinguish them, so use it as a concurrency signal; a + duration-based metric would be needed to identify a single transaction held open long enough to + hold back compaction. Both metrics are tracked only on the RocksDB write/read path. On an LMDB-backed database (`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` — indistinguishable @@ -220,12 +224,9 @@ source for spike detection. The aggregate `hdb_analytics` table is not a sum of each thread's `maxDepth` is first averaged across its raw samples for the period, then those per-thread averages are summed — so a brief single-thread spike is diluted rather than preserved. Always alert on `hdb_raw_analytics.maxDepth` (or lower the sampling/aggregation period) rather than -relying on the aggregate table to catch short spikes. A healthy system keeps -`write-transaction-queue-depth.maxDepth` near zero; a sustained non-zero peak that trends upward -indicates growing write concurrency worth investigating, but treat it as a throughput signal rather -than an early warning for the 503 — that rejection depends on a single commit's duration, not on how -many commits are queued. Tune the concrete alert threshold against a baseline for your workload, -since absolute depth scales with worker-thread count and per-transaction size. +relying on the aggregate table to catch short spikes. Tune the concrete alert threshold against a +baseline for your workload, since absolute depth scales with worker-thread count and per-transaction +size. #### `resource-usage` Metric diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index d99153b8..0aefc579 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -37,7 +37,7 @@ Type: `string` (duration) Default: `45s` -The maximum estimated time a write may wait in the commit queue before Harper rejects new writes with HTTP 503. Acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The maximum time a single write commit may remain unsettled before Harper starts rejecting new writes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold — it acts as backpressure when downstream disk I/O cannot keep up with incoming writes. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency. diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 88dd3363..b7938c3e 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -30,7 +30,7 @@ The `set_configuration` operation now accepts `"replicated": true` to apply a co ### Transaction queue depth metrics -New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread commit and read-snapshot concurrency against the storage engine, as a throughput/concurrency signal — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). +New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread write-commit and open-read-transaction concurrency against the storage engine, as a throughput/concurrency signal. A new `transaction-commit-time` metric records per-commit submit-to-settle duration, the leading indicator for the write-queue HTTP 503 rejection — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). ## CLI From a90548d987e924e19920a0115d849cfa039e1a37 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:49:58 -0600 Subject: [PATCH 6/7] fix(analytics): transaction-commit-time is silent on a true wedge, not rising MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Domain review caught an inversion in the prior commit: recordCommitLatency only fires on commitResolution.then(record, record), so a commit that never settles produces zero samples for transaction-commit-time — the opposite of "rising p99/p999 predicts the 503" for that failure mode. Reframe: commit-time is the leading indicator for gradual slowdowns toward the storage.maxTransactionQueueTime bound; write-transaction-queue-depth's depth staying elevated (plus Harper's own stuck-commit log line) is the signal for a genuine wedge. Also fix the transaction-commit-time table row (percentiles are aggregate-only per resources/analytics/write.ts, not present on hdb_raw_analytics) and note that per-thread analytics rows are activity-gated, so a quiet thread emits no row at all rather than an implicit zero. Co-Authored-By: Claude Sonnet 5 --- reference/analytics/overview.md | 55 +++++++++++++++++++-------------- release-notes/v5-lincoln/5.2.md | 2 +- 2 files changed, 33 insertions(+), 24 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 56579c50..0a18003c 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -179,7 +179,7 @@ Harper automatically tracks the following metrics for all services. Applications | `resource-usage` | (see below) | | various | Node.js process resource usage (see [resource-usage](#resource-usage-metric)) | | `storage-volume` | `available`, `free`, `size` | `database` | bytes | Storage volume size breakdown | | `table-size` | `size` | `database`, `table` | bytes | Table file size | -| `transaction-commit-time` | `median`, `mean`, `p95`, `p90` | | ms | Duration from write commit submission to settlement (see [transaction queue depth](#transaction-queue-depth-metrics)) | +| `transaction-commit-time` | `mean`, `median`, `p90`, `p95`, `p99`, `p999` | | ms | Duration from write commit submission to settlement (see [transaction queue depth](#transaction-queue-depth-metrics)) | | `utilization` | | | % | Percentage of time the worker thread was processing requests | | `write-transaction-queue-depth` | `depth`, `maxDepth` | | count | In-flight write transaction commits (see [transaction queue depth](#transaction-queue-depth-metrics)) | @@ -188,16 +188,21 @@ Harper automatically tracks the following metrics for all services. Applications `write-transaction-queue-depth` and `read-transaction-queue-depth` expose how many transactions are -in flight against the storage engine per worker thread — a concurrency and throughput signal. -`maxDepth` amplitude alone is not a reliable predictor of the `Outstanding write transactions have too -long of queue, please try again later` (HTTP 503) rejection: a thread can carry a high `maxDepth` with -fast commits and never trip it. But a single commit stuck in flight — the precondition for that -rejection — does show up here as a distinguishing pattern: `depth` pinned at 1 or more across many -consecutive raw samples (rather than a brief blip that clears) means a commit isn't settling. For a -direct duration reading on that specific risk, use `transaction-commit-time`, which records each -commit's submit-to-settle time on the same clock as the +in flight against the storage engine per worker thread — a concurrency and throughput signal, not a +reliable predictor on their own of the `Outstanding write transactions have too long of queue, please +try again later` (HTTP 503) rejection: `maxDepth` amplitude reflects concurrent commits, not whether +any single one is approaching the [`storage.maxTransactionQueueTime`](../database/storage-tuning.md#storagemaxtransactionqueuetime) -check (default 45s) — a rising p99/p999 there is the leading indicator for the 503. +duration limit (default 45s) that actually trips the 503. + +`transaction-commit-time` records each commit's submit-to-settle duration on that same clock, and a +rising `p99`/`p999` (in the `hdb_analytics` aggregate table, where percentiles are computed — they +aren't present on `hdb_raw_analytics`) is a leading indicator of _gradual_ slowdowns approaching that +limit. It doesn't help with a single commit that hangs indefinitely, though: the metric only records +once a commit settles, so a genuinely wedged commit contributes no sample at all, while +`write-transaction-queue-depth`'s `depth` stays elevated on that thread for as long as the commit +remains outstanding. Harper also logs once per stuck commit when the 503 check itself fires, which is +the authoritative signal for that specific failure. | Field | Unit | Description | | ---------- | ----- | ------------------------------------------------- | @@ -214,19 +219,23 @@ check (default 45s) — a rising p99/p999 there is the leading indicator for the duration-based metric would be needed to identify a single transaction held open long enough to hold back compaction. -Both metrics are tracked only on the RocksDB write/read path. On an LMDB-backed database -(`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` — indistinguishable -from a healthy, empty queue — regardless of actual read/write load. - -Both are gauges sampled per worker thread. The raw per-thread entries in `hdb_raw_analytics` retain -each thread's true instantaneous `depth` and per-period `maxDepth`; treat those as the reliable -source for spike detection. The aggregate `hdb_analytics` table is not a sum of per-thread peaks — -each thread's `maxDepth` is first averaged across its raw samples for the period, then those -per-thread averages are summed — so a brief single-thread spike is diluted rather than preserved. -Always alert on `hdb_raw_analytics.maxDepth` (or lower the sampling/aggregation period) rather than -relying on the aggregate table to catch short spikes. Tune the concrete alert threshold against a -baseline for your workload, since absolute depth scales with worker-thread count and per-transaction -size. +Both metrics are gauges tracked only on the RocksDB write/read path, sampled per worker thread. On an +LMDB-backed database (`storage.engine: lmdb`), `depth` and `maxDepth` for both metrics always read `0` +— indistinguishable from a healthy, empty queue — regardless of actual read/write load. All per-thread +analytics reporting, including these gauges, piggybacks on the thread having recorded some other +analytics-eligible activity in the period — a thread with no recordable activity in a given second +emits no row at all rather than an explicit `depth: 0`. Absence of a sample is not the same as a +healthy reading, particularly for `read-transaction-queue-depth` on an otherwise-quiet thread holding a +single long-lived read. + +The raw per-thread entries in `hdb_raw_analytics` retain each thread's true instantaneous `depth` and +per-period `maxDepth`; treat those as the reliable source for spike detection. The aggregate +`hdb_analytics` table is not a sum of per-thread peaks — each thread's `maxDepth` is first averaged +across its raw samples for the period, then those per-thread averages are summed — so a brief +single-thread spike is diluted rather than preserved. Always alert on `hdb_raw_analytics.maxDepth` (or +lower the sampling/aggregation period) rather than relying on the aggregate table to catch short +spikes. Tune the concrete alert threshold against a baseline for your workload, since absolute depth +scales with worker-thread count and per-transaction size. #### `resource-usage` Metric diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index b7938c3e..d95b12c9 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -30,7 +30,7 @@ The `set_configuration` operation now accepts `"replicated": true` to apply a co ### Transaction queue depth metrics -New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread write-commit and open-read-transaction concurrency against the storage engine, as a throughput/concurrency signal. A new `transaction-commit-time` metric records per-commit submit-to-settle duration, the leading indicator for the write-queue HTTP 503 rejection — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). +New `write-transaction-queue-depth` and `read-transaction-queue-depth` metrics report per-thread write-commit and open-read-transaction concurrency against the storage engine, as a throughput/concurrency signal. A new `transaction-commit-time` metric records per-commit submit-to-settle duration, a leading indicator of gradual slowdowns toward the write-queue HTTP 503 rejection — see [Transaction Queue Depth Metrics](/reference/v5/analytics/overview#transaction-queue-depth-metrics). ## CLI From 91d9fecd145a469123d616c3665b38dcee31d7f9 Mon Sep 17 00:00:00 2001 From: Kris Zyp Date: Fri, 31 Jul 2026 18:57:14 -0600 Subject: [PATCH 7/7] fix(docs): sync maxTransactionQueueTime bypass note and remove stale duplicate wording Verified against resources/Table.ts:2218,4407 (checkOverloaded guarded by !context?.source) and the 503 log text in DatabaseTransaction.ts:403: deletes and canonical-source writes (replication, caching sources) bypass the queue-time check entirely. Note that in storage-tuning.md, and sync configuration/options.md's duplicate one-line description (previously "Max write queue time before 503", which re-introduced the queue-length framing this branch corrects elsewhere) to point at the same section instead of drifting. Also tighten the maxDepth field description: the high-water mark resets on emit, which is activity-gated, not a fixed sampling interval. Co-Authored-By: Claude Sonnet 5 --- reference/analytics/overview.md | 8 ++++---- reference/configuration/options.md | 2 +- reference/database/storage-tuning.md | 2 +- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/reference/analytics/overview.md b/reference/analytics/overview.md index 0a18003c..86997e13 100644 --- a/reference/analytics/overview.md +++ b/reference/analytics/overview.md @@ -204,10 +204,10 @@ once a commit settles, so a genuinely wedged commit contributes no sample at all remains outstanding. Harper also logs once per stuck commit when the 503 check itself fires, which is the authoritative signal for that specific failure. -| Field | Unit | Description | -| ---------- | ----- | ------------------------------------------------- | -| `depth` | count | Instantaneous depth sampled at emit time | -| `maxDepth` | count | High-water mark observed over the sampling period | +| Field | Unit | Description | +| ---------- | ----- | ------------------------------------------------------- | +| `depth` | count | Instantaneous depth sampled at emit time | +| `maxDepth` | count | High-water mark since this thread's last emitted sample | - **`write-transaction-queue-depth`** counts write commits handed to the storage engine whose commit promises have not yet settled — how many commits this thread is juggling concurrently. This is diff --git a/reference/configuration/options.md b/reference/configuration/options.md index 482c7d9d..326f6997 100644 --- a/reference/configuration/options.md +++ b/reference/configuration/options.md @@ -253,7 +253,7 @@ storage: - `compression` — LZ4 record compression; _Default_: `true` (enabled by default since v4.3.0). Sub-options: `dictionary`, `threshold` - `compactOnStart` — Compact all non-system databases on startup; _Default_: `false` (Added in: v4.3.0) - `compactOnStartKeepBackup` — Retain compaction backups; _Default_: `false` -- `maxTransactionQueueTime` — Max write queue time before 503; _Default_: `45s` +- `maxTransactionQueueTime` — Max time a single write commit may stay unsettled before Harper starts rejecting writes with 503; see [Storage Tuning](../database/storage-tuning.md#storagemaxtransactionqueuetime); _Default_: `45s` - `noReadAhead` — Advise OS against read-ahead; _Default_: `false` - `prefetchWrites` — Prefetch before write transactions; _Default_: `true` - `path` — Database files directory; _Default_: `/database` diff --git a/reference/database/storage-tuning.md b/reference/database/storage-tuning.md index 0aefc579..e9964657 100644 --- a/reference/database/storage-tuning.md +++ b/reference/database/storage-tuning.md @@ -37,7 +37,7 @@ Type: `string` (duration) Default: `45s` -The maximum time a single write commit may remain unsettled before Harper starts rejecting new writes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold — it acts as backpressure when downstream disk I/O cannot keep up with incoming writes. +The maximum time a single write commit may remain unsettled before Harper starts rejecting new application-originated writes on that thread with HTTP 503. This is a per-commit duration check, not a queue-length threshold — it acts as backpressure when downstream disk I/O cannot keep up with incoming writes. Deletes and writes applied from a canonical source (e.g. replication or a caching source) bypass this check. Lower this in latency-sensitive systems where it is better to shed load early than to let request queues grow. Raise it when occasional disk-write bursts are expected and the application can tolerate longer commit latency.