[POC] Bucket storage report (operations vs rows)

[bean1352]

Bucket storage report

What this adds

A new admin endpoint, POST /api/admin/v1/bucket-report. It shows how much operation history each bucket carries compared to its actual rows.

How it works

The report only reads bucket_state. It never scans the operation history.

• operations, operation_bytes: kept up to date by writers and compactors. Exact.
• rows: the row count captured by the bucket's last full compact. null if the bucket has never been fully compacted.
• fragmentation: operations / rows. Near 1 is healthy. 167 means new clients download 167 operations per live row.
• uncompacted_operations: operations written after that compact, which is how stale rows might be.
• last_full_compact_at, next_compact_at: when the row stats were captured, and when the scheduler next considers the bucket. A suggested compact with a future date is already planned, just throttled until then.
• suggested_action: compact, defragment, both, none, or unknown when there are no compact statistics to reason from.

Worst buckets come first. There is also a rollup per definition and totals for the instance. limit defaults to 50 and caps at 1000; invalid values get a 400 error instead of being clamped.

Example

POST /api/admin/v1/bucket-report
Authorization: Bearer <admin token>

{ "limit": 50 }

Response (from a real test run). This bucket was fully compacted at 07:35, then 15 more operations were written to it, so rows is a slightly stale snapshot and the report says so:

{
  "buckets": [
    {
      "bucket": "by_user.1.3[\"u1\"]",
      "operations": 48,
      "operation_bytes": 9022,
      "uncompacted_operations": 15,
      "rows": 5,
      "fragmentation": 9.6,
      "last_full_compact_at": "2026-08-26T07:35:41.325Z",
      "next_compact_at": "2026-08-26T07:41:34.066Z",
      "suggested_action": "both"
    }
  ],
  "definitions": [
    {
      "definition": "by_user.1.3",
      "bucket_count": 4,
      "operations": 192,
      "operation_bytes": 36124,
      "uncompacted_operations": 60,
      "rows": 20,
      "fragmentation": 9.6,
      "suggested_action": "both"
    }
  ],
  "totals": {
    "bucket_count": 11,
    "operations": 281,
    "operation_bytes": 62302,
    "estimated": false
  },
  "buckets_truncated": false,
  "definitions_truncated": false
}

Scaling

Up to 50k buckets the scan is exact. Above that the report ranks a sample of about 10k buckets, using only the _id index, and marks the totals estimated: true. Past 1M buckets it fails fast instead of scanning without bound.

Storage v1/v2 do not record compact statistics, so they report operation counts only: rows, fragmentation and the dates are null and the action is unknown. That keeps v1 cheap and nudges users towards v3.

AI disclaimer

I developed this change using Claude, and reviewed and tested it myself.

[changeset-bot]

🦋 Changeset detected

Latest commit: 082868c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages

Name	Type
@powersync/service-core	Minor
@powersync/service-types	Minor
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core-tests	Minor
@powersync/service-client	Minor
@powersync/service-module-convex	Patch
@powersync/service-module-core	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Minor
test-client	Patch
@powersync/service-schema	Minor
@powersync/lib-service-postgres	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

I had a quick check on $sampleRate performance, and unfortunately it's not a magic bullet in terms of performance. Essentially MongoDB still has to scan through the index at least before applying the $sampleRate. I can work, as long as we ensure:

1. Limit the number of index entries a query is scanning to around 100k-1M at most.
2. Make sure no document lookup is performed before filtering using $sampleRate. E.g. if we scan through 100k index entries to sample around 1000 of them, the query must scan through 1000 documents, not 100k documents (fetching documents is much slower than the index entries). Use MongoDB's explain to confirm this.

As an additional safeguard, we can use readPreference: 'secondaryOnly' for these queries, to make sure they don't affect performance on the primary node.

[rkistner]

$sample does not help for performance unless it's the first stage in the pipeline.

Potential options:

1. $sample first, then filter. That would require the initial sample size to be higher than the limit we want.
2. Filter first, then use $sampleRate. I'm not actually what the performance is like for $sampleRate - would need some testing.

And if you go for option 2, node that current _id.b / _id.g filters aren't efficient either, and require a full collection scan. Do filter efficiently, you need to use a pattern such as _id: {$gte: ..., $lt: ...} - there should be a couple of examples like that in this repo you can use as a starting point.

[bean1352]

The bucket report does not read bucket_data anymore.

The only sampling left is for picking the top buckets in bucket_state (when above 50k buckets) and it uses option 2

[rkistner]

@bean1352 The MongoDB V3 storage had a significant change now in what we store in bucket_state: #752

I believe that should allow using only the bucket_state collection to compute these reports, perhaps with minor tweaks. It would probably still require sampling on that collection to handle cases of say 100M+ buckets, but should not require reading bucket_data anymore.

More specifically:

1. bucket_state contains all the stats we use to determine whether or not to compact. Note that there are two types of compact per bucket now: A fast chunk-merge compact, and a full compact. In most cases, users should not need to compact manually, since a scheduled compact can run much more often now. But reporting on it could still be useful to see the cases such as "a compact would be beneficial, but it is throttled until X".
2. Fragmentation ratio can be read from the full compact state, something like 1.0 - (last_full_compact.puts / last_full_compact.count). That would not be quite a live counter, but should give a good enough indication. And since it's computed when the compact was run, each put operation is generally unique, so there's no need to try and count actual unique rows.

This would still leave cases with V1 storage. I'd recommend focusing on V3, and give more limited stats for V1, rather than attempting to do a more expensive scan for V1. That would further encourage moving users over to V3.

[bean1352]

@bean1352 The MongoDB V3 storage had a significant change now in what we store in bucket_state: #752

I believe that should allow using only the bucket_state collection to compute these reports, perhaps with minor tweaks. It would probably still require sampling on that collection to handle cases of say 100M+ buckets, but should not require reading bucket_data anymore.

More specifically:

1. bucket_state contains all the stats we use to determine whether or not to compact. Note that there are two types of compact per bucket now: A fast chunk-merge compact, and a full compact. In most cases, users should not need to compact manually, since a scheduled compact can run much more often now. But reporting on it could still be useful to see the cases such as "a compact would be beneficial, but it is throttled until X".
2. Fragmentation ratio can be read from the full compact state, something like 1.0 - (last_full_compact.puts / last_full_compact.count). That would not be quite a live counter, but should give a good enough indication. And since it's computed when the compact was run, each put operation is generally unique, so there's no need to try and count actual unique rows.

This would still leave cases with V1 storage. I'd recommend focusing on V3, and give more limited stats for V1, rather than attempting to do a more expensive scan for V1. That would further encourage moving users over to V3.

The bucket report only reads bucket_state now.

The row count of a bucket comes from last_full_compact.puts, like you suggested. For the fragmentation score, we report operations / rows. Should I change that to 1.0 - (last_full_compact.puts / last_full_compact.count)?

1. The rows field for a bucket stays null until the first full compact runs on that bucket. The compact that runs after initial replication only merges chunks and does not write the last_full_compact stats. Should it write them too, so rows are available from the start?
2. Above 50k buckets the report falls back to sampling and above 1M buckets it returns a 422 response. Is this fine for now?

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 631f5e022a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Avoid sampling away the worst offenders

When an active configuration has more than 50,000 buckets, this uniform sample is applied before the operation-count sort. At 50,001 buckets, a single pathological bucket has roughly an 80% chance of being omitted, causing both the returned “worst offenders” and the scaled operation total to miss precisely the outlier this endpoint is intended to diagnose. The totals.estimated flag does not make the randomly truncated top-bucket list actionable; determine the top buckets from the full matched set, even if sampling remains necessary for aggregate totals.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Include legacy v1/v2 buckets in the report

On upgraded storage-version 1/2 instances, buckets created before bucket_state tracking can be absent from this collection indefinitely: BucketStateDocumentBase explicitly notes at models.ts:138-139 that no migration populated existing data and only new updates create state. Querying only bucketStateV1 therefore omits untouched legacy buckets and their operations while returning estimated: false, contradicting the endpoint's exact-count contract for supported v1/v2 storage; use the bucket-data source or backfill/flag incomplete results.

AGENTS.md reference: AGENTS.md:L70-L77

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 082868cf17

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Apply the fragmentation tie-break before limiting buckets

When more than limit buckets share the operation count at the cutoff, this database sort selects an arbitrary subset because it considers only operations; assembleBucketReport() applies the documented fragmentation tie-break only after the subset has already been limited. Consequently, a more fragmented tied bucket can be omitted from the worst-offender list. Include the fragmentation tie-break in the aggregation before $limit, or avoid promising that secondary ranking.

Useful? React with 👍 / 👎.