Skip to content

feat(admin): Add replication delay by table system query - #8576

Open
onewland wants to merge 1 commit into
masterfrom
oliver/admin-replication-delay-query
Open

onewland wants to merge 1 commit into
masterfrom
oliver/admin-replication-delay-query

Conversation

@onewland

Copy link
Copy Markdown
Contributor

Adds a ReplicationDelayByTable predefined system query to Snuba Admin. For every replicated table on the selected host it shows:

  • absolute_delay from system.replicas
  • the host's max_replica_delay_for_distributed_queries setting
  • considered_stale: whether the delay is above that threshold, which means distributed queries skip this replica (unless every replica of the shard is stale and fallback_to_stale_replicas_for_distributed_queries is on)
  • is_readonly

Host-level replication delay monitoring reports only the max across all tables. This query makes it easy to find which table is driving that number and whether reads are affected. For example, a table whose replication queues were deliberately stopped will show a steadily climbing delay.

Tested locally: the query passes is_query_using_only_system_tables and tests/admin/test_api.py::test_predefined_system_queries passes.

🤖 Generated with Claude Code

Add a predefined system query showing absolute_delay for every replicated
table on the selected host, along with whether that delay exceeds
max_replica_delay_for_distributed_queries (i.e. whether distributed queries
would treat the replica as stale).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@onewland
onewland requested a review from a team as a code owner October 20, 2026 20:45

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 1711d71. Configure here.

table,
absolute_delay,
getSetting('max_replica_delay_for_distributed_queries') AS max_delay,
absolute_delay > max_delay AND max_delay > 0 AS considered_stale,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect stale replica threshold comparison

Low Severity

considered_stale flags a replica only when absolute_delay is strictly greater than max_delay. ClickHouse skips a replica when delay is greater than or equal to max_replica_delay_for_distributed_queries, so a replica sitting at the threshold is reported as not stale even though distributed queries skip it.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 1711d71. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant