Skip to content

[ISSUE #190] Support queue-level consumer metrics behind a topic whitelist - #191

Open
xuanskyer wants to merge 2 commits into
apache:masterfrom
xuanskyer:pr/queue-level-metrics
Open

[ISSUE #190] Support queue-level consumer metrics behind a topic whitelist#191
xuanskyer wants to merge 2 commits into
apache:masterfrom
xuanskyer:pr/queue-level-metrics

Conversation

@xuanskyer

Copy link
Copy Markdown

What is the purpose of the change

Closes #190.

All consumer-side metrics are aggregated to broker granularity today, so a single lagging queue is invisible in Prometheus — rocketmq_group_diff only shows that the whole group is behind, and one has to fall back to the console to see per-queue offsets. This PR exposes the queue dimension that the collector already has in hand.

MetricsCollectTask#collectConsumerOffset iterates consumeStats.getOffsetTable(), which is a Map<MessageQueue, OffsetWrapper>, and already computes lagTime one queue at a time. Both loops then fold the values into a HashMap<brokerName, Long> and discard queueId. Publishing the per-queue values therefore needs no additional admin callqueryMsgByOffset is still invoked exactly once per queue, as before.

Note that the label list for this already exists in RMQMetricsCollector (GROUP_PULL_LATENCY_LABEL_NAMES, containing queueid) but has never been referenced by any metric.

Brief changelog

New metrics, all labelled cluster, broker, topic, group, queueid:

metric meaning
rocketmq_queue_group_diff per-queue unconsumed messages (brokerOffset - consumerOffset)
rocketmq_queue_group_get_latency_by_storetime per-queue consume latency in ms (reuses the already computed lagTime)
rocketmq_queue_consumer_offset per-queue consumer offset

Queue-level series multiply the consumer series count by the number of queues per broker, so they are disabled by default and gated by an explicit topic whitelist:

rocketmq:
  config:
    queueLevelTopics: ""   # empty = disabled (default); "topic-a,topic-b" = whitelist; "*" = all topics

Changes:

  • RMQConfigure — new queueLevelTopics property, parsed once in the setter into a Set, plus isQueueLevelTopic(String).
  • ConsumerQueueMetric (new) — metric key, i.e. ConsumerMetric plus queueId.
  • RMQMetricsCollector — three caches reusing the existing outOfTimeSeconds expiry, collectQueueGroupNums() and three addXxxMetric() methods, following the existing groupDiff pattern.
  • MetricsCollectTask — publishes the per-queue values inside the two existing loops, guarded by the whitelist. The broker-level aggregation is left exactly as is, so existing dashboards and alerts are unaffected.
  • application.yml — documents the new option.

The change is purely additive: no existing line of behaviour was modified.

The second commit fixes a pre-existing checkstyle violation (if( missing a space, MetricsCollectTask.java) that currently makes mvn clean install fail on master regardless of this PR — without it the build command below cannot pass. Happy to split it out if you prefer.

Verifying this change

  • mvn -B clean install -DskipITsBUILD SUCCESS, Tests run: 5, Failures: 0, Errors: 0
  • mvn -B clean apache-rat:check checkstyle:checkstyle — RAT summary Unapproved: 0, unknown: 0, approved: 39; checkstyle clean after the second commit
  • New unit test RMQConfigureQueueLevelTest covers the whitelist parsing: disabled by default, blank stays disabled, comma list is trimmed and matched exactly (topic-a must not enable topic-a-extra, otherwise the series budget silently blows up), * wildcard, and resetting back to disabled. The test was also verified in reverse — replacing the exact match with startsWith makes it fail as expected.

Note that findbugs:findbugs from the checklist was not run: the plugin does not support the JDK used locally (17). spotbugs would be the modern replacement, but that seemed out of scope for this PR.

One thing worth mentioning for the maintainers: the existing test sources are JUnit 4, while surefire 3.2.2 auto-selects the JUnit Platform provider, so mvn test currently reports Tests run: 0 and no existing test actually executes. The new test is therefore written against JUnit 5 so that it really runs. Adding junit-vintage-engine would revive the existing ones, but again felt out of scope here.

@xuanskyer

Copy link
Copy Markdown
Author

@Jason918 @lizhanhui

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support queue-level consumer metrics (diff / latency / consumer offset)

1 participant