Skip to content

6.2.7 and above (6.3.2 inclusive): "Failed to remove consumer" NPE (ConditionalNetworkBridgeFilterFactory) during broker shutdown - stale consumers left behind #2519

Description

@dpavlov2024

Issue body (paste below the title field)

Bug Report

Describe the bug

After upgrading to ActiveMQ Classic 6.2.7 (also 6.2.8), brokers in a broker
network log Failed to remove consumer: ID:... with a NullPointerException
when a broker is stopped/restarted or a network bridge drops under load. The
consumer/connection cleanup is interrupted, so consumers are left registered
(stale connections) until the broker is restarted.

The NPE comes from the advisory path introduced by "Send advisory messages
using Broker connection context" (#2071 / #2075):

WARN  | Failed to remove consumer: ID:... | TransportConnection
java.lang.NullPointerException: Cannot invoke "...isQueue()" because the
return value of "...MessageEvaluationContext.getDestination()" is null
	at ConditionalNetworkBridgeFilterFactory$ConditionalNetworkBridgeFilter
	   .matchesForwardingFilter(ConditionalNetworkBridgeFilterFactory.java:116)
	...
	at AdvisoryBroker.removeConsumer(AdvisoryBroker.java:400)
	at TransportConnection.processRemoveConsumer(TransportConnection.java:739)

We believe the shared broker-admin ConnectionContext (one mutable
MessageEvaluationContext) is cleared by one concurrent advisory dispatch
while another is evaluating a conditional network bridge filter.

Expected behavior per documentation

This should work correctly per ActiveMQ's documented behavior:

  • Broker networks with demand forwarding and replay-when-no-consumers
    (ConditionalNetworkBridgeFilterFactory) are a supported, documented
    topology; the 6.2.7 release notes
    only describe the advisory change as a
    permission improvement (the broker now publishes advisories under its own
    context, so users no longer need write access to advisory topics) - no
    change in dispatch or cleanup semantics is documented.
  • Consumers and connections are expected to be removed cleanly when clients
    disconnect and during broker shutdown/reconnect; no stale consumers should
    remain.
  • Advisory messages are broker-internal bookkeeping and should not be able to
    break consumer cleanup.

Reality on 6.2.7/6.2.8 with an unchanged 6.2.6 configuration: the NPE above
appears during storms and cleanup is left incomplete - which is why we treat
this as a regression.

To Reproduce

  • A docker-compose reproduction (3 brokers, loadgen, one command) will be
    attached to this issue; steps to run it are included in the attachment.
  • In short: a broker network with ConditionalNetworkBridgeFilterFactory
    (replayWhenNoConsumers) on the destination policies and a
    network-mirrored wildcard topic subscription (a > topic consumer).
    With many concurrent consumer create/remove operations, gracefully stop
    one broker; clients reconnect -> the NPE below appears.
  • Expected on 6.2.7 in the first storm round; on 6.2.6 the same setup stays
    clean.

Expected behavior

Consumer/connection removal completes cleanly during shutdown/reconnect
storms; no NPE, no stale consumers.

Your Environment

  • Version: 6.2.7 (also reproduced on 6.2.8; not present in 6.2.6)
  • Configuration: static network connectors (duplex/dynamic demand),
    conditionalNetworkBridgeFilterFactory on queue/topic policies
  • OS: Linux (docker)

Additional context

Introducing change: commit 15e78425, PRs #2071/#2075. A null guard at
ConditionalNetworkBridgeFilterFactory:116 stops this exact NPE, but giving
each advisory dispatch its own evaluation context would fix the underlying
race.

docker.zip

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions