Skip to content

Drill a RabbitMQ dynamic credential rotation under a connected consumer #369

New issue
New issue
Closed
#684
Closed
Drill a RabbitMQ dynamic credential rotation under a connected consumer#369
#684
Labels
area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.component: securitySecurity, permissions, secrets, or vulnerability handling.priority: P2Medium; normal planned work.type: choreMaintenance work without intended behavior change.

Description

@ExtraToast
ExtraToast
opened on Oct 12, 2026

What to do

Rotate a RabbitMQ dynamic credential under a consumer that holds an open connection, and record what the consumer sees. Decide what each settled decision does about the claim under it that failed keeps rabbitmq in the grant union as a dynamic credential on the condition that this works.

The estate dropped the dynamic credential once, after Vault revoked a RabbitMQ user under a running pod. The cause recorded in fleet-infra is the Vault Agent injector, which rendered once and never renewed the lease. Vault Secrets Operator (VSO) renews a lease and then replaces the credential and restarts its consumers, which the estate already runs for Postgres. No workload has run that cycle against RabbitMQ, and what a client with an open connection sees when its user is deleted is unverified (docs/research/unsupported-claims.md, claims C7 to C9).

The drill writes to the live Vault and the live cluster, so it is run with the maintainer present, in a throwaway namespace, against a throwaway RabbitMQ role and vhost.

  1. Read the rabbitmq mount's lease settings from Vault (rabbitmq/config/lease). The bootstrap script passes default_ttl and max_ttl to the role, where the API defines neither, and writes no config/lease.
  2. Set a short lease on a drill role, or on the mount for the length of the drill, so that a renewal and a replacement both happen within minutes.
  3. Deliver the credential with a VaultDynamicSecret that names a Deployment in rolloutRestartTargets. The Deployment runs a consumer that holds one AMQP connection open and logs every connection event.
  4. Watch one renewal and one replacement. Record whether the consumer was restarted with the new credential before the old user was deleted, and what the old connection saw.
  5. Remove the namespace, the role and any changed lease setting.

Done when

  • The mount's real lease lifetime is recorded.
  • One renewal and one replacement are observed, with the consumer's log of each.
  • The result is posted here: either the cycle is clean, or the failure is described well enough to decide between a dynamic and a static credential.
  • Nothing the drill created is left in Vault, RabbitMQ or the cluster.

Notes

This runs before chapter 17 (Secrets and access) is rewritten, because a failure reopens the grant decision. It sits outside the map Map: re-establish deploy-kit's goals, rewrite its spec, and order the work until the estate is handed over, which is not held up by it.

Activity

  1. ExtraToast
    added
    type: choreMaintenance work without intended behavior change.
    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.
    component: securitySecurity, permissions, secrets, or vulnerability handling.
    priority: P2Medium; normal planned work.
    on Oct 12, 2026
  2. ExtraToast
    marked this as blocking on Oct 12, 2026
  3. ExtraToast
    added a parent issue on Oct 27, 2026

  4. ExtraToast commented on Oct 29, 2026

    @ExtraToast
    ExtraToast
    on Oct 29, 2026
    ContributorAuthor

    Drill result: a RabbitMQ dynamic credential rotated under a connected consumer

    Run 369-20261029T114030Z of drills/rabbitmq-rotation.sh (estate d595317), against kubectl context personal, 2026-10-29 11:40–11:47 UTC.

    The live mount's lease

    Read before the drill and unchanged after it: sys/mounts/rabbitmq/tune is {"default_lease_ttl":2764800,"max_lease_ttl":2764800}, and rabbitmq/config/lease is none written (No value found at rabbitmq/config/lease). Role app-consumer: {"tags":"","vhost_topics":null,"vhosts":{"/":{"configure":".*","read":".*","write":".*"}}}.

    The drill

    A separate engine, rabbitmq-drill, on the same broker, leasing for 60s up to 240s, so the live mount and its consumer were never touched. VSO (hashicorp/vault-secrets-operator:1.5.1) delivered it with a VaultDynamicSecret (renewalPercent: 67, rolloutRestartTargets naming the consumer) to one pika consumer holding one connection, with a 10 s heartbeat and a publish every 5 s.

    time what happened
    +1:13 pod consumer-5b9bcccbdc-rlcx7 runs on frankfurt-contabo-1
    +1:13 pod consumer-5b9bcccbdc-rlcx7 is terminating
    +1:13 pod consumer-66f9fbcf48-h9l25 runs on frankfurt-contabo-1
    +1:13 pod consumer-67c6874dbf-lhk7t runs on frankfurt-contabo-1
    +1:13 pod consumer-67c6874dbf-lhk7t is terminating
    +1:15 11:41:40 consumer-5b9bcccbdc-rlcx7 start, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:40 consumer-5b9bcccbdc-rlcx7 connected, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:42 consumer-5b9bcccbdc-rlcx7 sigterm, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:45 consumer-5b9bcccbdc-rlcx7 closed (by=consumer), as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:45 consumer-5b9bcccbdc-rlcx7 exit, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:39 consumer-66f9fbcf48-h9l25 start, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:39 consumer-66f9fbcf48-h9l25 connected, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:38 consumer-67c6874dbf-lhk7t start, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:15 11:41:39 consumer-67c6874dbf-lhk7t connected, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:24 first lease rabbitmq-drill/creds/drill/iAEM5ZLyilMkHCm8MSRpuGSM, 60s, user kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +1:43 lease renewed (#1): 60s on rabbitmq-drill/creds/drill/iAEM5ZLyilMkHCm8MSRpuGSM
    +2:25 lease renewed (#2): 60s on rabbitmq-drill/creds/drill/iAEM5ZLyilMkHCm8MSRpuGSM
    +3:09 lease renewed (#3): 60s on rabbitmq-drill/creds/drill/iAEM5ZLyilMkHCm8MSRpuGSM
    +3:51 lease renewed (#4): 60s on rabbitmq-drill/creds/drill/iAEM5ZLyilMkHCm8MSRpuGSM
    +4:40 credential replaced: kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0 → kubernetes-drill-369-consumer-698afbd0-b82f-44e1-2cda-db8033f5fd20, a 60s lease
    +4:54 pod consumer-5f7549c94d-fz8s4 runs on frankfurt-contabo-1
    +4:54 11:45:24 consumer-66f9fbcf48-h9l25 closed (by=broker code=320 text=CONNECTION_FORCED - user 'kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0' is deleted), as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +4:58 pod consumer-66f9fbcf48-h9l25 is terminating
    +4:58 11:45:24 consumer-5f7549c94d-fz8s4 start, as kubernetes-drill-369-consumer-698afbd0-b82f-44e1-2cda-db8033f5fd20
    +4:58 11:45:24 consumer-5f7549c94d-fz8s4 connected, as kubernetes-drill-369-consumer-698afbd0-b82f-44e1-2cda-db8033f5fd20
    +4:58 11:45:25 consumer-66f9fbcf48-h9l25 sigterm, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +4:58 11:45:27 consumer-66f9fbcf48-h9l25 exit, as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0
    +5:10 RabbitMQ no longer has kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0

    What the old connection (kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0) saw:

    • 11:41:39 consumer-66f9fbcf48-h9l25 start
    • 11:41:39 consumer-66f9fbcf48-h9l25 connected
    • 11:45:24 consumer-66f9fbcf48-h9l25 closed by=broker code=320 text=CONNECTION_FORCED - user 'kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0' is deleted
    • 11:45:25 consumer-66f9fbcf48-h9l25 sigterm
    • 11:45:27 consumer-66f9fbcf48-h9l25 exit
    • last successful tick 11:45:20
    • 11:41:40 consumer-5b9bcccbdc-rlcx7 start
    • 11:41:40 consumer-5b9bcccbdc-rlcx7 connected
    • 11:41:42 consumer-5b9bcccbdc-rlcx7 sigterm
    • 11:41:45 consumer-5b9bcccbdc-rlcx7 closed by=consumer
    • 11:41:45 consumer-5b9bcccbdc-rlcx7 exit
    • last successful tick 11:41:40
    • 11:41:38 consumer-67c6874dbf-lhk7t start
    • 11:41:39 consumer-67c6874dbf-lhk7t connected
    • last successful tick 11:41:59

    Failures the consumer logged:

    • 11:45:24 consumer-66f9fbcf48-h9l25 as kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0: the broker closed the connection, by=broker code=320 text=CONNECTION_FORCED - user 'kubernetes-drill-369-consumer-32b7a0fd-8be7-17cb-97a3-6ffebc8834e0' is deleted

    Verdict: the cycle is not clean; the consumer saw 1 failure(s), listed below. That is the evidence for choosing between a dynamic and a static credential.

    Done when

    • The mount's real lease lifetime is recorded.
    • One renewal and one replacement are observed, with the consumer's log of each.
    • The result is posted here.
    • Nothing the drill created is left in Vault, RabbitMQ or the cluster.

  5. ExtraToast commented on Oct 29, 2026

    @ExtraToast
    ExtraToast
    on Oct 29, 2026
    ContributorAuthor

    Drill result: a RabbitMQ dynamic credential rotated under a connected consumer

    Run 369-20261029T122236Z of drills/rabbitmq-rotation.sh (estate 5f1107a), against kubectl context personal, 2026-10-29 12:22–12:28 UTC.

    The live mount's lease

    Read before the drill and unchanged after it: sys/mounts/rabbitmq/tune is {"default_lease_ttl":2764800,"max_lease_ttl":2764800}, and rabbitmq/config/lease is none written (No value found at rabbitmq/config/lease). Role app-consumer: {"tags":"","vhost_topics":null,"vhosts":{"/":{"configure":".*","read":".*","write":".*"}}}.

    The drill

    A separate engine, rabbitmq-drill, on the same broker, leasing for 60s up to 240s, so the live mount and its consumer were never touched. VSO (hashicorp/vault-secrets-operator:1.5.1) delivered it with a VaultDynamicSecret (renewalPercent: 33, rolloutRestartTargets naming the consumer) to one pika consumer holding one connection, with a 10 s heartbeat and a publish every 5 s, which waits 30s after starting before it connects and is Ready only once connected.

    time what happened
    +1:39 pod consumer-68f585c778-5wjc8 runs on frankfurt-contabo-1
    +1:39 pod consumer-85969d5ccf-fnb2v runs on frankfurt-contabo-1
    +1:39 pod consumer-85969d5ccf-fnb2v is terminating
    +1:41 12:23:42 consumer-68f585c778-5wjc8 start, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:41 12:24:12 consumer-68f585c778-5wjc8 connected, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:42 12:23:42 consumer-85969d5ccf-fnb2v start, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:42 12:24:12 consumer-85969d5ccf-fnb2v connected, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:42 12:24:14 consumer-85969d5ccf-fnb2v sigterm, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:42 12:24:17 consumer-85969d5ccf-fnb2v closed (by=consumer), as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:42 12:24:17 consumer-85969d5ccf-fnb2v exit, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +1:54 first lease rabbitmq-drill/creds/drill/UHtSLjjqyadQUQn1p4UQAXwH, 60s, user kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +2:05 lease renewed (#1): 60s on rabbitmq-drill/creds/drill/UHtSLjjqyadQUQn1p4UQAXwH
    +2:31 lease renewed (#2): 60s on rabbitmq-drill/creds/drill/UHtSLjjqyadQUQn1p4UQAXwH
    +3:16 lease renewed (#3): 60s on rabbitmq-drill/creds/drill/UHtSLjjqyadQUQn1p4UQAXwH
    +3:39 lease renewed (#4): 60s on rabbitmq-drill/creds/drill/UHtSLjjqyadQUQn1p4UQAXwH
    +4:03 credential replaced: kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c → kubernetes-drill-369-consumer-fed331bd-5db7-7b84-c77d-9a77838826f4, a 60s lease
    +4:13 pod consumer-66f694b8dd-hp7g5 runs on frankfurt-contabo-1
    +4:16 12:26:50 consumer-66f694b8dd-hp7g5 start, as kubernetes-drill-369-consumer-fed331bd-5db7-7b84-c77d-9a77838826f4
    +4:33 lease renewed (#5): 60s on rabbitmq-drill/creds/drill/v77pUVnTw86jhnAI4sPobnUv
    +4:48 pod consumer-68f585c778-5wjc8 is terminating
    +4:48 12:27:20 consumer-66f694b8dd-hp7g5 connected, as kubernetes-drill-369-consumer-fed331bd-5db7-7b84-c77d-9a77838826f4
    +4:48 12:27:21 consumer-68f585c778-5wjc8 sigterm, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +4:48 12:27:23 consumer-68f585c778-5wjc8 closed (by=consumer), as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +4:48 12:27:23 consumer-68f585c778-5wjc8 exit, as kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c
    +4:51 lease renewed (#6): 60s on rabbitmq-drill/creds/drill/v77pUVnTw86jhnAI4sPobnUv
    +5:00 RabbitMQ no longer has kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c

    What the old connection (kubernetes-drill-369-consumer-744c03c5-e53f-2bae-dba0-bf5280cdf75c) saw:

    • 12:23:42 consumer-68f585c778-5wjc8 start

    • 12:24:12 consumer-68f585c778-5wjc8 connected

    • 12:27:21 consumer-68f585c778-5wjc8 sigterm

    • 12:27:23 consumer-68f585c778-5wjc8 closed by=consumer

    • 12:27:23 consumer-68f585c778-5wjc8 exit

    • last successful tick 12:27:18

    • 12:23:42 consumer-85969d5ccf-fnb2v start

    • 12:24:12 consumer-85969d5ccf-fnb2v connected

    • 12:24:14 consumer-85969d5ccf-fnb2v sigterm

    • 12:24:17 consumer-85969d5ccf-fnb2v closed by=consumer

    • 12:24:17 consumer-85969d5ccf-fnb2v exit

    • last successful tick 12:24:12

    • The replacement came 57 s before RabbitMQ deleted the old user, and the consumer was on the new user 16 s before it (renewal at 33%, a 30 s startup).

    Verdict: the cycle is clean: the consumer ran on the new user 16s before RabbitMQ dropped the old one, and no connection was refused or closed by the broker. The dynamic credential holds.

    Done when

    • The mount's real lease lifetime is recorded.
    • One renewal and one replacement are observed, with the consumer's log of each.
    • The result is posted here.
    • Nothing the drill created is left in Vault, RabbitMQ or the cluster.

  6. ExtraToast commented on Oct 29, 2026

    @ExtraToast
    ExtraToast
    on Oct 29, 2026
    ContributorAuthor

    Decision on the result

    The maintainer keeps rabbitmq in the grant union as a dynamic credential, rotated early (2026-10-29).

    • At VSO's default renewalPercent: 67, the replacement raced the old user's deletion: the broker cut the old connection the second the restarted consumer connected. A slower consumer would lose its connection first.
    • At renewalPercent: 33, the replacement came 57 s before the deletion, and a consumer that waits 30 s before connecting was on the new user 16 s before it, with nothing refused or force-closed (result).

    So when chapter 17 (secrets and access) is rewritten and adds the rabbitmq arm, its rendered VaultDynamicSecret sets renewalPercent low enough for its consumers' restart to finish before the old lease ends; 33 is the value the drill proved. The findings are in docs/research/unsupported-claims.md, C7 to C9.

  7. github-actions
    added
    status: in-progressActively being worked.
    on Oct 29, 2026
  8. ExtraToast
    closed this as completedin #684on Oct 29, 2026
  9. github-actions
    removed
    status: in-progressActively being worked.
    on Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.component: securitySecurity, permissions, secrets, or vulnerability handling.priority: P2Medium; normal planned work.type: choreMaintenance work without intended behavior change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions