Skip to content

Drill a Flagger promotion of a Process spread across nodes #371

Closed

Description

@ExtraToast

What to do

Promote a new revision of a blue-green Process that is spread across nodes, with as many replicas as eligible nodes, and record whether the promotion completes. Decide which conditions on the settled decisions change the model derives matchLabelKeys: [pod-template-hash] on the across: node anti-affinity term, so that a surging rollout can place its new pod.

A drill for that ticket confirmed the mechanism on a plain Deployment (k3s v1.35.2+k3s1): without the key the surge pod stays Pending, and with it the rollout completes with one new pod per node. Flagger was not part of that drill. Flagger copies the Deployment into a -primary one and rewrites the anti-affinity selector of the copy (Flagger, FAQ). Whether the copy keeps matchLabelKeys, and whether its promotion then rolls, is unverified (docs/research/unsupported-claims.md, claim C21).

The drill writes to the live cluster, so it runs in a throwaway namespace that is deleted afterwards.

  1. Create a two-replica Deployment selected onto two nodes, with the required anti-affinity term on the hostname and matchLabelKeys: [pod-template-hash], and a Canary for it in the blue-green shape chapter 30 renders.
  2. Read the term on the -primary Deployment and on one of its pods.
  3. Change the template and let Flagger analyse and promote.
  4. Record where every pod ran during the promotion, and whether any pod was Pending.
  5. Delete the namespace.

Done when

  • The term Flagger writes on the primary is recorded.
  • One promotion is observed from start to finish, with pod placement at each step.
  • The result is posted here: either the promotion is clean, or the failure is described well enough to choose between the revision key and a rollout without surge.
  • Nothing the drill created is left in the cluster.

Notes

This runs before chapter 18 (Placement) is rewritten, because a failure reopens the choice. It sits outside the map Map: re-establish deploy-kit's goals, rewrite its spec, and order the work until the estate is handed over, which is not held up by it.

Activity

  1. added
    type: choreMaintenance work without intended behavior change.
    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.
    component: deployDeployment pipeline, manifests, or release rollout.
    priority: P2Medium; normal planned work.
    on Oct 12, 2026
  2. ExtraToast commented on Oct 29, 2026

    @ExtraToast
    ContributorAuthor

    Drill result: a Flagger promotion of a Process spread across nodes

    Run 371-20261029T112235Z of drills/flagger-spread-promotion.sh (estate 24afe79), against kubectl context personal, 2026-10-29 11:22–11:35 UTC.

    A two-replica Deployment held to enschede-pi-3 and enschede-pi-2, with a required anti-affinity term on kubernetes.io/hostname carrying matchLabelKeys: [pod-template-hash], under a blue-green Canary (provider: kubernetes, 2 iterations of 15 s, no webhooks, so the Release Gate took no part).

    The term on the primary

    Flagger's drill-primary keeps matchLabelKeys, and the promotion left it unchanged.

    {
      "requiredDuringSchedulingIgnoredDuringExecution": [
        {
          "labelSelector": {
            "matchLabels": {
              "app": "drill-primary"
            }
          },
          "matchLabelKeys": [
            "pod-template-hash"
          ],
          "topologyKey": "kubernetes.io/hostname"
        }
      ]
    }

    On its pod drill-primary-74cfdc4c74-tlpsz, after admission:

    {
      "requiredDuringSchedulingIgnoredDuringExecution": [
        {
          "labelSelector": {
            "matchExpressions": [
              {
                "key": "pod-template-hash",
                "operator": "In",
                "values": [
                  "74cfdc4c74"
                ]
              }
            ],
            "matchLabels": {
              "app": "drill-primary"
            }
          },
          "matchLabelKeys": [
            "pod-template-hash"
          ],
          "topologyKey": "kubernetes.io/hostname"
        }
      ]
    }

    The promotion

    It ended Succeeded after 6m12s. Where every pod was at each phase (app hash@node):

    time Canary pods
    +2:14 Initialized drill-primary 74cfdc4c74@enschede-pi-2 (ready), drill-primary 74cfdc4c74@enschede-pi-3 (ready)
    +2:29 Progressing drill 7974d478df@enschede-pi-2 (not ready), drill 7974d478df@enschede-pi-3 (not ready), drill-primary 74cfdc4c74@enschede-pi-2 (ready), drill-primary 74cfdc4c74@enschede-pi-3 (ready)
    +4:54 Promoting drill 7974d478df@enschede-pi-2 (ready), drill 7974d478df@enschede-pi-3 (ready), drill-primary 74cfdc4c74@enschede-pi-2 (ready), drill-primary 74cfdc4c74@enschede-pi-3 (ready), drill-primary 855f75bf5f@enschede-pi-2 (not ready)
    +8:26 Finalising drill 7974d478df@enschede-pi-2 (ready), drill 7974d478df@enschede-pi-3 (ready), drill-primary 855f75bf5f@enschede-pi-3 (ready), drill-primary 855f75bf5f@enschede-pi-2 (ready)
    +8:41 Succeeded drill-primary 855f75bf5f@enschede-pi-3 (ready), drill-primary 855f75bf5f@enschede-pi-2 (ready)
    +8:42 after drill-primary 855f75bf5f@enschede-pi-3 (ready), drill-primary 855f75bf5f@enschede-pi-2 (ready)

    No pod was ever unschedulable.

    Verdict: the promotion is clean: Flagger keeps the revision key on the primary, and the surge pod placed beside the old one.

    Done when

    • The term Flagger writes on the primary is recorded.
    • One promotion is observed from start to finish, with pod placement at each step.
    • The result is posted here.
    • Nothing the drill created is left in the cluster.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: deployhomelab-deploy, deploy-v2, manifests, and rollout flow.component: deployDeployment pipeline, manifests, or release rollout.priority: P2Medium; normal planned work.type: choreMaintenance work without intended behavior change.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions