Skip to content

Bump operators to k8s 0.36.5 and Go 1.26 - #5154

Open
riadelhajjaji-scality wants to merge 5 commits into
development/136.0from
feature/MK8S-343-bump-k8s-package
Open

riadelhajjaji-scality wants to merge 5 commits into
development/136.0from
feature/MK8S-343-bump-k8s-package

Conversation

@riadelhajjaji-scality

@riadelhajjaji-scality riadelhajjaji-scality commented Oct 12, 2026 •

Copy link
Copy Markdown

Component: operator, storage-operator, build

Context:

metalk8s-operator and storage-operator are built with Go 1.24,
k8s.io libraries 0.33.13 and controller-runtime 0.21.0. These
versions are getting old (Go 1.24 is out of support), and keeping
them up to date eases future Kubernetes upgrades.

The operators are regenerated with tools/upgrade-operator-sdk
(operator-sdk v1.42.3), but this operator-sdk scaffolds older
k8s.io libraries and controller-runtime, so the script needs to
keep versions pinned ahead of the scaffold.

Summary:

  • Bump, for both metalk8s-operator and storage-operator:
    • Go toolchain from go1.24.13 to go1.26.9 (go 1.26.0 in go.mod,
      builder image golang:1.26)
    • k8s.io libraries (api, apimachinery, client-go) from
      0.33.13 to 0.36.5
    • controller-runtime from 0.21.0 to 0.24.1
    • controller-tools (controller-gen) from v0.18.0 to v0.21.0
    • golangci-lint from v2.1.0 to v2.14.0
    • ginkgo 2.22.0 -> 2.27.4, gomega 1.36.1 -> 1.39.0, and
      indirect dependencies
  • Regenerate the CRDs and deploy/manifests.yaml: only the
    controller-gen version annotation and field descriptions
    inherited from the new k8s.io API types change (e.g. Lt/Gt
    toleration operators behind a feature gate), no schema change
  • tools/upgrade-operator-sdk/upgrade.py:
    • keep a version pinned in config.yaml when it is ahead of what
      operator-sdk scaffolds, instead of falling back to the scaffold
      version
    • new optional pins controller_runtime, controller_tools and
      golangci_lint, controller-runtime being pinned together with
      the k8s.io libraries, as both must share the same minor
    • the Dockerfile builder image now follows the Go toolchain
      (e.g. go1.26.9 -> golang:1.26)

Acceptance criteria:

  • The CI passes: operators build, lint and unit tests, and the
    installation, expansion, upgrade and downgrade tests
  • Both operators run and reconcile their resources
    (ClusterConfig, VirtualIPPool, Volume) on a cluster
  • tools/upgrade-operator-sdk/upgrade.py regenerates both operators
    with no diff from this PR

Closes: #MK8S-343

Note:
These libraries target Kubernetes 1.36, this should be merged after
the Kubernetes 1.36 bump (#5146).

@bert-e

bert-e commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Hello riadelhajjaji-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@bert-e

bert-e commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Foreign commits detected in source branch

The source branch feature/MK8S-343-bump-k8s-package shares history with the following
release line(s), which are not ancestors of development/134.0:

  • development/136.0

This typically happens when the feature branch was accidentally based on
commits from a higher release line rather than directly on development/134.0.
Common causes include:

  • Rebasing on a Bert-E integration branch (e.g. a w/ branch) instead of
    directly on development/134.0
  • Branching from a higher development branch instead of development/134.0
  • Merging a higher development branch into the feature branch

How to fix

Create a new branch directly from development/134.0 and cherry-pick your
changes onto it:

git checkout -b <new-branch-name> origin/development/134.0
git cherry-pick <your-commits>

Then open a new pull request from that branch.

If this is a false positive

If your branch is a legitimate backport and was previously merged into one
of the branches above before being extended with new commits, an
administrator can bypass this check with:

@bert-e bypass_source_branch_lineage

@riadelhajjaji-scality
riadelhajjaji-scality changed the base branch from development/134.0 to development/136.0 October 12, 2026 11:14
@bert-e

bert-e commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

Peer approvals must include at least 1 approval from the following list:

@riadelhajjaji-scality riadelhajjaji-scality self-assigned this Oct 12, 2026
@riadelhajjaji-scality riadelhajjaji-scality added the go Pull requests that update Go code label Oct 12, 2026
@riadelhajjaji-scality riadelhajjaji-scality changed the title Feature/mk8 s 343 bump k8s package Feature/mk8s 343 bump k8s package Oct 12, 2026
@riadelhajjaji-scality

riadelhajjaji-scality commented Oct 12, 2026 •

Copy link
Copy Markdown
Author

/changie Enhanced
Bump k8s Go libraries of metalk8s-operator and storage-operator to 0.36.5 (Go 1.26, controller-runtime 0.24.1)

@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as ready for review October 12, 2026 13:22
@riadelhajjaji-scality
riadelhajjaji-scality requested a review from a team as a code owner October 12, 2026 13:22
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as draft October 12, 2026 14:04
@riadelhajjaji-scality
riadelhajjaji-scality marked this pull request as ready for review October 12, 2026 14:52
Comment thread operator/Makefile

# Force Go toolchain version
export GOTOOLCHAIN = go1.24.13
export GOTOOLCHAIN = go1.26.9

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go.mod refers to go 1.26.0, why using 1.26.9 here ?

@riadelhajjaji-scality riadelhajjaji-scality Oct 15, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same split we had before (go 1.24.0 in go.mod with GOTOOLCHAIN = go1.24.13)
Here the explaination:

  • The go directive(go 1.26.0 in go.mod) is the minimum Go version required to build the module, and since Go 1.21 it's a hard requirement that propagates to every module importing ours (go mod tidy bumps their go line to match). 1.26.0 is what the k8s.io 0.36 libraries require and we don't use anything introduced in later patches, so raising it to 1.26.9 would only force consumers (e.g. anyone importing the operator/api types) to upgrade for no benefit. This follows upstream convention: k8s.io and controller-runtime also declare x.y.0.

  • GOTOOLCHAIN = go1.26.9 in the Makefile is a different thing: it pins the exact compiler used to build our binaries, so builds are reproducible across dev machines and CI and include the latest security fixes (1.26.1 → 1.26.9). It has no effect on modules that depend on us.

Comment thread storage-operator/Makefile

# Force Go toolchain version
export GOTOOLCHAIN = go1.24.13
export GOTOOLCHAIN = go1.26.9

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above

@bert-e

bert-e commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

Peer approvals must include at least 1 approval from the following list:

@bert-e

bert-e commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Branches have diverged

This pull request's source branch feature/MK8S-343-bump-k8s-package has diverged from
development/136.0 by more than 50 commits.

To avoid any integration risks, please re-synchronize them using one of the
following solutions:

  • Merge origin/development/136.0 into feature/MK8S-343-bump-k8s-package
  • Rebase feature/MK8S-343-bump-k8s-package onto origin/development/136.0

Note: If you choose to rebase, you may have to ask me to rebuild
integration branches using the reset command.

Comment thread tools/upgrade-operator-sdk/upgrade.py
Comment thread tools/upgrade-operator-sdk/upgrade.py Outdated
Comment thread operator/go.mod
Comment thread operator/go.mod
@riadelhajjaji-scality
riadelhajjaji-scality requested review from a team, eg-ayoub and ezekiel-alexrod and removed request for a team October 16, 2026 08:36
@riadelhajjaji-scality riadelhajjaji-scality changed the title Feature/mk8s 343 bump k8s package Bump operators to k8s 0.36.5 and Go 1.26 Oct 16, 2026
@riadelhajjaji-scality
riadelhajjaji-scality requested review from a team as code owners October 16, 2026 11:22
Comment thread tools/upgrade-operator-sdk/upgrade.py
@claude

claude Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor
  • reconcile_versions compares versions as strings, so a two-digit patch (go1.26.10, v0.36.10) is reported as older than the pin. The new "ahead of scaffold, keeping pin" branch then logs the wrong message and hides the "newer available" warning.
    - Compare parsed numeric tuples instead of raw strings

    Review by Claude Code

@riadelhajjaji-scality
riadelhajjaji-scality force-pushed the feature/MK8S-343-bump-k8s-package branch from 36d63e7 to cd29b57 Compare October 16, 2026 11:24
@claude

claude Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor

LGTM

Review by Claude Code

else:
log_warn(f" {key}: pinned {pinned} > detected {found}, using detected")
cfg[key] = found
log_warn(f" {key}: pinned {pinned} ahead of scaffold {found}, keeping pin")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bot thread on this line was resolved, but found > pinned (l.332) is still a string compare. 'go1.26.10' > 'go1.26.9' is False, so we land here with pinned go1.26.9 ahead of scaffold go1.26.10 and never see the newer patch. The pin is kept either way, only the warning is wrong. Comparing the numbers fixes it (tested locally, black 26.3.1 OK):

        elif tuple(map(int, re.findall(r"\d+", found))) > tuple(
            map(int, re.findall(r"\d+", pinned))
        ):

Comment on lines +8 to +9
go_toolchain: go1.26.9
k8s_libs: v0.36.5

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: 220e15da4 only bumps these 2 lines, and the first commit already does the same for operator/config.yaml. I'd squash it into the first commit.

Comment on lines +340 to +343
def _minor(version: str) -> str:
"""Return the `vX.Y` part of a `vX.Y.Z` version, or empty string."""
m = re.match(r"(v\d+\.\d+)\.", version)
return m.group(1) if m else ""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: _latest_k8s_patch (l.239) has the same regex. Maybe use _minor() there too?

riadelhajjaji-scality and others added 4 commits October 19, 2026 12:16
Bump, for both metalk8s-operator and storage-operator:
- Go toolchain from go1.24.13 to go1.26.9 (`go 1.26.0` in go.mod,
  builder image golang:1.26)
- k8s.io libraries (api, apimachinery, client-go) from 0.33.13 to
  0.36.5
- controller-runtime from 0.21.0 to 0.24.1
- controller-tools (controller-gen) from v0.18.0 to v0.21.0
- golangci-lint from v2.1.0 to v2.14.0
- ginkgo from 2.22.0 to 2.27.4, gomega from 1.36.1 to 1.39.0, and
  indirect dependencies

The go directive stays on 1.26.0, the minimum version required by
the k8s.io 0.36 libraries, as it is imposed on modules importing ours.
GOTOOLCHAIN pins the exact compiler used to build our binaries.

The operators are regenerated with tools/upgrade-operator-sdk, the
go_toolchain and k8s_libs pins are updated in both config.yaml.

The CRDs and deploy/manifests.yaml only change the controller-gen
version annotation and field descriptions inherited from the k8s.io
0.36 API types (e.g. Lt/Gt toleration operators behind a feature
gate), there is no schema change.

Refs: MK8S-343
…operator` to 0.36.5 (Go 1.26, `controller-runtime` 0.24.1)

Auto-generated-by: changie-add-changelog
operator-sdk v1.42.3 still scaffolds k8s.io 0.33 and
controller-runtime 0.21, so a k8s_libs or go_toolchain pin ahead of
the scaffold was replaced by the scaffold version and the bump was
lost. Keep the pinned value instead, and compare versions
numerically, as a string compare sorts go1.26.10 before go1.26.9.

Add optional controller_runtime, controller_tools and golangci_lint
pins in config.yaml, controller-runtime having to follow the k8s.io
libs minor. The Dockerfile builder image now follows go_toolchain.

Refs: MK8S-343
Warn when the controller-runtime version (pinned or from the
scaffold) requires a different k8s.io minor than `k8s_libs`, by
reading its go.mod on the Go module proxy, so a k8s_libs bump
cannot silently leave controller-runtime behind.

Document in BUMPING.md that pins ahead of the scaffold are kept,
the new controller_runtime, controller_tools and golangci_lint
keys, and this check.

Refs: MK8S-343
@riadelhajjaji-scality
riadelhajjaji-scality force-pushed the feature/MK8S-343-bump-k8s-package branch from cd29b57 to 0614c75 Compare October 19, 2026 10:16
@bert-e

bert-e commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

Waiting for approval

The following approvals are needed before I can proceed with the merge:

  • the author

  • 2 peers

Peer approvals must include at least 1 approval from the following list:

else:
log_warn(f" {key}: pinned {pinned} > detected {found}, using detected")
cfg[key] = found
log_warn(f" {key}: pinned {pinned} ahead of scaffold {found}, keeping pin")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With pins ahead of the scaffold, this branch is now the normal path, and the "newer patch available" warning can no longer fire for go_toolchain/k8s_libs: detect_latest_patches only looks up the latest patch of the scaffold minor (go1.24 / v0.33), so a go1.26.10 or v0.36.6 release is never reported. Consider detecting the latest patch of the pinned minor here (e.g. call _latest_go_patch/_latest_k8s_patch on the pin) and warning if it is newer.

— Claude Code

@claude

claude Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor
  • upgrade.py reconcile_versions: when go_toolchain/k8s_libs are pinned past the scaffold minor, newer patches of the pinned minor are never detected or warned about
    - Look up the latest patch for the pinned minor (_latest_go_patch / _latest_k8s_patch on the pin) and warn if it is newer

    Review by Claude Code

The latest Go and k8s.io patches are looked up for the scaffold minor
only (go1.24, v0.33). With go_toolchain and k8s_libs pinned ahead of
the scaffold, a newer patch of the pinned minor (e.g. go1.26.10) was
never reported. Look it up too and warn when it is newer than the pin.
The pin is still used.

Refs: MK8S-343
@github-actions

Copy link
Copy Markdown
Contributor

✅ Changelog check passed.

@claude

claude Bot commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

LGTM

Review by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go Pull requests that update Go code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants