Bump kube-prometheus-stack to 92.10.0 and Thanos to v0.42.4 by riadelhajjaji-scality · Pull Request #5174 · scality/metalk8s

riadelhajjaji-scality · 2026-10-16T16:13:50Z

Component: charts, salt, buildchain

Context:
Regular dependency bump of the monitoring stack. kube-prometheus-stack
moves from 86.3.1 to 92.10.0 (six major versions) and Thanos from
v0.41.0 to v0.42.4. The bitnami thanos chart has no newer release than
17.3.1.

Summary:

Bump the kube-prometheus-stack chart to 92.10.0 and its images:
prometheus-operator/config-reloader v0.94.1, prometheus
v3.15.0-distroless, alertmanager v0.34.1, grafana 13.2.4-distroless,
k8s-sidecar 2.14.1, kube-state-metrics v2.20.0, node-exporter
v1.12.1-distroless
Bump Thanos to v0.42.4, and the k8s-sidecar pin of the Loki chart
Grafana chart 14 validates values against a schema: set the Grafana
replicas with a codegen --patch (same as Loki)
Control-plane ServiceMonitors now authenticate with a
prometheus-operator-prometheus-token Secret: replace_object keeps
the data of service account token Secrets, so a highstate no longer
wipes the token filled in by the token controller
Drop the insecureSkipVerify overrides removed by chart 90 (no
rendered change)

Acceptance criteria:

Build, install and upgrade CI pass
All Prometheus targets are up, including kubelet, apiserver, etcd,
scheduler, controller-manager and kube-proxy
prometheus-operator-prometheus-token has its data filled in, and
a second highstate does not change the token
Grafana starts and shows the dashboards and datasources
tools/rule_extractor/*.json updated from the CI alert_rules artifact

Closes: #MK8S-349

Bump the kube-prometheus-stack chart from 86.3.1 to 92.10.0 and Thanos to v0.42.4. The bitnami thanos chart stays at 17.3.1, it has no newer release. Images bumped accordingly: - prometheus-operator: v0.91.0 -> v0.94.1 - prometheus-config-reloader: v0.91.0 -> v0.94.1 - prometheus: v3.12.0-distroless -> v3.15.0-distroless - alertmanager: v0.33.0 -> v0.34.1 - grafana: 13.0.2 -> 13.2.4-distroless - k8s-sidecar: 2.7.3 -> 2.14.1 - kube-state-metrics: v2.19.1 -> v2.20.0 - node-exporter: v1.11.1-distroless -> v1.12.1-distroless - thanos: v0.41.0 -> v0.42.4 The k8s-sidecar pin of the Loki chart follows, as BUMPING.md asks. Breaking changes that impact MetalK8s: - Grafana chart 14 validates values against a schema, and rejects the string `__var__` placeholder we used for `grafana.replicas`. Set it with a `--patch` in the codegen task instead, as done for Loki. - Chart 90 moves the control-plane ServiceMonitors from token and CA files to a `prometheus-operator-prometheus-token` Secret and the `kube-root-ca.crt` ConfigMap. Salt replaces objects on every apply, which would wipe the token filled in by the token controller, so `replace_object` now keeps the data of service account token Secrets. - Chart 90 removes `serviceMonitor.insecureSkipVerify`, now a default, for the scheduler and controller-manager. Drop our overrides, the rendered ServiceMonitors do not change. - Grafana chart 13 runs Grafana from the distroless image with a read-only root filesystem. We set no `GF_*__FILE` or plugin env var, logs go to the console and every write path is an emptyDir. Other changes reviewed, no action needed: - Chart 92 adds `kubernetes.io/os: linux` to the nodeSelector of the operator, Prometheus and Alertmanager; pods restart once on upgrade. - Chart 91 tightens the operator RBAC. - Operator 0.93 keeps compaction when the Thanos sidecar uploads to object storage; we do not upload, `objectStorageConfig` is empty. - Alertmanager 0.34 splits `authError` and `rateLimited` out of `reason="clientError"`; no rule or dashboard of ours matches on it. - kube-state-metrics 2.20 only emits `kube_pod_status_reason` for the reason that is set; our rule compares `== 1`, unaffected. - Thanos 0.42 breaking changes only touch Receive and Store flags. - k8s-sidecar 2.9 moves from CMD to ENTRYPOINT; we override no sidecar command. Ref: MK8S-349

bert-e · 2026-10-16T16:13:54Z

Hello riadelhajjaji-scality,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/bypass_source_branch_lineage`	Bypass the cross-branch contamination check	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request.
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-10-16T16:13:59Z

Foreign commits detected in source branch

The source branch feature/MK8S-349-bump-kube-prometheus-stack shares history with the following
release line(s), which are not ancestors of development/134.0:

development/136.0

This typically happens when the feature branch was accidentally based on
commits from a higher release line rather than directly on development/134.0.
Common causes include:

Rebasing on a Bert-E integration branch (e.g. a w/ branch) instead of
directly on development/134.0
Branching from a higher development branch instead of development/134.0
Merging a higher development branch into the feature branch

How to fix

Create a new branch directly from development/134.0 and cherry-pick your
changes onto it:

git checkout -b <new-branch-name> origin/development/134.0
git cherry-pick <your-commits>

Then open a new pull request from that branch.

If this is a false positive

If your branch is a legitimate backport and was previously merged into one
of the branches above before being extended with new commits, an
administrator can bypass this check with:

@bert-e bypass_source_branch_lineage

bert-e · 2026-10-16T16:14:48Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

claude · 2026-10-16T16:16:27Z

The control-plane ServiceMonitors now send a non-expiring service account token, and etcd, kube-proxy and CoreDNS get it over plain HTTP
- Set serviceMonitor.authorization: null for kubeEtcd, kubeProxy and coreDns in charts/kube-prometheus-stack.yaml, then regenerate chart.sls
- No changelog entry for this bump: .changes/unreleased/Enhanced-20260727-091019-556322166.yaml still lists chart 86.3.1, Thanos v0.41.0 is not covered, and neither is the new long-lived token Secret
- Update that entry, or add one with a /changie comment

Review by Claude Code

riadelhajjaji-scality · 2026-10-16T16:20:31Z

/changie Enhanced Bump kube-prometheus-stack chart version to 92.10.0
The following images have also been bumped accordingly:

alertmanager to v0.34.1
grafana to 13.2.4
k8s-sidecar to 2.14.1
kube-state-metrics to v2.20.0
node-exporter to v1.12.1
prometheus to v3.15.0
prometheus-config-reloader to v0.94.1
prometheus-operator to v0.94.1
thanos to v0.42.4

…s://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.10.0) Auto-generated-by: changie-add-changelog

kube-prometheus-stack 92.10.0 adds the AlertmanagerClusterFailedPeers alert and the "Kubernetes / Compute Resources / Nodes Overview" dashboard, and changes the KubePdbNotEnoughHealthyPods and KubePodNotReady queries. Update the expected alert rules from the CI alert_rules artifact, and add the new dashboard to the expected dashboard UIDs. Ref: MK8S-349

bert-e · 2026-10-19T08:44:10Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

Peer approvals must include at least 1 approval from the following list:

claude · 2026-10-19T08:46:13Z

LGTM

Review by Claude Code

Document in BUMPING.md that the Thanos image tag is pinned in charts/thanos.yaml and that the expected Grafana dashboards must follow the chart. Ref: MK8S-349

Since chart 90, the control-plane ServiceMonitors authenticate with the non-expiring prometheus-operator-prometheus-token Secret. The etcd, kube-proxy and CoreDNS metrics endpoints are scraped over plain HTTP and do not check the token, so sending it there only exposes it to anyone sniffing node traffic. Set their `authorization` to null. Ref: MK8S-349

github-actions · 2026-10-19T14:51:45Z

✅ Changelog check passed.

claude · 2026-10-19T14:53:57Z

LGTM

Review by Claude Code

ezekiel-alexrod

LGTM

riadelhajjaji-scality requested review from a team as code owners October 16, 2026 16:13

riadelhajjaji-scality requested review from TeddyAndrieux and ezekiel-alexrod and removed request for a team October 16, 2026 16:13

riadelhajjaji-scality changed the base branch from development/134.0 to development/136.0 October 16, 2026 16:14

claude Bot reviewed Oct 16, 2026

View reviewed changes

Comment thread salt/metalk8s/addons/prometheus-operator/deployed/chart.sls Outdated

riadelhajjaji-scality changed the title ~~Feature/mk8 s 349 bump kube prometheus stack~~ bump kube prometheus stack Oct 16, 2026

riadelhajjaji-scality changed the title ~~bump kube prometheus stack~~ Bump kube-prometheus-stack to 92.10.0 and Thanos to v0.42.4 Oct 16, 2026

riadelhajjaji-scality and others added 2 commits October 16, 2026 16:20

changelog: Bump kube-prometheus-stack chart version to [92.10.0](http…

bf17fe0

…s://github.com/prometheus-community/helm-charts/releases/tag/kube-prometheus-stack-92.10.0) Auto-generated-by: changie-add-changelog

riadelhajjaji-scality added 2 commits October 19, 2026 16:50

doc: document Thanos tag and dashboard UIDs bump

1da9adf

Document in BUMPING.md that the Thanos image tag is pinned in charts/thanos.yaml and that the expected Grafana dashboards must follow the chart. Ref: MK8S-349

ezekiel-alexrod approved these changes Oct 19, 2026

View reviewed changes

Navigation Menu

Repository navigation

Bump kube-prometheus-stack to 92.10.0 and Thanos to v0.42.4 - #5174

Bump kube-prometheus-stack to 92.10.0 and Thanos to v0.42.4#5174
riadelhajjaji-scality wants to merge 5 commits into
development/136.0from
feature/MK8S-349-bump-kube-prometheus-stack

riadelhajjaji-scality commented Oct 16, 2026

Uh oh!

bert-e commented Oct 16, 2026

Uh oh!

bert-e commented Oct 16, 2026

Uh oh!

bert-e commented Oct 16, 2026

Uh oh!

Uh oh!

claude Bot commented Oct 16, 2026

Uh oh!

riadelhajjaji-scality commented Oct 16, 2026

Uh oh!

bert-e commented Oct 19, 2026

Uh oh!

claude Bot commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

claude Bot commented Oct 19, 2026

Uh oh!

ezekiel-alexrod left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

riadelhajjaji-scality commented Oct 16, 2026

Uh oh!

bert-e commented Oct 16, 2026

Hello riadelhajjaji-scality,

Uh oh!

bert-e commented Oct 16, 2026

Foreign commits detected in source branch

Uh oh!

bert-e commented Oct 16, 2026

Waiting for approval

Uh oh!

Uh oh!

claude Bot commented Oct 16, 2026

Uh oh!

riadelhajjaji-scality commented Oct 16, 2026

Uh oh!

bert-e commented Oct 19, 2026

Waiting for approval

Uh oh!

claude Bot commented Oct 19, 2026

Uh oh!

github-actions Bot commented Oct 19, 2026

Uh oh!

claude Bot commented Oct 19, 2026

Uh oh!

ezekiel-alexrod left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants