Merge main into release/v2.4.x - #828
Merged
Merged
Conversation
AIWB reads OPENBAO_TOKEN once at import and builds its OpenBao client at startup, so the value is snapshotted when the pod starts. The scoped token was minted with -ttl=768h and re-minted (and the old one revoked) once it expired, which stranded the running aiwb-api pod on a dead credential: every API-key create returned 500 from a 403 on secrets/data/apikeys/*, until someone restarted the deployment. Observed on app-dev. Mint the token periodic and renew it on each run so the stored value stays constant and nothing has to notice a rotation. A non-periodic token cannot be renewed past the system max TTL, so renewal alone would not have helped. Re-mint remains the fallback when the token is absent or no longer renewable, which also migrates the existing non-periodic token on its next expiry. The policy gains auth/token/renew-self, matching read-policy which already grants it to the external-secrets identity.
The ai-gateway mutating webhook took its serving cert from the chart's built-in
Helm genCA path, which upstream's own values.yaml describes as "not recommended
for production use". On int-test the CA advertised in the
MutatingWebhookConfiguration stopped matching the cert the controller served, and
pod admission began failing with:
x509: certificate signed by unknown authority ("ai-gateway-controller-ca")
Because the webhook is failurePolicy: Fail and selects envoy-gateway managed
pods, the ai-gateway proxy Deployment could no longer create pods. It sat at
UP-TO-DATE 0 for 14h, still serving from a ReplicaSet created during cluster
bring-up whose Envoy configuration was stale, and every inference request
returned an empty 500 regardless of whether a valid key, an invalid key, or no
key was supplied. Recovery required deleting the webhook by hand so the
Deployment could roll.
Exactly how the CA and the served cert diverged is not established; the failure
mode is what matters here. The genCA path leaves two copies of the trust
material that must agree - the caBundle written into the webhook object, and the
cert in the Secret the controller mounts - with nothing keeping them consistent
once either is rewritten.
Enable the chart's cert-manager path instead. The Certificate and Issuer are
declarative, and ca-injector maintains the caBundle from the issued cert, so the
two cannot drift apart. Both paths populate the same Secret the controller
mounts (self-signed-cert-for-mutating-webhook), and cert-manager is already
installed at syncWave -40, ahead of this app at -5.
Renew AIWB OpenBao token instead of rotating it
cluster-auth no longer authenticates anything. The AIM serving routes are covered by the route-scoped workloads-extauth policy and ai-gateway by its gateway-scoped default-deny, and neither the airm nor the aiwb chart has a live reference to it. On app-dev its own counters show 230k checks over 15 days, all fast_path with authenticator=none and zero denies. - drop the cluster-auth and cluster-auth-config apps from root/values.yaml and from enabledApps in all three sizings, plus the medium resources override - delete the cluster-auth (0.5.0, 0.5.9) and cluster-auth-config chart sources - delete the cluster-auth ext_authz SecurityPolicy and its ReferenceGrant - stop writing the OpenBao root token to secrets/cluster-auth-openbao-token, and drop the now-inert INIT_MODE export that only drove that write - drop the security-policy-extauth patch block from the OpenShift installer, which would have aborted the install under set -e once the template is gone - refresh the SBOM and the stale comments that referenced the ext_authz hop
[EAI-7305]: Remove cluster-auth from the stack
Openshift custom values feature
Issue the ai-gateway webhook cert via cert-manager
Rebase onto main after #824 (cert-manager webhook TLS). Drop chart template caBundle prevention; keep operator heal path, post-handoff platform gates, CI install-script checks, and OpenShift install wiring. Co-authored-by: Cursor <[email protected]>
* feat(extra-apps): Add blueprint for vllm-sr Example blueprint for adding vllm-sr as an extra app. The config for it is just a placeholder for demonstrating the idea. The user is required to just use these as reference and implement these in the per cluster cluster-values repo. * redo docs * fix(dashboard): update dashboard docs and router access * feat: retries and timeouts * more retry docs
feat(EAI-8292): webhook heal script and platform gates (complements #824)
pre
approved these changes
Aug 28, 2026
pre
left a comment
pre
left a comment
Contributor
There was a problem hiding this comment.
OK but there's the related discussion about treating the release branch as a stable point in time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recent changes from main into the release branch