EAI-8229 Fix CLUSTER_SIZE small: keep global.domain, hooks, drop gitea - #815
EAI-8229 Fix CLUSTER_SIZE small: keep global.domain, hooks, drop gitea#815pre wants to merge 3 commits into
Conversation
The root Application is rendered two times. bloom renders it one time
with helm --set, and ArgoCD then renders the same chart again to manage
the app-of-apps. The second render reads only the value files, so each
value that was given with --set and is not repeated as a parameter is
lost, and the root app overwrites its own spec without it.
global.domain and global.clusterSize were lost this way. An empty
domain made envoy-gateway-config write the CoreDNS rewrite
`regex .*\.`, which matches every name, so each name in the cluster
resolved to the gateway service. The ArgoCD controller could not reach
its repo-server, all 39 applications stayed in ComparisonError, and no
secret was made. An empty clusterSize dropped values_<size>.yaml from
the value files, so the cluster got the wrong set of applications.
Only the small path is affected. With externalValues.enabled the two
values come from cluster-values/values.yaml in the in-cluster Gitea.
To see the difference:
helm template cluster-forge root --show-only templates/cluster-forge.yaml \
--values root/values.yaml --values root/values_small.yaml \
--set global.domain=1.2.3.4.nip.io | grep -A1 global.domain
The certgen hooks carry "helm.sh/hook" but no delete policy. ArgoCD reads a missing policy as BeforeHookCreation, so it deletes each hook before it makes the hook again, and it does that each time it resumes the operation. The operation resumes about every four seconds, and the certgen job needs more time than that, so the job never started. envoy-gateway stayed at PreSync with "waiting for completion of hook", the Gateway API CRDs were never installed, and each application that uses a Gateway, an HTTPRoute or a TLSRoute could not sync. Measured on a one-node cluster: the operation was Running for 30 minutes and the namespace held no service account, no job and no pod. hook-succeeded keeps the hooks until the job completes, and removes them after it.
Test on a clean one-node clusterKaytoo VM, Ubuntu 24.04, no GPU, Correction 1, global.domain: confirmed
The CoreDNS rewrite is correct, and name resolution now goes both ways as Before the correction both names gave the gateway address. Correction 2, the PreSync hooks: confirmedBefore, The Gateway API CRDs are installed, and both gateways are programmed: Applications not This branch alone does not give a cluster that fully settlesThe run still failed, and the two remaining faults are NOT corrected here. a. gitea cannot start on the small path. bloom skips b. the AI gateway ext-auth port does not exist.
|
bloom does not bootstrap gitea when CLUSTER_SIZE is small. The task
clusterforge_setup.yaml holds `when: (CLUSTER_SIZE | default('medium'))
!= 'small'`, and bootstrap_gitea.yaml is the only thing that makes the
secret gitea-admin-credentials.
values_small.yaml enabled the gitea and gitea-config applications all the
same, so the pod waited for a secret that nothing makes. Measured on a
one-node cluster: `secret "gitea-admin-credentials" not found`, 255
restarts in 59 minutes, and the application stayed Degraded.
The small path reads the repository from GitHub, because
externalValues.enabled is false and clusterForge.repoUrl names GitHub.
It therefore has no use for the in-cluster Gitea, and nothing else in
values_small.yaml names it.
pre
changed the title
Clean-room test of this branchOne more test, on a machine that never had bloom on it, against the branch as it now stands (
The three faults this PR corrects1. The root application keeps CoreDNS got the name it must rewrite, not a wildcard: Applications in 2. The envoy-gateway PreSync hooks complete. The certgen job ran and left its output: The application is 3. gitea is not enabled. 0 gitea applications and 0 gitea pods. Before, the pod asked for Applications49 of 51 are What this test does NOT showThe cluster does not serve traffic yet. Two faults hold it, and this PR does not claim to correct either one.
|
1 participant
Fixes EAI-8229.
CLUSTER_SIZE: smalldoes not give a platform that works. Three faults stopit, one after the other. This pull request corrects all three.
global.domain, and CoreDNS then sends every name to the gatewayenvoy-gatewaynever leaves PreSync, so the Gateway API CRDs are never installedgiteawaits for a secret that nothing makes on this pathOnly the small path is affected. With
externalValues.enabled, the values offault 1 come from
cluster-values/values.yamlin the in-cluster Gitea.1. The root application loses global.domain
The root Application is rendered two times. bloom renders it with
helm template ... --set global.domain=...and applies the result. ArgoCD thenrenders the same chart again to manage the app-of-apps, and that render reads
only the value files.
root/templates/cluster-forge.yamlcopiedclusterForge.targetRevisionintospec.source.helm.parameters, but notglobal.domainand notglobal.clusterSize. The second render saw them empty, and the rootapplication overwrote its own specification without them.
An empty domain made
envoy-gateway-configwrite.*\.matches every name:10.243.148.243is theenvoy-gateway-system/httpsservice. The ArgoCDcontroller could not reach its repo-server, and all 39 applications stayed in
ComparisonError.An empty
global.clusterSizealso removedvalues_small.yamlfrom the valuefiles, so the cluster got the wrong set of applications. The count decreased
from 39 to 26.
Correction. Copy both values into the parameters, in the same way as
clusterForge.targetRevision. The render now gives the same result each time:2. envoy-gateway never leaves PreSync
The hooks in
sources/envoy-gateway/v1.8.1carriedhelm.sh/hookbut nohelm.sh/hook-delete-policy. ArgoCD reads a missing policy asBeforeHookCreation, so it deletes each hook before it makes the hook again,and it does that each time it resumes the operation. The operation resumes
about every four seconds, and the certgen job needs more time than that, so
the job never started.
Measured: the operation was
Runningfor 30 minutes, andenvoy-gateway-systemheld no service account, no job and no pod. The GatewayAPI CRDs were never installed, so each application that uses a
Gateway, anHTTPRouteor aTLSRoutefailed withTen applications stayed
OutOfSync/Missing, among themenvoy-gateway-config,keycloak,gitea-config,openbao-configandseaweedfs-config.Correction.
hook-delete-policy: hook-succeededon the seven hooks of thatchart. The hooks now stay until the job completes.
3. gitea cannot start on the small path
bloom does not bootstrap gitea when
CLUSTER_SIZEis small.clusterforge_setup.yamlholdswhen: (CLUSTER_SIZE | default('medium')) != 'small', andbootstrap_gitea.yamlis the only thing that makes the secretgitea-admin-credentials.values_small.yamlenabled thegiteaandgitea-configapplications all the same.Measured:
secret "gitea-admin-credentials" not found, 255 restarts in 59minutes, and the application stayed
Degraded.Correction. Do not enable the two applications on the small path. That path
reads the repository from GitHub, because
externalValues.enabledis false andclusterForge.repoUrlnames GitHub, so it has no use for the in-cluster Gitea.Nothing else in
values_small.yamlnames it.Test
One Kaytoo VM, Ubuntu 24.04, no GPU,
CLUSTER_SIZE: small,DOMAIN: <ip>.nip.io,CERT_OPTION: generate. Installed with bloom, thencluster-forge from this branch.
clusterForge.targetRevisiononlyglobal.domain,global.clusterSizevalues.yamlvalues.yaml,values_small.yamlComparisonErrorSynced+HealthyName resolution now goes both ways, as intended:
envoy-gateway installs and both gateways are programmed:
What stays open
One application,
envoy-gateway-config, is stillOutOfSync, and it is notcorrected here.
security-policy-ai-gateway-default-deny.yamlasks for port8083 on
ai-gateway-system/ai-gateway-discovery, and envoy-gateway refuses thepolicy:
ai-gateway-discovery-chart:2.0.0gives one port,health:8081, and thecontainer gives only
health:8081, so there is no ext-auth port. This is amismatch of versions and not a wrong number, and port 8081 is the health probe,
so no correction is guessed here.
EAI-8231 records it.