gitops: workloads of the demo

Argo CD reconciles this repository. It contains every workload of the demo The Sovereign, Self-Healing Platform. The cluster preparation and the root Application are in the ansible repo. Read AGENTS.md before changing anything.

Request flow

client ──> Route maas-router (ansible) ──> Gateway openshift-ai-inference (ansible)
           │  AuthPolicy: API key per tier, sets x-team      ┐ component frontdoor
           │  TokenRateLimitPolicy: token budget per tier     ┘
           │  AuthPolicy also refuses /metrics (403)                ┘
           └──> LiteLLM (component litellm-router): efficiency gate, then privacy gate
                  ├── Presidio analyzer (component presidio): PII detection, EN + IT
                  ├── local model (component local-model): Qwen3.8 27B on GPU, or Qwen2.5 0.5B on CPU
                  ├── decision model (component decision-model, optional): DiffusionGemma on its own GPU
                  ├── external SOTA model (OpenAI compatible), fallback to the local model
                  ├ ─ traces (OTLP) ─> collector otel ─> Tempo (component observability)
                  └ ─ metrics :9091 <─ user workload Prometheus (ServiceMonitor litellm)

Traces and metrics stay in the cluster. See docs/observability.md.

Components

Component	Namespace	Sync wave	Objects
observability	observability	0	TempoMonolithic tempo (traces on a 10Gi volume, 72h, multi-tenancy openshift, tenant router), OpenTelemetryCollector otel (OTLP in, Tempo gateway out). Only with observability.enabled
secrets	maas-routing	0	ESO Password generator, ExternalSecrets for the API keys, SOTA key, classifier
local-model	local-models	1	ServingRuntime (copy of the RHOAI template), InferenceService, HardwareProfile (shown in the RHOAI dashboard), NetworkPolicies
decision-model	local-models	1	Only with decisionModel.enabled: ServingRuntime (vLLM structured-read and the example decision server), InferenceService dgemma-decision, HardwareProfile, ConfigMap (decision server script), NetworkPolicies. See "Decision model"
presidio	maas-routing	1	Deployment (2 replicas), PodDisruptionBudget, Service, NetworkPolicy (no egress)
litellm-router	maas-routing	2	ConfigMap (config + hook code + policies), Deployment (2 replicas), PodDisruptionBudget, Service (API port 80, metrics 9091), ServiceMonitor, NetworkPolicy (4000 from the gateway, 9091 from monitoring), Perses dashboards sovereign-at-a-glance (audience), routing-decisions (demo) and stack-operations (team) and their datasources (only with observability.enabled)
frontdoor	maas-routing	3	HTTPRoute, AuthPolicy (API key; refuses /metrics), TokenRateLimitPolicy (POST requests only)
routing-live-view	maas-routing	5	Live page of the routing decisions and of the namespace labels (repo routing-live-view): Deployment with the OpenShift oauth-proxy, Service, Route (reencrypt), ServiceAccount, Role to read the LiteLLM logs, cookie secret (ESO generator). See "Namespace policy"
ticketing-system	agentic-triage	1	Deployment (1 replica, SQLite on a PVC), Service, Route (incident dashboard); shared by every agent
prometheus-mcp-server	agentic-triage	2	MCPServer CR, ServiceAccount (bound by ansible to cluster-monitoring-view); shared by every agent
ticketing-mcp-server	agentic-triage	2	MCPServer CR; shared by every agent
ocp-mcp-server	agentic-triage	2	MCPServer CR, ServiceAccount (bound by ansible to the built-in view ClusterRole), read-only pods/logs/events tools; shared by every agent
triage-agent	agentic-triage	4	Off by default (legacy): ServiceAccount, own tier ExternalSecret, Deployment, Service, Route (Gradio chat UI). Use triage-agent-operator-cr instead
payments-buggy-app, rh-reserve-buggy-app, reverse-proxy-buggy-app	payments, rh-reserve, reverse-proxy-app	1	The same chart (chart: quarkus-buggy-app), once per monitored-application namespace, each with its own appName (see "Per-component value overrides" in AGENTS.md): Deployment, Service, ServiceMonitor, Route (target of the triage demo, injects random 500s/503s/latency)
triage-agent-payments, triage-agent-rh-reserve, triage-agent-reverse-proxy-app	payments, rh-reserve, reverse-proxy-app	4	The same chart (chart: triage-agent-operator-cr), once per monitored-application namespace, each with its own apiKeyTier (see "Per-component value overrides" in AGENTS.md): the TriageAgent CR and its own tier ExternalSecret. The triage-agent-operator (ansible repo) creates the ServiceAccount, Deployment, Service, Route and ConfigMaps from each CR. Never enable triage-agent (legacy) in the same namespace as one of these
ogx-alert-translator	agentic-triage	5	ServiceAccount, Deployment (KEDA owns the replicas), Service, HTTPScaledObject (KEDA HTTP add-on, 0 to 3 pods), one AlertmanagerConfig per monitored-application namespace (webhook to the interceptor for the Quarkus error-rate alert): a single Alertmanager bridge, shared by every agent, that forwards each alert to the /trigger endpoint of the agent in the alert's own namespace

Presidio and LiteLLM run 2 replicas each (value replicas of the component), so one pod or node can fail without stopping the router. A preferred pod anti-affinity puts the replicas on different nodes when it can; it never blocks scheduling. The token budgets per tier (TokenRateLimitPolicy) are shared by all pods: Limitador keeps one counter per tier. The SOTA token cap of the router (sota_token_budget in components/litellm-router/files/chain.yaml) is off (0): it would count in memory per pod, with no time window. The spend limit on SOTA is the quota of the provider on its API key.

Known problems of the platform components and their workarounds (for example the token budgets count POST requests only) are in docs/known-issues.md.

Values

The seed in the ansible repo sets these values on the root Application. All the defaults are in bootstrap/values.yaml.

Value	Example	Meaning
appsDomain	apps.cluster.example.com	Apps domain; the router answers on router.<appsDomain>
modelProfile	gpu	gpu: Qwen3.8 27B (NVFP4) on a GPU node; cpu: Qwen2.5 0.5B on CPU
sota.enabled	true	false = local-only mode (see below)
sota.apiBase	https://provider.example.com/v1	External OpenAI-compatible endpoint
sota.model	openai/<model-id>	LiteLLM model string of the external model
sota.servedMatch	<model-id>	Part of the served model id, used by the cost gate
sota.reasoning	false	false: the SOTA model answers without reasoning (see "Long answers and streaming"); true: the model decides
secretStore.enabled	false	true when the ClusterSecretStore exists (see below)
classifier.mode	local	C2 classifier of the privacy gate: local (the local model), external, off (see below)
observability.enabled	true	false: no Tempo, no collector, no traces (see "Observability")
decisionModel.enabled	false	true: the decision model on its own GPU node (see "Decision model"); needs modelProfile: gpu
namespacePolicy.scan	false	true: the router finds namespace names in the request text (router v0.11.0, see "Namespace policy")
namespacePolicy.hint	false	true: the router reads the namespaces that the agents send, and each agent sends them
sotaBudget.enabled	false	true: SOTA token budget per tier, with a small Redis (router v0.12.0, see "SOTA budget per tier")
triageAgentOperator.enabled	true	false: skip every triage-agent-operator-cr instance when the ansible repo did not install the TriageAgent CRD and the triage-agent-operator
namespaces.payments, namespaces.rhReserve, namespaces.reverseProxyApp	payments, rh-reserve, reverse-proxy-app	Monitored-application namespaces, each with its own buggy app (payments-buggy-app, rh-reserve-buggy-app, reverse-proxy-buggy-app) and its own agent

Observability

The routing decisions are visible as traces in the console (Observe → Traces) and as metrics (Observe → Metrics). Everything stays in the cluster.

• Traces: component observability. The OpenTelemetry collector otel receives OTLP on otel-collector.observability.svc:4318 (http) and :4317 (grpc) and writes to the Tempo instance tempo. Tempo runs with multi-tenancy in openshift mode, the supported setup on OpenShift: the collector writes the tenant router with its service account token. The ansible repo installs the operators, grants that write permission and adds the console plugin. Reading the traces needs the read permission on the tenant (cluster-admin has it).
• Metrics: user workload monitoring is turned on by the ansible repo. The ServiceMonitor litellm scrapes port 9091 of LiteLLM: router metrics (router_requests_total, router_privacy_score, router_sota_budget_used_tokens) and LiteLLM metrics (tokens, latency, fallbacks per model). The public route refuses /metrics (AuthPolicy, 403). RHOAI creates the ServiceMonitor of the local model (<model>-metrics, vLLM metrics on port 8080) by itself.
• Router traces: LiteLLM sends one trace per request to the collector (otel callback, only with observability.enabled): the proxy span, router.chain with one span per gate, and the model call. Prompt and answer text are in the model call span (LiteLLM default); only users with the read permission on the Tempo tenant can see them. Span tree, metrics and a test: docs/observability.md.
• observability.enabled: false (seed value, from observability_enabled in the ansible repo) removes the component. The rest of the platform works as before.

Support status: the collector and Tempo come from the Red Hat build of OpenTelemetry and the Tempo Operator. LiteLLM, which produces the router traces, is community software, not supported by Red Hat.

Long answers and streaming

• A SOTA model with reasoning and a large token budget can take a few minutes. LiteLLM waits sota.timeout seconds (default 300) for one SOTA call, and requestTimeout seconds (540) for one call to the local model (components/litellm-router/values.yaml). A timeout is not retried; errors 500, 502, 503, 504 are retried once (retryPolicy).
• Without streaming, no byte flows until the answer is complete, so every hop needs a long idle timeout. The ansible repo sets 10 minutes on the Route and on the AWS load balancer of the ingress (the AWS default is 60 s).
• Reasoning is off by default (sota.reasoning: false). With reasoning on, the tested SOTA model often spent the whole token budget on hidden reasoning and returned an empty answer, and its provider sends nothing while the model reasons, even with streaming. Set sota.reasoning: true (Ansible: sota_reasoning: true) to show a reasoning model; then give the calls a large token budget.
• Clients should use streaming ("stream": true): the text arrives while it is generated, and no idle timeout applies. LiteLLM adds the token usage at the end of every stream (always_include_stream_usage), so the token limits count streaming calls too.

Privacy classifier (C2)

The privacy gate scores each prompt with rules (regex, lexicons, Presidio NER). When the score is in the gray zone and the efficiency gate wants to send the prompt to the SOTA model, an LLM classifier gives an extra opinion. classifier.mode selects that LLM:

Mode	Classifier	Notes
local (default)	The local model (Qwen3.8 27B on GPU); with decisionModel.enabled, the decision model first (systemone backend, router v0.7.0), then Qwen3.8 as fallback	The prompt never leaves the cluster, also while it is classified. No secret needed
external	An external model	Settings from the secret store (classifier-provider-secret)
off	None	The rules alone decide

The classifier can only make a prompt more sensitive, never less. On an error or a timeout (8 s), the prompt is treated as sensitive and stays on the local model. With the decision model, an error or a timeout of the decision server first falls back to Qwen3.8; only when that fails too, the prompt stays local. The log reason names the backend that decided: systemone/[email protected](legal) (router v0.9.0 names the question) or fallback/[email protected].

With the decision model, Presidio NER (C1) is off: the LiteLLM Deployment sets NER_ENABLED=0 (router v0.10.0). On agent contexts Presidio read technical names as people or places, and benign requests stayed local; the decision model questions (person, family, health...) catch the personal data. The rules (regex, lexicons) still run. Presidio stays deployed, and the policy keeps ner.enabled: true for the setups without the decision model.

Decision model

With decisionModel.enabled the demo gets a second local model, with one job: typed decisions. DiffusionGemma 26B-A4B FP8-dynamic runs in the vLLM structured-read mode, behind the example decision server of vLLM. The server fills a fixed answer canvas, runs one denoising step and reads the probability of each answer, so the reply is a real probability with no free text to parse. Qwen3.8 and its node do not change.

Support status: the decision model runs on an unsupported preview image (registry.redhat.io/rhaii-preview/vllm-cuda-rhel9:diffusiongemma-jev), for prototypes and proofs of concept. The example decision server is planned as Developer Preview in Red Hat AI Inference Server 3.6 GA and as Technology Preview in 3.7 EA1; 3.7 GA is the path to general availability. Its API is not a vLLM API and may change. LiteLLM and Presidio are community software, not supported by Red Hat.

The predictor pod has two containers:

Container	Port	What
kserve-container	8080	The example decision server (files/structured_server.py, vLLM commit 1b3b88e, not in the image): /health, /v1/systemone, /v1/chat/completions, /v1/raw/chat/completions
vllm	8000	vllm serve with the structured-read mode, one GPU, /metrics

The decision server answers /health with 200 even when vLLM is not up, so the readiness probe of the pod asks vLLM (127.0.0.1:8000/health). The HardwareProfile has no GPU identifier: the RHOAI webhook would add the GPU to kserve-container as well, and the pod would ask for two GPUs.

Clients call http://dgemma-decision-predictor.local-models.svc.cluster.local/v1/systemone. The router namespace can reach it; add other namespaces (for example the self-heal agents) with decisionModel.extraIngressNamespaces. Example:

curl -s http://dgemma-decision-predictor.local-models.svc.cluster.local/v1/systemone \
  -H 'content-type: application/json' -d '{
  "model": "dgemma",
  "state": {"service": "checkout-api", "symptom": "HTTP 503 rate rose from 0.1% to 23% after a deploy"},
  "questions": {
    "urgent": {"type": "noul", "instructions": "Does this need response within 15 minutes?"},
    "needs_human": {"type": "noul", "instructions": "Should human review before automated action?"}}}'
# {"model": "dgemma", "answers": {"urgent": {"type": "noul", "noul": 0.99}, "needs_human": {...}}, "usage": ..., "diagnostics": ...}

Measured on 2026-09-30 on one L40S (g6e.2xlarge): vLLM ready 5 min 11 s after the pod start (weights 207 s), a call with three yes/no questions about 350 ms for a short prompt (134 ms with "samples": 1), about 570 ms for 3,800 tokens and 5.8 s for 40,000 tokens. The first call after the start takes about 30 s (warm-up).

Local-only mode

The external SOTA model is optional. Its settings and key come from the ansible-vault of the ansible repo. Without them the seed sets sota.enabled: false, and the router runs in local-only mode: the alias sota-smart points to the local model, so the gates and policies work as usual, but every request is served by the local model. The router logs still show routed_to: sota-smart when a gate chooses "SOTA".

Namespace policy

Since router v0.11.0 a request about a namespace labelled sovereign-selfheal.io/data-class=restricted stays on the local model, before the gates. No label or public keeps the normal routing. The ansible repo creates the namespaces with their labels (agentic-triage: public; payments, rh-reserve and reverse-proxy-app: set per namespace, for example restricted for payments) and gives the ServiceAccount maas-routing/litellm get/list/watch on namespaces. A label can change live:

oc label namespace payments sovereign-selfheal.io/data-class=public --overwrite

The router reads the labels again every 5 seconds (refresh_s in chain.yaml). Two switches of the root Application turn the policy on, both off by default:

• namespacePolicy.scan: the router finds the namespace names in the request text (PromQL namespace="payments", JSON and YAML namespace keys, payments.svc, -n payments).
• namespacePolicy.hint: the router reads the body field selfheal_namespaces that the agents send, and triage-agent sends it (env ROUTER_NAMESPACE_HINT; needs a triage-agent version with namespaces in /trigger). The field is removed before any model call.

The alert AppHighErrorRate exists once per monitored-application namespace (alerts.quarkusBuggyApp.namespaceRefs of the observability component: payments, rh-reserve, reverse-proxy-app) and keeps the namespace label (sum by (namespace)). ogx-alert-translator reads that label to pick which agent's /trigger endpoint to call (see "AI-driven triage demo"). The knowledge file of each agent writes <namespace> in its queries: the agent uses the namespace of the alert or of the question, and agentic-triage (public, the shared-services namespace) only when there is none. Do not write a restricted namespace in a fixed prompt: with the scan on, every request of the agent would stay local. Details: router repo, README "Namespace policy" and docs/namespace-policy.md (the contract for agents).

The page routing-live-view (Route routing-live-view-maas-routing.<appsDomain>, sign-in with the cluster users who can list namespaces) shows every decision while it happens, with the agent that asked (the agent name of its API-key tier in tiers, else <tier> key) and, for a tier with a SOTA budget, a tag such as SOTA 25k/80k (red when the budget is used); the labels of the demo namespaces; and a button that changes a label: the page checks first, with the token of the signed-in user, that this user may patch the namespace, then its ServiceAccount (allowed on the demo namespaces only, ansible repo) writes the label.

SOTA budget per tier

With sotaBudget.enabled (seed: sota_budget_enabled), each API-key tier with sotaTokens (in tiers, bootstrap/values.yaml) gets a budget of SOTA tokens per window (sotaBudget.windowSeconds, default 300). When a tier used it, the router (v0.12.0) keeps the requests of that tier on the local model: the agent goes on, without the 429 of the gateway. The gateway limit (tokens) still counts all the tokens and stays the ceiling, so keep sotaTokens below tokens.

Tier	Gateway limit	SOTA budget	Who
agents-payments	200k / 5m	80k / 5m	agent of payments (one SOTA investigation is about 30k-40k tokens: about 2 in 5 minutes, then local)
agents-rh-reserve	200k / 5m	80k / 5m	agent of rh-reserve
agents-reverse-proxy-app	200k / 5m	80k / 5m	agent of reverse-proxy-app
agents-critical	600k / 5m	150k / 5m	agents of business-critical applications not covered by a dedicated tier above (set their apiKeyTier): about 13 investigations in 5 minutes
validation	100k / 5m	1	the validation repo (check B1: its second SOTA request stays local)

• The tier comes from the API key (label maas-group, set by the gateway in x-team). With one agent per application namespace, each agent has its own key and tier.
• The counters are in Redis (litellm-redis in maas-routing, one replica, Red Hat rhel9/redis-7, no persistence, password from an ESO generator, reachable only from the LiteLLM pods). A Redis restart starts the counters from zero. When Redis is down the router does not apply the budget (fail-open: it controls cost, the privacy gates still run) and counts the errors.
• Only the answers of the SOTA model count: a fallback to the local model and the local-only mode do not.
• The log line of a tier with a budget has sota_budget_used and sota_budget_limit; the reason when the budget is used: efficiency: SOTA budget of tier agents used (... tokens in 5m) -> LOCAL.
• Dashboard stack-operations: row "SOTA budget per tier" (one gauge per tier, SOTA tokens per minute, Redis errors).

AI-driven triage demo

Originally imported from matteo-grimaldi/ocp-trobleshooter-demo, the demo now spans four namespaces: a shared-services namespace (agentic-triage) and one namespace per monitored application (payments, rh-reserve, reverse-proxy-app), each with its own buggy app and its own agent:

                      agentic-triage (shared services)
                      ├── ticketing-system
                      ├── prometheus-mcp-server, ticketing-mcp-server, ocp-mcp-server
                      └── ogx-alert-translator

payments / rh-reserve / reverse-proxy-app (one of each, per namespace)
  <namespace>-buggy-app (payments-buggy-app / rh-reserve-buggy-app / reverse-proxy-buggy-app)
  triage-agent (Gradio UI, TriageAgent CR) ──> OGX sidecar (server-side agentic loop, same pod)
                                                  ├──> router.<appsDomain>/v1, model "auto"   (own tier, e.g. "agents-payments")
                                                  ├──> prometheus-mcp-server (MCP, agentic-triage) ──> Thanos Querier
                                                  ├──> ticketing-mcp-server (MCP, agentic-triage) ──> ticketing-system
                                                  └──> ocp-mcp-server (MCP, agentic-triage) ──> Kubernetes API (read-only)

• buggy app (components/quarkus-buggy-app chart): a target application that injects random failures (30% 500 on /api/products, 40% 503 on /api/inventory, 20% 3s latency on /api/orders) and generates its own traffic, so error metrics move even without manual requests. One instance per monitored-application namespace, each with its own appName so console navigation and dashboards can tell them apart (bootstrap components payments-buggy-app, rh-reserve-buggy-app, reverse-proxy-buggy-app, same chart).
• ticketing-system: a ServiceNow Table API simulator (dashboard at its Route). Runs once, in agentic-triage, shared by every agent.
• prometheus-mcp-server / ticketing-mcp-server: MCP tool servers (MCPServer CR, needs the ansible-owned MCP lifecycle operator). Run once, in agentic-triage; every agent calls them cross-namespace.
• ocp-mcp-server: read-only Kubernetes/OpenShift API tools (pods, logs, events, generic resources) for the agents, run in --read-only --toolsets core mode. Upstream containers/kubernetes-mcp-server project (quay.io/containers/kubernetes_mcp_server), not a sovereign-selfheal repo: pinned by digest like the OGX sidecar image, no source contract. Its ServiceAccount is bound by the ansible repo to the built-in view ClusterRole (cluster-wide read access). Runs once, in agentic-triage.
• triage-agent (one TriageAgent CR per monitored-application namespace, components/triage-agent-operator-cr): keeps the upstream demo's OGX sidecar (server-side agentic loop, native MCP tool calling) but drops its external MaaS endpoint — OGX is configured to call this platform's own LiteLLM router (model auto) instead, so the privacy gate and local/SOTA routing apply to its traffic too. Each instance has its own router credential and tier (agents-payments, agents-rh-reserve, agents-reverse-proxy-app), set with a per-component values override in bootstrap/values.yaml (see AGENTS.md §2 "Per-component value overrides"); all three use the same CR name (triage-agent), so ogx-alert-translator can reach any of them by the same convention. Source and images: triage-agent, mock-ticketing-system, quarkus-buggy-app (image name; see components/quarkus-buggy-app's appName for the per-instance object names), prometheus-mcp-server repos (see AGENTS.md §4); the OGX sidecar image itself is a third-party image (ogxai/distribution-starter), pinned by digest in gitops only, not built by any of those repos.
• ogx-alert-translator: a single Alertmanager webhook bridge (/webhook), shared by every agent, deployed once in agentic-triage. It turns an alert into a prompt and calls the /trigger endpoint of the agent that runs in the alert's own namespace — it reads the alert's namespace label and builds http://triage-agent.<namespace>.svc:7860/trigger, instead of a fixed target. It scales from zero with the KEDA HTTP add-on, so senders POST to the interceptor (keda-add-ons-http-interceptor-proxy.openshift-keda.svc:8080, see AGENTS.md §4). Ansible enables alertmanagerMain.enableUserAlertmanagerConfig; this repo adds one AlertmanagerConfig per monitored-application namespace, all pointing to the same translator.
• Timing of the alert path. The rule fires after 5 minutes with more than 10% HTTP 5xx (the share counts only real HTTP statuses, not the series status="RESET"). The apps of the demo keep failing, so the alert stays firing. The first investigation of each agent starts about 5.5 minutes after its app starts (for: 5m plus the group wait and the cold start of the translator). Then Alertmanager sends the alert again every hour (alertmanager.repeatInterval), and each agent investigates again. To show an investigation at a given moment, start it from the agent's chat UI.
• Reset for another recording. Run scripts/reset-demo-state.sh on the cluster (with oc logged in). It clears ticketing incidents, Tempo traces, SOTA budgets (Redis restart), router metric counters (LiteLLM restart), and the routing-live-view page. Firing alerts are unchanged; after a reset, start an investigation from an agent chat UI or wait for the hourly Alertmanager repeat.

Try the demo prompts from any agent's chat UI Route, or open an incident manually against the ticketing system's /api/incidents endpoint and ask the agent of the matching namespace to investigate it.

Router code and images

Two custom images and the router code come from other repos of the project:

Item	Source	Pinned in
Hook code components/litellm-router/files/*.py	router, at the tag routerVersion	components/litellm-router/values.yaml (routerVersion)
LiteLLM image (LiteLLM + fastText)	quay.io/sovereign-selfheal/router, same tag	components/litellm-router/values.yaml (image, by digest)
Presidio image (English and Italian NER)	presidio, quay.io/sovereign-selfheal/presidio	components/presidio/values.yaml (image, by digest)

Never edit the *.py files here. To move to a new router release:

scripts/sync-router-code.sh sync vX.Y.Z   # copies the hook code at the tag, sets routerVersion
# then pin the digest of quay.io/sovereign-selfheal/router:vX.Y.Z in components/litellm-router/values.yaml
scripts/sync-router-code.sh check         # also run by CI

The policies (chain.yaml, privacy-plus.yaml) stay in this repo.

Secrets

No Secret is stored in git. The External Secrets Operator creates them:

Secret (namespace maas-routing)	Keys	Source
apikey-<tier>-1	api_key (sk- + 40 random characters)	ESO Password generator, in the cluster, generated once
sota-provider-secret	COMPANY_API_KEY	Store: key sota, property api_key
classifier-provider-secret (only classifier.mode: external)	CLASSIFIER_BASE_URL, CLASSIFIER_MODEL, CLASSIFIER_API_KEY, CLASSIFIER_ENABLED, CLASSIFIER_GRAY_LOW	Store: key classifier, properties base_url, model, api_key

The store is the ClusterSecretStore sovereign-selfheal, created by Ansible (roles/secrets_bootstrap). Today it uses the ESO provider kubernetes: Ansible takes the values from ansible-vault (or from AgnosticV) and writes them into Secrets of the namespace sovereign-selfheal-secrets; ESO copies them here. Another backend (for example HashiCorp Vault) needs only a different store, not changes in this repo.

Without a SOTA key the store does not exist and secretStore.enabled is false: the API keys still work, and LiteLLM starts without the SOTA key (the Secret reference is optional). The expected result is that SOTA calls fail and LiteLLM falls back to the local model.

To read an API key:

oc get secret apikey-research-1 -n maas-routing -o jsonpath='{.data.api_key}' | base64 -d

Check your changes

uvx yamllint .
for chart in bootstrap components/*/; do helm lint "$chart"; done
for p in gpu cpu; do uv run --no-project --with pyyaml scripts/render.sh "$p" "rendered/$p"; done
kubeconform -summary -ignore-missing-schemas rendered/gpu/*.yaml

The Python tools (yamllint, and PyYAML for scripts/render.sh) run with uv, never pip; the CI pins their versions.

scripts/render.sh renders the charts the way Argo CD does, then checks that no component renders a Namespace or a Secret, or uses a namespace outside the contract. Use Helm 3 (HELM=/path/to/helm3): Argo CD uses Helm 3 too.

License

Apache License 2.0, see LICENSE.