From 31263f813428d71442fe916cb2d15ef76de5c7b2 Mon Sep 17 00:00:00 2001 From: Daniel Vaskivaara Date: Wed, 2 Sep 2026 22:35:03 +0300 Subject: [PATCH 1/3] docs(aim-catalog): describe shipped catalog behaviour, not pending work The AIM catalog docs described behaviour that only exists in unmerged work, so operators reading them would configure the wrong thing. - Drop AIM_HARDWARE_FAMILY auto-detection claims from the lifecycle doc and chart README; auto-detection is not in cluster-bloom main, where the setting has no default and is injected only when set. Document that an install leaving it unset lands on the unfiltered catalog. - Replace the "published into the chart" source-of-truth framing with the actual manual copy at release time, and drop the dead public aim-build link. Point operators at cluster-managed additions when they need an AIM version before the next release. - Note that aim-cluster-model-source-additional is not shipped and needs one-time operator setup. - Fix stale "legacy" naming in root/values.yaml after the legacy.yaml -> unfiltered.yaml rename. Co-authored-by: Cursor --- docs/aim_model_management.md | 126 ++++++++++++++------- root/values.yaml | 5 +- sources/aim-cluster-model-source/README.md | 7 +- 3 files changed, 94 insertions(+), 44 deletions(-) diff --git a/docs/aim_model_management.md b/docs/aim_model_management.md index da411b43..e58e4497 100644 --- a/docs/aim_model_management.md +++ b/docs/aim_model_management.md @@ -1,9 +1,8 @@ # AIM model catalog lifecycle Reference for how AIM model catalog sources are packaged, extended, and retired on -Enterprise AI reference stack clusters. This guide is for **cluster operators and platform -administrators** who need to understand -catalog behaviour and manage models over the life of a cluster. +Enterprise AI reference stack clusters. This guide is for **cluster operators** who +need to understand catalog behaviour and manage models over the life of a cluster. It is **not tied to the Cluster Forge release cadence**. Use it whenever you need to add, replace, or remove catalog entries: private builds, early access @@ -21,32 +20,46 @@ The catalog has two complementary layers: | **Packaged baseline** | `aim-cluster-model-source` Helm chart (Argo CD) | Default AMD catalog for the cluster hardware family; refreshed when you upgrade Cluster Forge | | **Cluster-managed additions** | Gitea `cluster-values` + `aim-cluster-model-source-additional` | Any extra model or base images you choose to expose — RCs, private builds, site-specific tags, or images not yet in the packaged chart | -Cluster-managed additions do not replace the AIM team release process or Platform -packaging workflow. They are an **operator-controlled extension** of the catalog, -available at any time. +Cluster-managed additions do not replace the packaged baseline that ships with the +cluster; the baseline keeps arriving and updating through Cluster Forge releases. +Additions are an **operator-controlled extension** on top of it, available at any +time. ## Packaged baseline catalog Cluster Forge installs the in-tree Helm chart `sources/aim-cluster-model-source`. `AIMClusterModelSource` resources are selected by `hardwareFamilies`, which -cluster-bloom sets from `AIM_HARDWARE_FAMILY` (auto-detected when omitted). +cluster-bloom sets from `AIM_HARDWARE_FAMILY`. | `hardwareFamilies` | Template | Result | |--------------------|----------|--------| | Non-empty list (`instinct`, `epyc`, `cpu`, `radeon`) | `templates/profiles.yaml` | Only listed families. The Instinct profile includes generic `amd-aim-release-*` sources (0.8.5–0.11.0) plus Instinct 0.11.1+. `cpu` is a placeholder and renders no sources. | | Empty list (`[]`, chart default) | `templates/unfiltered.yaml` | Instinct **0.11.1, 0.12.0, 0.13.0** plus mixed base images (`aim-base`, `aim-epyc-base`, `aim-radeon-base`). | -A typical new cluster-bloom install injects a non-empty list, so it uses -**profiles**. Clearing `hardwareFamilies` to `[]` in Gitea switches to -**unfiltered**; it does not fail chart rendering. See the +`AIM_HARDWARE_FAMILY` has no default. cluster-bloom injects `hardwareFamilies` +only when the install sets it, so an install that leaves it unset takes the +**unfiltered** path. Set `AIM_HARDWARE_FAMILY` in `bloom.yaml` to get a +family-filtered catalog: + +```yaml +AIM_HARDWARE_FAMILY: "instinct" +``` + +Clearing `hardwareFamilies` to `[]` in Gitea on an existing cluster switches it +back to **unfiltered**; it does not fail chart rendering. See the [aim-cluster-model-source README](../sources/aim-cluster-model-source/README.md). ### Model release sources vs base catalog sources -| Kind | Purpose | Examples | -|------|---------|----------| -| **Model release source** | Version-pinned model-specific images for a hardware family | `amd-aim-instinct-0.12.0`, `amd-aim-epyc-0.13.0` | -| **Base catalog source** | Generic base images for AI Workbench custom model onboarding (runtime AIM ID) | `amdenterpriseai/aim-base`, `amdenterpriseai/aim-epyc-base`, `amdenterpriseai/aim-radeon-base` | +| Kind | Purpose | Example source name | Example images it lists | +|------|---------|---------------------|-------------------------| +| **Model release source** | Version-pinned model-specific images for a hardware family | `amd-aim-instinct-0.12.0` | `amdenterpriseai/aim-google-gemma-3-1b-it:0.12.0`, `amdenterpriseai/aim-zai-org-glm-4-7:0.12.0` | +| **Model release source** | Same, EPYC family | `amd-aim-epyc-0.13.0` | `amdenterpriseai/aim-epyc-qwen-qwen3-8b:0.13.0` | +| **Base catalog source** | Generic base images for AI Workbench custom model onboarding (runtime AIM ID) | `aim-base-models` | `amdenterpriseai/aim-base:0.13.1`, `amdenterpriseai/aim-epyc-base:0.13`, `amdenterpriseai/aim-radeon-base:0.12` | + +Source names such as `amd-aim-instinct-0.12.0` are `AIMClusterModelSource` +resource names, not image references. The images each source lists are fully +qualified `repository/name:tag` values, as in the last column. Model release sources list **model-specific** images. Base catalog sources list **base** images only. @@ -55,20 +68,35 @@ Bases are split **by hardware family** in the packaged chart so AI Workbench does not show large numbers of not-deployable entries on clusters without matching hardware. -When ROCm ships multi-hardware base images in a future release, family gating -for bases may be relaxed. Until then, each family profile installs only its own -base images (Instinct → `aim-base`, EPYC → `aim-epyc-base`, Radeon → -`aim-radeon-base`, and so on). +Each family profile installs only its own base images: Instinct → `aim-base`, +EPYC → `aim-epyc-base`, Radeon → `aim-radeon-base`. The `unfiltered` template is +the one exception — it installs all three, because it has no family to filter +on. ### Source of truth -The AIM team maintains canonical model source and base-image lists in the -[public AIM build repository](https://github.com/amd-enterprise-ai/aim-build). -Those lists are published into the `aim-cluster-model-source` Helm chart for -clusters to consume. +The AIM team maintains the canonical model source and base-image lists in an +AMD-internal build repository. Operators have no direct access to it, and there +is no automated feed from it into a cluster. + +Those lists are copied into `sources/aim-cluster-model-source/templates/` by +hand as part of preparing a Cluster Forge release. The chart templates in a +given release are therefore a **point-in-time snapshot**, not a live mirror: an +AIM version published after that release does not appear in the packaged +baseline until a later Cluster Forge release picks it up. + +For a cluster, the operator-visible source of truth is +`sources/aim-cluster-model-source/` in the Cluster Forge release you installed. +To read the catalog it will install before you deploy, render the chart: -The Platform release process incorporates approved manifests into -`sources/aim-cluster-model-source/` at each Cluster Forge release. +```bash +helm template aim-cluster-model-source sources/aim-cluster-model-source \ + --set-json 'hardwareFamilies=["instinct"]' | grep -E '^ name:' +``` + +If you need an AIM version sooner than the next Cluster Forge release, use +[cluster-managed additions](#cluster-managed-catalog-additions) — that is the +only path that does not wait on packaging. Environment-specific CI snapshots are not packaged in Cluster Forge. @@ -76,23 +104,41 @@ Environment-specific CI snapshots are not packaged in Cluster Forge. | Scenario | Policy | |----------|--------| -| **New installation (cluster-bloom)** | Auto-detect or explicit `AIM_HARDWARE_FAMILY` injects a non-empty list → **profiles** branch. Instinct profile still includes generic `amd-aim-release-*` 0.8.5–0.11.0. | +| **New installation, `AIM_HARDWARE_FAMILY` set** | Injects a non-empty list → **profiles** branch. The Instinct profile installs generic `amd-aim-release-*` 0.8.5–0.11.0 alongside `amd-aim-instinct-*` 0.11.1, 0.12.0, 0.13.0, so such an install starts with all of them. | +| **New installation, `AIM_HARDWARE_FAMILY` unset** | Nothing is injected → chart default `[]` → **unfiltered** catalog. | | **Empty `hardwareFamilies` in Gitea** | **unfiltered** catalog: Instinct 0.11.1+ only (no generic 0.8.x–0.11.0 sources). | | **Platform upgrade** | New AIM versions are **added**. Older versions are **not** removed automatically. | -| **Catalog cleanup** | Installation owner removes deprecated sources or models when no longer needed. | +| **Catalog cleanup** | Manual. The cluster operator removes deprecated sources; nothing expires on its own. | + +There is no automated deprecation schedule. An older AIM version stays in the +catalog until a cluster operator removes its source. -There is no automated deprecation schedule. Teams depending on an older AIM -version retain its source until they remove it. +To remove one, delete the source manifest in Gitea and sync with prune, as +described in +[Replace or remove a source](adding_aim_catalog_models.md#replace-or-remove-a-source). +Deleting an `AIMClusterModelSource` garbage-collects the `AIMClusterModel` +resources AIM Engine derived from it — you do not delete those separately. + +This applies to cluster-managed additions. Packaged baseline sources are owned +by the `aim-cluster-model-source` chart and are restored by the next Argo CD +sync if deleted in the cluster. The chart's only selector is `hardwareFamilies`, +which switches whole family profiles; it cannot drop an individual packaged +version. ## Cluster-managed catalog additions -Operators add or remove sources through Gitea and Argo CD; see +Cluster operators add or remove sources through Gitea and Argo CD; see [Adding AIM catalog models](adding_aim_catalog_models.md). -Typical lifecycle: +`aim-cluster-model-source-additional` is not shipped in Cluster Forge. The +operator defines it once in Gitea `cluster-values` (`enabledApps` plus an `apps` +entry) before this path is available — see +[One-time setup](adding_aim_catalog_models.md#one-time-setup). + +Typical lifecycle, once it is enabled: 1. Gitea stores `AIMClusterModelSource` manifests in `cluster-values`. -2. `cluster-forge` creates `aim-cluster-model-source-additional`. +2. The `cluster-forge` parent application creates `aim-cluster-model-source-additional`. 3. AIM Engine discovers images and creates `AIMClusterModel` resources. 4. AI Workbench refreshes its catalog periodically. @@ -104,8 +150,8 @@ need — regardless of whether a Cluster Forge upgrade is planned. The packaged baseline is family-filtered when `hardwareFamilies` is non-empty. Cluster-managed additions can list any image, but entries for the wrong accelerator appear as **not deployable** in AI Workbench. Prefer family-matched -images and the `{family}-*.yaml` filename convention described in the how-to -guide. +images and the `{family}-*.yaml` filename convention described in +[Add a model](adding_aim_catalog_models.md#add-a-model). ## Lifecycle constraints @@ -122,21 +168,23 @@ exists. Disable the application only after sources are pruned. When upgrading Cluster Forge, review the incoming packaged catalog and remove cluster-managed manifests that duplicate newly packaged models or bases before -syncing — see the how-to guide. +syncing — see +[Before a platform upgrade](adding_aim_catalog_models.md#before-a-platform-upgrade). ## Responsibilities | Responsibility | Owner | |----------------|-------| -| AIM release manifests and base-image lists | AIM team | -| Packaging into Cluster Forge | Platform release process | -| Cluster-managed catalog additions and removals | Cluster / installation operator | -| Removing deprecated catalog entries | Installation owner | +| Canonical AIM release manifests and base-image lists | AIM team (internal repository) | +| Copying those lists into the packaged chart at release time | Cluster Forge release process (manual step) | +| Choosing `AIM_HARDWARE_FAMILY` at install | Whoever runs cluster-bloom | +| Cluster-managed catalog additions and removals | Cluster operator | +| Removing deprecated catalog entries | Cluster operator | ## Related documentation - [Adding AIM catalog models](adding_aim_catalog_models.md) — procedural guide - for operators + for cluster operators - [aim-cluster-model-source README](../sources/aim-cluster-model-source/README.md) — Helm chart reference - [Values inheritance pattern](values_inheritance_pattern.md) — how diff --git a/root/values.yaml b/root/values.yaml index 3d4f6783..b67c6eb3 100644 --- a/root/values.yaml +++ b/root/values.yaml @@ -53,8 +53,9 @@ apps: syncWave: -20 valuesObject: # Hardware families to install (cpu,epyc,instinct,radeon). - # Empty = legacy (install all generic model sources). cluster-bloom - # injects the selected families as a YAML list at deploy time. + # Empty = unfiltered catalog (templates/unfiltered.yaml). cluster-bloom + # injects the selected families as a YAML list at deploy time, and only + # when AIM_HARDWARE_FAMILY is set. hardwareFamilies: [] airm: repoURL: "{{ .Values.ociRegistry.dockerHub }}" diff --git a/sources/aim-cluster-model-source/README.md b/sources/aim-cluster-model-source/README.md index b84249ae..17d6b8d0 100644 --- a/sources/aim-cluster-model-source/README.md +++ b/sources/aim-cluster-model-source/README.md @@ -14,9 +14,10 @@ exclusive branches, selected by `hardwareFamilies`: | Empty (`[]`, chart default) | `templates/unfiltered.yaml` | Instinct model sources **0.11.1, 0.12.0, 0.13.0** plus a mixed base catalog (`aim-base`, `aim-epyc-base`, `aim-radeon-base`) | | Non-empty list | `templates/profiles.yaml` | Only the listed families (see table below) | -cluster-bloom injects a YAML list at install (`AIM_HARDWARE_FAMILY`, auto-detected -when omitted), so a typical new install takes the **profiles** path. Clearing the -list to `[]` in Gitea selects `unfiltered.yaml`. +cluster-bloom injects a YAML list at install from `AIM_HARDWARE_FAMILY`. That +setting has no default and is injected only when set, so an install that leaves +it unset falls through to the chart default `[]` and takes the **unfiltered** +path. Clearing the list to `[]` in Gitea also selects `unfiltered.yaml`. ## `hardwareFamilies` From d96e794073cdd7f833423c10e479b965b00d5e04 Mon Sep 17 00:00:00 2001 From: Daniel Vaskivaara Date: Wed, 2 Sep 2026 23:41:49 +0300 Subject: [PATCH 2/3] docs(aim-catalog): document filter removal vs source deletion AIM Engine discovery is append-only, so dropping an image from a source filter leaves discovered models in place, while deleting the source CR garbage-collects them and can break running deployments. Empty filters are invalid (MinItems=1). Packaged source names are a stable API. Co-authored-by: Cursor --- docs/adding_aim_catalog_models.md | 11 +++++++- docs/aim_model_management.md | 45 ++++++++++++++++++++++++------- 2 files changed, 45 insertions(+), 11 deletions(-) diff --git a/docs/adding_aim_catalog_models.md b/docs/adding_aim_catalog_models.md index 44f8e725..12d7a68f 100644 --- a/docs/adding_aim_catalog_models.md +++ b/docs/adding_aim_catalog_models.md @@ -137,7 +137,16 @@ family-specific (`aim-base` on Instinct, `aim-epyc-base` on EPYC, and so on). ## Replace or remove a source **Do not** remove entries by narrowing filters on an existing source — discovery -is append-only. +is append-only. The image stays in the catalog as an `AIMClusterModel` until the +source itself is deleted. Emptying `spec.filters` is not a workaround: the CRD +requires at least one filter or image. + +Deleting the source **is** how models leave the catalog, and it is a breaking +change for anything still using them: Argo CD prune removes the +`AIMClusterModelSource`, then Kubernetes garbage-collects its owned +`AIMClusterModel` resources. Confirm no running deployments depend on those +images before you prune. See +[Filter removal vs source removal](aim_model_management.md#filter-removal-vs-source-removal). **Replace:** diff --git a/docs/aim_model_management.md b/docs/aim_model_management.md index e58e4497..ff9e28c4 100644 --- a/docs/aim_model_management.md +++ b/docs/aim_model_management.md @@ -111,19 +111,16 @@ Environment-specific CI snapshots are not packaged in Cluster Forge. | **Catalog cleanup** | Manual. The cluster operator removes deprecated sources; nothing expires on its own. | There is no automated deprecation schedule. An older AIM version stays in the -catalog until a cluster operator removes its source. - -To remove one, delete the source manifest in Gitea and sync with prune, as -described in -[Replace or remove a source](adding_aim_catalog_models.md#replace-or-remove-a-source). -Deleting an `AIMClusterModelSource` garbage-collects the `AIMClusterModel` -resources AIM Engine derived from it — you do not delete those separately. +catalog until its `AIMClusterModelSource` is deleted. Narrowing a filter is not +enough — see [Filter removal vs source removal](#filter-removal-vs-source-removal). This applies to cluster-managed additions. Packaged baseline sources are owned by the `aim-cluster-model-source` chart and are restored by the next Argo CD sync if deleted in the cluster. The chart's only selector is `hardwareFamilies`, which switches whole family profiles; it cannot drop an individual packaged -version. +version. Packaged source names (`amd-aim-instinct-0.12.0`, +`amd-aim-epyc-0.13.0`, and so on) are therefore a **stable API**: a Cluster +Forge release adds tracks; it does not delete source names from the chart. ## Cluster-managed catalog additions @@ -153,12 +150,40 @@ accelerator appear as **not deployable** in AI Workbench. Prefer family-matched images and the `{family}-*.yaml` filename convention described in [Add a model](adding_aim_catalog_models.md#add-a-model). +## Filter removal vs source removal + +AIM Engine discovery is append-only. Two edits that look similar in YAML have +opposite runtime effects: + +| Action | What happens | Running deployments | +|--------|--------------|---------------------| +| Drop an image from `spec.filters` (or `spec.images`) on an existing source | Already-discovered `AIMClusterModel` resources **stay**. The catalog can keep showing the old image. | Unaffected | +| Delete the `AIMClusterModelSource` (Gitea manifest + Argo CD prune, or drop the source **name** from the packaged chart) | The CR is deleted. Kubernetes garbage-collects every `AIMClusterModel` it owned. | **Can break** workloads that still use those models | + +A “hollow shell” — keep the source CR but empty the filter list — **does not +work**. The `AIMClusterModelSource` CRD requires at least one entry in +`spec.filters` or `spec.images` (`MinItems=1`), so empty filters fail +validation. + +**To retire a model you actually want gone**, delete the source (and sync with +prune), as in +[Replace or remove a source](adding_aim_catalog_models.md#replace-or-remove-a-source). +You do not delete `AIMClusterModel` resources separately. + +**To keep running deployments** while stopping new discoveries of an image, +leave the source name in place and stop listing that image. Expect catalog +clutter until a later source deletion. + +**Do not delete packaged source names** from `sources/aim-cluster-model-source` +between Cluster Forge releases. Retire a track by adding a new source; leaving +the old name is what keeps existing clusters from having their models +garbage-collected on the next chart sync. + ## Lifecycle constraints - **Discovery is additive** — adding a filter can create another `AIMClusterModel`. -- **Removing a filter does not remove discovered models** — delete and replace - the source, or remove its manifest and sync with prune. +- **Removing a filter does not remove discovered models** — see above. - **Removing the additional Argo CD Application does not delete its resources** — child apps lack a cascading-resources finalizer. - **Deleting an `AIMClusterModelSource` garbage-collects its owned models.** From 958d9aff5e70c089e27fdab74735abd831c2a3e1 Mon Sep 17 00:00:00 2001 From: Daniel Vaskivaara Date: Wed, 2 Sep 2026 23:54:32 +0300 Subject: [PATCH 3/3] docs(aim-catalog): drop internal-team names from operator responsibilities Operators see the packaged baseline as the Cluster Forge release they installed. Who authored the internal lists is not needed to operate a cluster. Co-authored-by: Cursor --- docs/aim_model_management.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/aim_model_management.md b/docs/aim_model_management.md index ff9e28c4..42557223 100644 --- a/docs/aim_model_management.md +++ b/docs/aim_model_management.md @@ -75,15 +75,14 @@ on. ### Source of truth -The AIM team maintains the canonical model source and base-image lists in an -AMD-internal build repository. Operators have no direct access to it, and there -is no automated feed from it into a cluster. - -Those lists are copied into `sources/aim-cluster-model-source/templates/` by -hand as part of preparing a Cluster Forge release. The chart templates in a -given release are therefore a **point-in-time snapshot**, not a live mirror: an -AIM version published after that release does not appear in the packaged -baseline until a later Cluster Forge release picks it up. +Canonical model source and base-image lists live in an AMD-internal repository +that operators cannot access. There is no automated feed from it into a +cluster. Those lists are copied into +`sources/aim-cluster-model-source/templates/` by hand when a Cluster Forge +release is prepared. The chart templates in a given release are therefore a +**point-in-time snapshot**, not a live mirror: an AIM version published after +that release does not appear in the packaged baseline until a later Cluster +Forge release picks it up. For a cluster, the operator-visible source of truth is `sources/aim-cluster-model-source/` in the Cluster Forge release you installed. @@ -200,8 +199,7 @@ syncing — see | Responsibility | Owner | |----------------|-------| -| Canonical AIM release manifests and base-image lists | AIM team (internal repository) | -| Copying those lists into the packaged chart at release time | Cluster Forge release process (manual step) | +| Packaged baseline catalog (what a new cluster gets) | The Cluster Forge release you installed | | Choosing `AIM_HARDWARE_FAMILY` at install | Whoever runs cluster-bloom | | Cluster-managed catalog additions and removals | Cluster operator | | Removing deprecated catalog entries | Cluster operator |