Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/adding_aim_catalog_models.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,15 @@ You need:

Use images that match the cluster's hardware family. Listing images for other
accelerators creates catalog entries that AI Workbench marks as not deployable.
Active families:
```
Active families:

```bash
kubectl get application -n argocd aim-cluster-model-source -o go-template='{{ index (fromYaml .spec.source.helm.values) "hardwareFamilies" }}{{ println }}'
```
```

...also in Gitea **cluster-values** → `values.yaml` → `apps.aim-cluster-model-source.valuesObject.hardwareFamilies`.
An empty list there selects `templates/unfiltered.yaml` (Instinct 0.11.1+ plus mixed
bases).

For private registries, set `spec.imagePullSecrets` on the source to a secret in
`aim-system`. Do not commit credentials to Gitea.
Expand Down Expand Up @@ -174,7 +178,7 @@ Removing the application first leaves orphaned sources in the cluster.
## Troubleshooting

| Symptom | Check |
|---------|--------|
| --------- | -------- |
| Additional app missing | `enabledApps` entry, `apps` definition, parent `cluster-forge` synced |
| Model remains after app removed | Source was not pruned first — `kubectl delete aimclustermodelsource <name>` |
| Filter removed but model remains | Append-only discovery — delete and replace the source |
Expand Down
33 changes: 20 additions & 13 deletions docs/aim_model_management.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# AIM model catalog lifecycle

Reference for how AIM model catalog sources are packaged, extended, and retired on
Cluster Forge clusters. This guide is for **cluster operators and platform
Enterprise AI reference stack clusters. This guide is for **cluster operators and platform
administrators** who need to understand
catalog behaviour and manage models over the life of a cluster.

Expand All @@ -27,14 +27,19 @@ available at any time.

## Packaged baseline catalog

Cluster Forge installs `aim-cluster-model-source`. Its
`AIMClusterModelSource` resources are selected by `AIM_HARDWARE_FAMILY` from
cluster-bloom (auto-detected when omitted).
Cluster Forge installs the in-tree Helm chart `sources/aim-cluster-model-source`.
`AIMClusterModelSource` resources are selected by `hardwareFamilies`, which
cluster-bloom sets from `AIM_HARDWARE_FAMILY` (auto-detected when omitted).

The chart renders **per-hardware-family profiles** for `instinct`, `epyc`,
`cpu`, and `radeon`. Only listed families are installed. The former legacy
behaviour (full generic `amd-aim-release-*` catalog when `hardwareFamilies` is
empty) is removed; an empty family list is not supported on new installations.
| `hardwareFamilies` | Template | Result |
|--------------------|----------|--------|
| Non-empty list (`instinct`, `epyc`, `cpu`, `radeon`) | `templates/profiles.yaml` | Only listed families. The Instinct profile includes generic `amd-aim-release-*` sources (0.8.5–0.11.0) plus Instinct 0.11.1+. `cpu` is a placeholder and renders no sources. |
| Empty list (`[]`, chart default) | `templates/unfiltered.yaml` | Instinct **0.11.1, 0.12.0, 0.13.0** plus mixed base images (`aim-base`, `aim-epyc-base`, `aim-radeon-base`). |

A typical new cluster-bloom install injects a non-empty list, so it uses
**profiles**. Clearing `hardwareFamilies` to `[]` in Gitea switches to
**unfiltered**; it does not fail chart rendering. See the
[aim-cluster-model-source README](../sources/aim-cluster-model-source/README.md).

### Model release sources vs base catalog sources

Expand Down Expand Up @@ -71,7 +76,8 @@ Environment-specific CI snapshots are not packaged in Cluster Forge.

| Scenario | Policy |
|----------|--------|
| **New installation** | Packaged catalog starts at AIM **0.11.1** and later. Pre-0.11.1 generic release sources are not installed. |
| **New installation (cluster-bloom)** | Auto-detect or explicit `AIM_HARDWARE_FAMILY` injects a non-empty list → **profiles** branch. Instinct profile still includes generic `amd-aim-release-*` 0.8.5–0.11.0. |
| **Empty `hardwareFamilies` in Gitea** | **unfiltered** catalog: Instinct 0.11.1+ only (no generic 0.8.x–0.11.0 sources). |
| **Platform upgrade** | New AIM versions are **added**. Older versions are **not** removed automatically. |
| **Catalog cleanup** | Installation owner removes deprecated sources or models when no longer needed. |

Expand All @@ -95,10 +101,11 @@ need — regardless of whether a Cluster Forge upgrade is planned.

### Hardware family and catalog UX

The packaged baseline is family-filtered. Cluster-managed additions can list any
image, but entries for the wrong accelerator appear as **not deployable** in AI
Workbench. Prefer family-matched images and the `{family}-*.yaml` filename
convention described in the how-to guide.
The packaged baseline is family-filtered when `hardwareFamilies` is non-empty.
Cluster-managed additions can list any image, but entries for the wrong
accelerator appear as **not deployable** in AI Workbench. Prefer family-matched
images and the `{family}-*.yaml` filename convention described in the how-to
guide.

## Lifecycle constraints

Expand Down
4 changes: 3 additions & 1 deletion docs/values_inheritance_pattern.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,9 @@ When ArgoCD renders applications with multi-source:
`apps.aim-cluster-model-source.valuesObject.hardwareFamilies` (see
`sources/aim-cluster-model-source`). The value travels as a structured list,
not a string, so no comma parsing is involved. The base `root/values.yaml`
default is an empty list, which selects the legacy (install-all) branch.
default is an empty list, which selects `templates/unfiltered.yaml` (Instinct
0.11.1+ plus mixed bases), not the per-family `templates/profiles.yaml`
branch.

GPU stack family (ROCm + GPU Operator) is injected the same way, driven by
cluster-bloom's `GPU_STACK_FAMILY`. Two child-app keys are set:
Expand Down
37 changes: 18 additions & 19 deletions sources/aim-cluster-model-source/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,46 +6,45 @@ SPDX-License-Identifier: MIT

# aim-cluster-model-source

Helm chart that installs `AIMClusterModelSource` resources. It renders one of
two mutually exclusive branches, selected by `hardwareFamilies`:
Helm chart that installs `AIMClusterModelSource` resources. Two mutually
exclusive branches, selected by `hardwareFamilies`:

- **Legacy (default):** when `hardwareFamilies` is empty, the chart installs the
full set of generic `amd-aim-release-*` model sources (versions 0.8.5, 0.9.0,
0.10.0, 0.11.0), unchanged from the pre-chart directory app.
- **Per-hardware-family profiles:** when `hardwareFamilies` is non-empty, the
chart installs only the `AIMClusterModelSource` resources for the listed
families. The legacy generic sources are not installed.
| `hardwareFamilies` | Template | What is installed |
|---|---|---|
| Empty (`[]`, chart default) | `templates/unfiltered.yaml` | Instinct model sources **0.11.1, 0.12.0, 0.13.0** plus a mixed base catalog (`aim-base`, `aim-epyc-base`, `aim-radeon-base`) |
| Non-empty list | `templates/profiles.yaml` | Only the listed families (see table below) |

cluster-bloom injects a YAML list at install (`AIM_HARDWARE_FAMILY`, auto-detected
when omitted), so a typical new install takes the **profiles** path. Clearing the
list to `[]` in Gitea selects `unfiltered.yaml`.

## `hardwareFamilies`

A YAML list (the primary form) or a comma-separated string. Allowed values:
`cpu`, `epyc`, `instinct`, `radeon`. Empty (the default) selects the legacy
branch.
`cpu`, `epyc`, `instinct`, `radeon`.

```yaml
hardwareFamilies:
- epyc
- instinct
```

| Family | Source name | Registry | Notes |
| Family | Model sources | Base images | Notes |
|---|---|---|---|
| `instinct` | `amd-aim-instinct-0.12.0` | docker.io | works today |
| `epyc` | `amd-aim-epyc-0.11.0` | docker.io | works today |
| `cpu` | `amd-aim-cpu-0.12.0-rc1` | docker.io | `silogenai/*` RC images; optional `dockerhub-regcred` if pulls are private |
| `radeon` | `amd-aim-radeon-0.12.0-rc1` | docker.io | `silogenai/aim-radeon-*` RC tags; optional `dockerhub-regcred` if pulls are private |
| `instinct` | `amd-aim-release-0.8.5` … `0.11.0`, `amd-aim-instinct-0.11.1`, `0.12.0`, `0.13.0` | `aim-base` 0.11–0.13.1 | Generic `amd-aim-release-*` sources are part of the Instinct profile, not the unfiltered catalog |
| `epyc` | `amd-aim-epyc-0.11.0`, `amd-aim-epyc-0.13.0` | `aim-epyc-base` 0.11, 0.13 | |
| `radeon` | `amd-aim-radeon-0.12.0` | `aim-radeon-base` 0.12 | Preview tags |
| `cpu` | | | Placeholder only; no `AIMClusterModelSource` is rendered |

`instinct` and `radeon` are GPU families; `cpu` and `epyc` are CPU inference
targets. `cpu` and `radeon` use Docker Hub (`docker.io`) under the `silogenai`
org. When the registry requires auth, the chart references `dockerhub-regcred`
in those namespaces; omit or replace that secret if images are public.
targets. Registry is `docker.io`.

## Installing

This chart is normally driven by cluster-bloom via the `AIM_HARDWARE_FAMILY`
install flag, which injects the selected families as a YAML list into
`apps.aim-cluster-model-source.valuesObject.hardwareFamilies` (see the
cluster-forge `root` chart). No comma parsing is involved on that path, the
cluster-forge `root` chart). No comma parsing is involved on that path the
value travels as a structured list.

For a manual `helm` install, prefer a values file or pass a JSON list. A
Expand Down
2 changes: 1 addition & 1 deletion sources/aim-cluster-model-source/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Normalize .Values.hardwareFamilies into a clean list of family tokens.
Accepts a native list (the primary path, injected by cluster-bloom) or a
comma-separated string. Trims whitespace and drops empty tokens. Empty input
yields an empty list, which triggers the legacy branch.
yields an empty list, which triggers the unfiltered catalog (templates/unfiltered.yaml).
*/}}
{{- define "aim.hardwareFamilies" -}}
{{- $raw := .Values.hardwareFamilies -}}
Expand Down
2 changes: 1 addition & 1 deletion sources/aim-cluster-model-source/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,5 @@
# SPDX-License-Identifier: MIT

# Hardware families to install (string or list). Allowed: cpu, epyc, instinct, radeon.
# Empty = legacy behavior (install all aim-models-* sources).
# Empty = unfiltered catalog (Instinct 0.11.1+ plus mixed bases).
hardwareFamilies: []
Loading