Skip to content

docs: restructure OpenShift guide to align with GPU operator pattern #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sundar-pds merged 1 commit into from
Aug 21, 2026
Merged

docs: restructure OpenShift guide to align with GPU operator pattern #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions .wordlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ GPUs
Grafana
gRPC
Gurunathan
hoc
honorLabels
honorTimestamps
host-device
Expand All @@ -116,10 +117,13 @@ insecureSkipVerify
installdefaultNFDRule
intra
io
iommu
IOMMU
IOV
ipam
ipc
IPC
IPs
IPv
JPG
json
Expand All @@ -141,8 +145,8 @@ Kustomize
labeller
Labeller
learnings
LIF
libionic
LIF
lifecycle
LSP
LTS
Expand Down Expand Up @@ -176,6 +180,8 @@ NFD
nic
NICCTL
NICs
NMState
NNCP
nodeAffinity
NodeFeatureDiscovery
nodelabeller
Expand All @@ -202,12 +208,13 @@ OpenShift's
OperatorGroup
OperatorHub
oyaml
passthrough
Passthrough
pci
pds
PDS
perf
Pensando
perf
Podman
Pollara
PostToolUse
Expand All @@ -231,6 +238,7 @@ relatedImageSignPullSecret
relatedImageWorker
relatedImageWorkerPullSecret
repo
RHCOS
RHEL
roce
RoCE
Expand All @@ -253,6 +261,7 @@ ServiceMonitor
serviceType
SessionEnd
SessionStart
srcversion
SR-IOV
sriov
staticAuthorization
Expand Down
26 changes: 22 additions & 4 deletions docs/_static/cluster-validation-job.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ data:
imagePullPolicy: Always
securityContext:
capabilities:
add: ["IPC_LOCK"]
add: ["IPC_LOCK", "SYS_CHROOT"]
envFrom:
- configMapRef:
name: cluster-validation-config
Expand Down Expand Up @@ -290,10 +290,13 @@ rules:
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update"]

# Allow listing pods by MPIJob controller wait steps
# Allow listing and managing pods, exec for SSH checks, logs for debugging
- apiGroups: [""]
resources: ["pods", "pods/exec", "pods/log"]
verbs: ["get", "list", "watch"]
resources: ["pods", "pods/log"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["get", "list", "watch", "create"]

---
apiVersion: rbac.authorization.k8s.io/v1
Expand All @@ -309,6 +312,21 @@ roleRef:
name: cluster-validation-role
apiGroup: rbac.authorization.k8s.io
---
# OpenShift only: grant privileged SCC to allow SYS_CHROOT capability for sshd in MPI workers.
# On vanilla Kubernetes this ClusterRoleBinding is created but has no effect (the referenced ClusterRole does not exist).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-validation-scc-privileged
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
name: cluster-validation-sa
namespace: default
---

apiVersion: batch/v1
kind: CronJob
Expand Down
15 changes: 15 additions & 0 deletions docs/cluster_validation_framework/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,21 @@ The following prerequisites need to be done on each new node which is intended t

Back-end networking should be set up on the new node, and connectivity to other nodes in the cluster should be configured.

### OpenShift-Specific Requirements

On OpenShift clusters, the following additional configuration is required:

* **`iommu=pt` kernel parameter** must be set via MachineConfig. Without it, GPU IPC
fails and RCCL tests crash. See
[OpenShift (OLM)](../installation/openshift-olm.md#iommu-pt-kernel-parameter) for the
MachineConfig YAML.

### Back-end RoCE Network Configuration

The back-end RoCE network should use **L3 (routed) networking** with each AINIC interface on a separate subnet. RoCE interfaces should be configured with **jumbo frames** (MTU 9000–9216) for optimal performance — this can be set via the [tuning CNI plugin](https://www.cni.dev/plugins/current/meta/tuning/) chained after the AMD Host Device CNI in the NetworkAttachmentDefinition.

For detailed NIC, switch, and routing configuration guidance, refer to the [RoCE Cluster Network Configuration](https://instinct.docs.amd.com/projects/cluster-documentation/latest/how-to/roce-network-config.html) guide.

---

## Architecture
Expand Down
2 changes: 1 addition & 1 deletion docs/drivers/drivers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ Some operating systems may include an in-tree `ionic` kernel module. That module
* sudo update-initramfs -u
* sudo reboot

**OpenShift:** The `spec.driver.blacklist` field is not supported on OpenShift. Instead, use a MachineConfig resource to blacklist the in-tree ionic module. See the [OpenShift Installation Guide](../openshift/installation-guide.md#2-blacklist-in-tree-ionic-driver-recommended) for details.
**OpenShift:** The `spec.driver.blacklist` field is not supported on OpenShift. Instead, use a MachineConfig resource to blacklist the in-tree ionic module. See the [OpenShift Installation Guide](../installation/openshift-olm.md#2-blacklist-in-tree-ionic-driver-recommended) for details.
```

For example:
Expand Down
Loading
Loading