NVIDIA Fleet Intelligence Agent on OpenShift

GPU telemetry collection for OpenShift Container Platform using the NVIDIA Fleet Intelligence Agent connected to a DCGM HostEngine, with metrics exposed to OCP user-workload Prometheus and visualized through the OCP monitoring console.

What is Fleet Intelligence Agent

NVIDIA Fleet Intelligence Agent is an open-source, lightweight host agent that collects GPU and system telemetry by connecting directly to a DCGM HostEngine (nv-hostengine). It is built on top of leptonai/gpud.

What it monitors:

Category	Metrics
GPU utilization	SM active, SM occupancy, GR engine, GPU util %
GPU memory	VRAM used/free/%, DRAM active, memory copy util
GPU compute	Tensor core (HMMA/IMMA), FP16, FP32, FP64 pipeline activity
GPU temperature	Core temp, memory temp, slowdown threshold
GPU power	Power usage, enforced power limit, violations
GPU clocks	SM clock, memory clock
GPU interconnect	NVLink RX/TX bandwidth, NVLink errors, PCIe RX/TX, PCIe replay
GPU health	ECC single/double-bit errors, row remapping, infoROM validity
Host system	CPU utilization, load average, memory, network throughput

Key properties:

Read-only — makes no modifications to the host
Lightweight — less than 500 MB RAM, less than 1% CPU
Exposes a Prometheus /metrics endpoint (port 15133)
Runs as a DaemonSet — one pod per GPU node

Architecture

┌─────────────────────────────────────────────────────────┐
│  DGX Worker Node                                        │
│                                                         │
│  NVIDIA GPUs                                            │
│       │ NVML                                            │
│       ▼                                                 │
│  nvidia-dcgm pod  ──────────────────────────────────┐  │
│  (DCGM HostEngine, port 5555)                       │  │
│  Service: nvidia-dcgm.nvidia-gpu-operator.svc:5555  │  │
│                                                     │  │
│  fleet-intelligence-agent pod  ◄────────────────────┘  │
│  (connects to DCGM HostEngine via DCGM_URL)             │
│  Exposes: http://pod-ip:15133/metrics                   │
└─────────────────────────────────────────────────────────┘
           │
           │  ServiceMonitor scrape every 60s
           ▼
┌──────────────────────────────────────────────┐
│  openshift-user-workload-monitoring          │
│  prometheus-user-workload-0                  │
│  (scrapes /metrics via headless Service)     │
└──────────────────────────────────────────────┘
           │
           │  Thanos federation
           ▼
┌──────────────────────────────────────────────┐
│  openshift-monitoring                        │
│  Thanos Querier                              │
└──────────────────────────────────────────────┘
           │
           │  OCP monitoring plugin queries
           ▼
┌──────────────────────────────────────────────┐
│  OCP Web Console                             │
│  Observe → Dashboards                        │
│  "NVIDIA Fleet Intelligence Agent"           │
└──────────────────────────────────────────────┘

Why DCGM HostEngine mode

The GPU Operator deploys two DCGM-related components when dcgm.enabled: true is set in the ClusterPolicy:

Component	Pod	Purpose
`dcgm`	`nvidia-dcgm-*`	Standalone DCGM HostEngine (`nv-hostengine`) — the telemetry backend
`dcgmExporter`	`nvidia-dcgm-exporter-*`	Prometheus exporter that reads from the HostEngine

The Fleet Intelligence Agent connects directly to the HostEngine (not to the exporter), giving it access to the full DCGM metric set including profiling counters, health checks, and event data that the exporter alone does not expose.

Prerequisites

Requirement	Version	Notes
OpenShift Container Platform	4.21	Tested version
NVIDIA GPU Operator	v26.3	Must be installed and running
DCGM HostEngine	≥ 4.2.3	Deployed automatically by GPU Operator
NVIDIA Driver	≥ 510	On GPU worker nodes
Helm	3.x	For chart installation
NFD Operator	any	Required by GPU Operator for node labeling

Files in this repository

File	Purpose
`nvidia-gpu-cluster.policy.yaml`	GPU Operator ClusterPolicy — enables DCGM HostEngine, driver, device plugin, etc.
`fleet-intelligence-monitoring.yaml`	Enables OCP user-workload Prometheus (one-time cluster setting)
`fleet-intelligence-ocp.yaml`	Namespace + SCC RoleBinding + headless Service + ServiceMonitor
`fleet-intelligence-values.yaml`	Helm values — DCGM URL, listen address, no cloud enrollment
`fleet-intelligence-dashboard.json`	Grafana dashboard for the OCP monitoring plugin

Installation

Step 1 — Install the GPU Operator and apply the ClusterPolicy

The GPU Operator must be installed before the ClusterPolicy CRD exists.

# Create the namespace
oc apply -f nvidia-gpu-operator-ns.yaml

# Create the OperatorGroup
oc apply -f nvidia-gpu-operatorgroup.yaml

# Create the Subscription (installs the operator and registers CRDs)
oc apply -f nvidia-gpu-sub.yaml

# Wait for the CSV to reach Succeeded (takes 2–5 minutes)
oc get csv -n nvidia-gpu-operator -w

Once the CSV shows Succeeded, apply the ClusterPolicy:

oc apply -f nvidia-gpu-cluster.policy.yaml

The ClusterPolicy has the following DCGM-relevant settings:

dcgm:
  enabled: true          # Deploys the DCGM HostEngine (nv-hostengine on port 5555)

dcgmExporter:
  enabled: true          # Deploys the Prometheus exporter (separate from fleet-intelligence)
  serviceMonitor:
    enabled: true        # Creates a ServiceMonitor for the exporter
  config:
    name: ''             # Optional: name of a ConfigMap with custom metrics CSV

Verify the DCGM HostEngine is running and its service is reachable:

oc get pods,svc -n nvidia-gpu-operator | grep dcgm

Expected output includes:

pod/nvidia-dcgm-xxxxx           1/1   Running
service/nvidia-dcgm             ClusterIP   ...   5555/TCP

Step 2 — Enable OCP user-workload monitoring

oc apply -f fleet-intelligence-monitoring.yaml

Wait approximately one minute for the openshift-user-workload-monitoring namespace to become active:

oc get pods -n openshift-user-workload-monitoring

Note on single-worker clusters: OCP hardcodes 2 replicas for prometheus-user-workload and thanos-ruler-user-workload. On a cluster with one schedulable worker node, one replica of each will remain Pending due to pod anti-affinity rules. This is expected and does not affect functionality — the active replica is fully operational.

Step 3 — Create OCP resources

oc apply -f fleet-intelligence-ocp.yaml

This creates:

Namespace fleet-intelligence — with pod-security.kubernetes.io/enforce: privileged required for the agent's hostPID: true container
RoleBinding system:openshift:scc:privileged — grants the agent's ServiceAccount the privileged SCC (equivalent to oc adm policy add-scc-to-user privileged -z fleet-intelligence-agent -n fleet-intelligence)
Service (headless, clusterIP: None) — enables Prometheus to discover and independently scrape each pod by IP
ServiceMonitor — instructs user-workload Prometheus to scrape /metrics on port 15133 every 60 seconds

Why headless? The agent runs as a DaemonSet — one pod per GPU node. A headless Service returns individual pod IPs from DNS so Prometheus scrapes each node separately rather than load-balancing to a single endpoint.

Why no openshift.io/cluster-monitoring: "true" label on the namespace? OCP user-workload Prometheus excludes namespaces that have that label (those are reserved for platform operators). Omitting it allows user-workload Prometheus to discover the ServiceMonitor.

Step 4 — Install the Helm chart

Check the latest available chart version:

helm show chart oci://ghcr.io/nvidia/charts/fleet-intelligence-agent | grep ^version

Install:

helm install fleet-intelligence-agent \
  oci://ghcr.io/nvidia/charts/fleet-intelligence-agent \
  --version 1.3.0 \
  --namespace fleet-intelligence \
  -f fleet-intelligence-values.yaml

Key settings in fleet-intelligence-values.yaml:

# Listen on all interfaces so Prometheus can reach the /metrics endpoint
listenAddress: "0.0.0.0:15133"

env:
  # Points the agent to the DCGM HostEngine deployed by the GPU Operator
  DCGM_URL: "nvidia-dcgm.nvidia-gpu-operator.svc:5555"
  DCGM_URL_IS_UNIX_SOCKET: "false"

# Schedule only on nodes where GPU Operator has deployed the DCGM HostEngine
nodeSelector:
  nvidia.com/gpu.deploy.dcgm: "true"

tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule

# No NVIDIA cloud enrollment — using OCP Prometheus instead
enroll:
  enabled: false

Step 5 — Load the Grafana dashboard

oc create configmap fleet-intelligence-dashboard \
  -n openshift-config-managed \
  --from-file=fleet-intelligence-dashboard.json=fleet-intelligence-dashboard.json \
  --dry-run=client -o yaml | oc apply -f -

oc label configmap fleet-intelligence-dashboard \
  -n openshift-config-managed \
  "console.openshift.io/dashboard=true" \
  "console.openshift.io/odc-dashboard=true" \
  --overwrite

The OCP monitoring plugin reads dashboards from ConfigMaps in openshift-config-managed labelled with console.openshift.io/dashboard=true. No Grafana instance is needed — the plugin renders dashboards natively in the OCP web console.

Dashboard format note: OCP 4.21 uses a monitoring plugin that replaced the standalone Grafana instance. It only renders Grafana 6.x schema (schemaVersion: 14, rows, graph, singlestat panel types, "$datasource" string references). Grafana 8+ format (panels, timeseries, stat) is silently ignored.

Verification

# Confirm pods are running (one per GPU node)
oc get pods -n fleet-intelligence -l app.kubernetes.io/name=fleet-intelligence-agent -o wide

# Confirm the /metrics endpoint is responding
POD=$(oc get pod -n fleet-intelligence -l app.kubernetes.io/name=fleet-intelligence-agent -o name | head -1)
oc exec -n fleet-intelligence $POD -- wget -qO- http://localhost:15133/healthz

# Sample GPU metrics from the endpoint
oc exec -n fleet-intelligence $POD -- wget -qO- http://localhost:15133/metrics | grep dcgm_fi_dev_gpu_util

# Confirm Prometheus is scraping the target
oc -n openshift-user-workload-monitoring exec -c prometheus prometheus-user-workload-0 -- \
  wget -qO- "http://localhost:9090/api/v1/targets?state=active" \
  | python3 -c "
import sys, json
data = json.load(sys.stdin)
for t in data['data']['activeTargets']:
    if 'fleet' in str(t.get('labels','')):
        print('Health:', t['health'], '| URL:', t['scrapeUrl'], '| Last error:', t.get('lastError','none'))
"

Expected Prometheus target output:

Health: up | URL: http://10.x.x.x:15133/metrics | Last error: none

Dashboard

Navigate to Observe → Dashboards → NVIDIA Fleet Intelligence Agent in the OCP web console.

The dashboard has 10 rows covering all GPU and system telemetry:

Row	Content
Overview	Stat panels — GPU count, avg utilization, avg temperature, total power, avg VRAM %, ECC errors, CPU %, host memory %
GPU Utilization	GPU util % per GPU
SM Active & Occupancy	SM active fraction, SM occupancy fraction
Compute Pipelines	Tensor/HMMA/IMMA active, FP16/FP32/FP64 pipelines, DRAM active, GR engine active
GPU Memory (VRAM)	VRAM used %, VRAM used/free bytes, memory copy utilization
Temperature	GPU core temperature + slowdown threshold, GPU memory temperature
Power	Power usage vs enforced limit, power/thermal/board violations
Clocks	SM clock MHz, memory clock MHz
NVLink & PCIe	NVLink RX/TX bandwidth, PCIe RX/TX bandwidth, NVLink errors
ECC Errors & Memory Health	ECC double-bit, single-bit errors, row remap failures, PCIe replay counter
Host System	CPU utilization + load average, host memory, network throughput

A GPU selector variable at the top lets you filter to individual GPUs or view all at once.

Available Metrics

Sample PromQL queries for the OCP Observe → Metrics page:

# GPU utilization per GPU
dcgm_fi_dev_gpu_util{job="fleet-intelligence-agent"}

# GPU memory used %
dcgm_fi_dev_fb_used_percent{job="fleet-intelligence-agent"}

# GPU temperature
dcgm_fi_dev_gpu_temp{job="fleet-intelligence-agent"}

# GPU power draw
dcgm_fi_dev_power_usage{job="fleet-intelligence-agent"}

# Tensor core activity
dcgm_fi_prof_pipe_tensor_active{job="fleet-intelligence-agent"}

# NVLink throughput (bytes/s)
rate(dcgm_fi_prof_nvlink_rx_bytes{job="fleet-intelligence-agent"}[2m])

# ECC double-bit errors (any value > 0 is critical)
dcgm_fi_dev_ecc_dbe_vol_total{job="fleet-intelligence-agent"}

Uninstall

# 1 — Remove the Helm release
helm uninstall fleet-intelligence-agent -n fleet-intelligence

# 2 — Remove OCP resources (namespace, SCC, service, servicemonitor)
oc delete -f fleet-intelligence-ocp.yaml --ignore-not-found

# 3 — Remove the Grafana dashboard
oc delete configmap fleet-intelligence-dashboard -n openshift-config-managed

Important: Always run step 1 before step 2. If you delete the namespace while the Helm release is still installed, the namespace may get stuck in Terminating state due to a stale metrics.k8s.io API discovery. If this happens, force-remove the finalizer:
oc get namespace fleet-intelligence -o json \
  | python3 -c "import sys,json; d=json.load(sys.stdin); d['spec']['finalizers']=[]; print(json.dumps(d))" \
  > /tmp/fleet-ns-finalize.json

oc replace --raw /api/v1/namespaces/fleet-intelligence/finalize \
  -f /tmp/fleet-ns-finalize.json

Troubleshooting

Pods not starting — CrashLoopBackOff or Error

oc logs -n fleet-intelligence -l app.kubernetes.io/name=fleet-intelligence-agent

DCGM connection failures in logs

Verify the DCGM HostEngine service is reachable from the agent namespace:

oc get svc nvidia-dcgm -n nvidia-gpu-operator

Confirm DCGM_URL in the Helm values matches the service FQDN: nvidia-dcgm.nvidia-gpu-operator.svc:5555

Pod stuck in Pending

oc describe pod -n fleet-intelligence -l app.kubernetes.io/name=fleet-intelligence-agent | grep -A 10 Events:

Common causes: missing SCC RoleBinding (re-apply fleet-intelligence-ocp.yaml), or nvidia.com/gpu.deploy.dcgm: "true" label not present on the node (check GPU Operator is healthy).

Prometheus target not appearing

Confirm the namespace does not have the openshift.io/cluster-monitoring: "true" label:

oc get ns fleet-intelligence --show-labels | grep cluster-monitoring

If the label is present, remove it:

oc label namespace fleet-intelligence openshift.io/cluster-monitoring-

Dashboard shows "Bad Gateway"

This indicates the Thanos Querier is down, which is unrelated to the agent itself. Check:

oc get pods -n openshift-monitoring | grep thanos-querier
oc get nodes

A NotReady worker node will cause Thanos Querier to have zero available replicas, which takes down all dashboards cluster-wide.

Name		Name	Last commit message	Last commit date
README.md		README.md

Navigation Menu

Repository files navigation

NVIDIA Fleet Intelligence Agent on OpenShift

Table of Contents

What is Fleet Intelligence Agent

Architecture

Why DCGM HostEngine mode

Prerequisites

Files in this repository

Installation

Step 1 — Install the GPU Operator and apply the ClusterPolicy

Step 2 — Enable OCP user-workload monitoring

Step 3 — Create OCP resources

Step 4 — Install the Helm chart

Step 5 — Load the Grafana dashboard

Verification

Dashboard

Available Metrics

Uninstall

Troubleshooting

About

Releases

Packages

Contributors

Latest commit

History

Folders and files

Repository files navigation

NVIDIA Fleet Intelligence Agent on OpenShift

Table of Contents

What is Fleet Intelligence Agent

Architecture

Why DCGM HostEngine mode

Prerequisites

Files in this repository

Installation

Step 1 — Install the GPU Operator and apply the ClusterPolicy

Step 2 — Enable OCP user-workload monitoring

Step 3 — Create OCP resources

Step 4 — Install the Helm chart

Step 5 — Load the Grafana dashboard

Verification

Dashboard

Available Metrics

Uninstall

Troubleshooting

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors