From e3326afc1c24ec68f879353de2804a3613fc5581 Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Mon, 17 Aug 2026 21:43:53 +0000 Subject: [PATCH 1/7] Add Qwen3-32B NVFP4 vLLM inference recipe for G4 (inference-perf) --- .../single-host-serving/vllm/README.md | 108 ++++++++++++++++++ .../vllm/inference-perf-config.yml | 36 ++++++ 2 files changed, 144 insertions(+) create mode 100644 inference/g4/qwen3_32b/single-host-serving/vllm/README.md create mode 100644 inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md new file mode 100644 index 00000000..e50fc085 --- /dev/null +++ b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md @@ -0,0 +1,108 @@ +# Single host inference benchmark of Qwen3-32B (NVFP4) with vLLM on G4 + +This recipe shows how to serve and benchmark the [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) model in **NVFP4** precision using [vLLM](https://github.com/vllm-project/vllm) on a single GCP VM with G4 GPUs, driving load with the [inference-perf](https://github.com/kubernetes-sigs/inference-perf) benchmarking tool. For more information on G4 machine types, see the [GCP documentation](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-machine-types). + +This is a high-throughput configuration: the model is served **data-parallel across all 8 GPUs** (one replica per GPU), which maximizes aggregate output throughput on the PCIe-connected G4 platform. + +## Benchmark results + +Measured on `g4-standard-384` (8× NVIDIA RTX PRO 6000 Blackwell), vLLM, NVFP4 weights + FP8 KV cache, ISL/OSL 1024/1024, concurrency 2000, 4000 requests: + +| Metric | Value | +| --- | --- | +| Output throughput | ~27,300 tokens/s | +| Total throughput | ~55,000 tokens/s | +| Request throughput | ~27.1 req/s | +| TTFT (p50) | ~2.1 s | +| TPOT (p50) | ~69.7 ms | +| Successful requests | 4000 / 4000 (0 failures) | + +## Before you begin + +### 1. Create a GCP VM with G4 GPUs + +Make sure you have the following prerequisites: +* [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) is initialized. +* You have a project with a GPU quota. See [Request a quota increase](https://cloud.google.com/docs/quota/view-request#requesting_higher_quota). +* [Enable required APIs](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com). + +The following commands set up environment variables and create a GCE instance. The `MACHINE_TYPE` is set to `g4-standard-384` for an 8-GPU VM. The boot disk is set to 200GB to accommodate the model and dependencies. + +```bash +export VM_NAME="${USER}-g4-vllm-qwen3-32b-nvfp4" +export PROJECT_ID="your-project-id" +export ZONE="your-zone" +export MACHINE_TYPE="g4-standard-384" +export IMAGE_PROJECT="ubuntu-os-accelerator-images" +export IMAGE_FAMILY="ubuntu-accelerator-2404-amd64-with-nvidia-570" + +gcloud compute instances create ${VM_NAME} \ + --machine-type=${MACHINE_TYPE} \ + --project=${PROJECT_ID} \ + --zone=${ZONE} \ + --image-project=${IMAGE_PROJECT} \ + --image-family=${IMAGE_FAMILY} \ + --maintenance-policy=TERMINATE \ + --boot-disk-size=200GB +``` + +### 2. Connect to the VM + +```bash +gcloud compute ssh ${VM_NAME?} --project=${PROJECT_ID?} --zone=${ZONE?} + +# Verify the driver installation and see the available GPUs. +nvidia-smi +``` + +## Serve the model + +### 1. Install Docker and the NVIDIA Container Toolkit + +Follow the official docs to install [Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html), then make sure the Docker daemon is running. + +### 2. Launch the vLLM server + +Serve `RedHatAI/Qwen3-32B-NVFP4` (a pre-quantized NVFP4 checkpoint) data-parallel across the 8 GPUs. The `expandable_segments` allocator setting avoids CUDA fragmentation OOMs in the FP4 GEMM activation buffers, and `gpu_memory_utilization=0.92` leaves headroom for them. + +```bash +docker run --gpus all --ipc=host --network=host \ + -v /scratch/cache:/root/.cache \ + -e VLLM_ATTENTION_BACKEND=FLASHINFER \ + -e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \ + vllm/vllm-openai:latest \ + --model RedHatAI/Qwen3-32B-NVFP4 \ + --tensor-parallel-size 1 \ + --data-parallel-size 8 \ + --kv-cache-dtype fp8 \ + --max-model-len 2560 \ + --gpu-memory-utilization 0.92 \ + --max-num-seqs 512 \ + --max-num-batched-tokens 8192 \ + --no-enable-prefix-caching \ + --block-size 256 +``` + +Wait until the server logs `Application startup complete` before benchmarking. + +## Run the benchmark with inference-perf + +In a second shell on the same VM: + +```bash +pip install inference-perf +``` + +Use the provided [`inference-perf-config.yml`](./inference-perf-config.yml) (concurrency 2000, 4000 requests, ISL/OSL 1024/1024): + +```bash +inference-perf --config_file inference-perf-config.yml +``` + +inference-perf writes a summary report (throughput and TTFT/TPOT/ITL latency distributions) to the output directory. See the [inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) to sweep other input/output shapes or concurrency levels. + +## Clean up + +```bash +gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID} --quiet --delete-disks=all +``` diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml b/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml new file mode 100644 index 00000000..2d848d6b --- /dev/null +++ b/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml @@ -0,0 +1,36 @@ +# inference-perf benchmark config for Qwen3-32B (NVFP4) on G4 with vLLM. +# High-throughput operating point: concurrency 2000, 4000 requests, ISL/OSL 1024/1024. +load: + type: concurrent + stages: + - concurrency_level: 2000 + num_requests: 4000 +api: + type: completion + streaming: true +server: + type: vllm + model_name: RedHatAI/Qwen3-32B-NVFP4 + base_url: http://0.0.0.0:8000 + ignore_eos: true +tokenizer: + pretrained_model_name_or_path: RedHatAI/Qwen3-32B-NVFP4 +data: + type: random + input_distribution: + min: 1024 + max: 1024 + mean: 1024 + std_dev: 0 + total_count: 4000 + output_distribution: + min: 1024 + max: 1024 + mean: 1024 + std_dev: 0 + total_count: 4000 +report: + request_lifecycle: + summary: true + per_stage: true + per_request: false From 5884f2acbad03b139468c52c4bb276ca728eb471 Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Mon, 17 Aug 2026 23:37:31 +0000 Subject: [PATCH 2/7] Convert G4 Qwen3-32B vLLM recipe to inference-perf; add measured throughput (16,181 out tok/s, dp8, 1024/1024) --- .../single-host-serving/vllm/README.md | 108 ------------------ .../g4/single-host-serving/vllm/README.md | 50 ++++++++ .../vllm/inference-perf-config.yml | 6 +- 3 files changed, 53 insertions(+), 111 deletions(-) delete mode 100644 inference/g4/qwen3_32b/single-host-serving/vllm/README.md rename inference/g4/{qwen3_32b => }/single-host-serving/vllm/inference-perf-config.yml (78%) diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md deleted file mode 100644 index e50fc085..00000000 --- a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md +++ /dev/null @@ -1,108 +0,0 @@ -# Single host inference benchmark of Qwen3-32B (NVFP4) with vLLM on G4 - -This recipe shows how to serve and benchmark the [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) model in **NVFP4** precision using [vLLM](https://github.com/vllm-project/vllm) on a single GCP VM with G4 GPUs, driving load with the [inference-perf](https://github.com/kubernetes-sigs/inference-perf) benchmarking tool. For more information on G4 machine types, see the [GCP documentation](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-machine-types). - -This is a high-throughput configuration: the model is served **data-parallel across all 8 GPUs** (one replica per GPU), which maximizes aggregate output throughput on the PCIe-connected G4 platform. - -## Benchmark results - -Measured on `g4-standard-384` (8× NVIDIA RTX PRO 6000 Blackwell), vLLM, NVFP4 weights + FP8 KV cache, ISL/OSL 1024/1024, concurrency 2000, 4000 requests: - -| Metric | Value | -| --- | --- | -| Output throughput | ~27,300 tokens/s | -| Total throughput | ~55,000 tokens/s | -| Request throughput | ~27.1 req/s | -| TTFT (p50) | ~2.1 s | -| TPOT (p50) | ~69.7 ms | -| Successful requests | 4000 / 4000 (0 failures) | - -## Before you begin - -### 1. Create a GCP VM with G4 GPUs - -Make sure you have the following prerequisites: -* [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) is initialized. -* You have a project with a GPU quota. See [Request a quota increase](https://cloud.google.com/docs/quota/view-request#requesting_higher_quota). -* [Enable required APIs](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com). - -The following commands set up environment variables and create a GCE instance. The `MACHINE_TYPE` is set to `g4-standard-384` for an 8-GPU VM. The boot disk is set to 200GB to accommodate the model and dependencies. - -```bash -export VM_NAME="${USER}-g4-vllm-qwen3-32b-nvfp4" -export PROJECT_ID="your-project-id" -export ZONE="your-zone" -export MACHINE_TYPE="g4-standard-384" -export IMAGE_PROJECT="ubuntu-os-accelerator-images" -export IMAGE_FAMILY="ubuntu-accelerator-2404-amd64-with-nvidia-570" - -gcloud compute instances create ${VM_NAME} \ - --machine-type=${MACHINE_TYPE} \ - --project=${PROJECT_ID} \ - --zone=${ZONE} \ - --image-project=${IMAGE_PROJECT} \ - --image-family=${IMAGE_FAMILY} \ - --maintenance-policy=TERMINATE \ - --boot-disk-size=200GB -``` - -### 2. Connect to the VM - -```bash -gcloud compute ssh ${VM_NAME?} --project=${PROJECT_ID?} --zone=${ZONE?} - -# Verify the driver installation and see the available GPUs. -nvidia-smi -``` - -## Serve the model - -### 1. Install Docker and the NVIDIA Container Toolkit - -Follow the official docs to install [Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) and the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html), then make sure the Docker daemon is running. - -### 2. Launch the vLLM server - -Serve `RedHatAI/Qwen3-32B-NVFP4` (a pre-quantized NVFP4 checkpoint) data-parallel across the 8 GPUs. The `expandable_segments` allocator setting avoids CUDA fragmentation OOMs in the FP4 GEMM activation buffers, and `gpu_memory_utilization=0.92` leaves headroom for them. - -```bash -docker run --gpus all --ipc=host --network=host \ - -v /scratch/cache:/root/.cache \ - -e VLLM_ATTENTION_BACKEND=FLASHINFER \ - -e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \ - vllm/vllm-openai:latest \ - --model RedHatAI/Qwen3-32B-NVFP4 \ - --tensor-parallel-size 1 \ - --data-parallel-size 8 \ - --kv-cache-dtype fp8 \ - --max-model-len 2560 \ - --gpu-memory-utilization 0.92 \ - --max-num-seqs 512 \ - --max-num-batched-tokens 8192 \ - --no-enable-prefix-caching \ - --block-size 256 -``` - -Wait until the server logs `Application startup complete` before benchmarking. - -## Run the benchmark with inference-perf - -In a second shell on the same VM: - -```bash -pip install inference-perf -``` - -Use the provided [`inference-perf-config.yml`](./inference-perf-config.yml) (concurrency 2000, 4000 requests, ISL/OSL 1024/1024): - -```bash -inference-perf --config_file inference-perf-config.yml -``` - -inference-perf writes a summary report (throughput and TTFT/TPOT/ITL latency distributions) to the output directory. See the [inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) to sweep other input/output shapes or concurrency levels. - -## Clean up - -```bash -gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID} --quiet --delete-disks=all -``` diff --git a/inference/g4/single-host-serving/vllm/README.md b/inference/g4/single-host-serving/vllm/README.md index 1daab6c1..2e2164c1 100644 --- a/inference/g4/single-host-serving/vllm/README.md +++ b/inference/g4/single-host-serving/vllm/README.md @@ -209,6 +209,56 @@ P99 ITL (ms): XX ================================================== ``` +## High-throughput serving (8 GPUs, data-parallel) + +For maximum aggregate throughput on the balanced `1024/1024` workload, serve the model **data-parallel** (one replica per GPU) instead of tensor-parallel. This saturates all 8 GPUs with independent request streams: + +```bash +sudo docker run \ + --runtime nvidia \ + --gpus all \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ + --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ + -p 8000:8000 \ + --ipc=host \ + vllm/vllm-openai:latest \ + --model Qwen/Qwen3-32B-FP8 \ + --kv-cache-dtype fp8 \ + --max-model-len 2560 \ + --gpu-memory-utilization 0.92 \ + --max-num-seqs 512 \ + --max-num-batched-tokens 8192 \ + --no-enable-prefix-caching \ + --block-size 256 \ + --tensor-parallel-size 1 \ + --data-parallel-size 8 +``` + +## Benchmark with inference-perf + +[inference-perf](https://github.com/kubernetes-sigs/inference-perf) is a model-server-agnostic benchmarking tool that reports standardized throughput and latency metrics. Install it and run against the server above: + +```bash +pip install inference-perf +inference-perf --config_file inference-perf-config.yml +``` + +The provided [`inference-perf-config.yml`](./inference-perf-config.yml) drives 4000 requests at concurrency 2000 with ISL/OSL 1024/1024. See the [inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) to sweep other shapes or concurrency levels. + +### Results (Qwen3-32B-FP8, 8× G4, ISL/OSL 1024/1024) + +| Metric | Value | +| --- | --- | +| Output throughput | **16,181 tokens/s** | +| Total throughput | 32,820 tokens/s | +| Request throughput | 16.25 req/s | +| TTFT (p50) | 9.7 s | +| TPOT (p50) | 95 ms | +| Successful requests | 4000 / 4000 (0 failures) | +> This is a throughput-maximizing operating point (concurrency 2000). For latency-sensitive serving, lower the `concurrency_level` in the config to trade throughput for lower TTFT. + + ## Clean up ### 1. Delete the VM diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml b/inference/g4/single-host-serving/vllm/inference-perf-config.yml similarity index 78% rename from inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml rename to inference/g4/single-host-serving/vllm/inference-perf-config.yml index 2d848d6b..9674ccc4 100644 --- a/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml +++ b/inference/g4/single-host-serving/vllm/inference-perf-config.yml @@ -1,4 +1,4 @@ -# inference-perf benchmark config for Qwen3-32B (NVFP4) on G4 with vLLM. +# inference-perf benchmark config for Qwen3-32B-FP8 on G4 (8 GPUs, data-parallel). # High-throughput operating point: concurrency 2000, 4000 requests, ISL/OSL 1024/1024. load: type: concurrent @@ -10,11 +10,11 @@ api: streaming: true server: type: vllm - model_name: RedHatAI/Qwen3-32B-NVFP4 + model_name: Qwen/Qwen3-32B-FP8 base_url: http://0.0.0.0:8000 ignore_eos: true tokenizer: - pretrained_model_name_or_path: RedHatAI/Qwen3-32B-NVFP4 + pretrained_model_name_or_path: Qwen/Qwen3-32B-FP8 data: type: random input_distribution: From 1f33ed0d1fda80177ac11af6bdeb7f2cce5e8f46 Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Tue, 18 Aug 2026 05:14:38 +0000 Subject: [PATCH 3/7] Move G4 Qwen3-32B vLLM recipe under inference/g4/qwen3_32b/single-host-serving/vllm (matches repo model-dir convention); inference-perf benchmarking with optimized results --- inference/g4/{ => qwen3_32b}/single-host-serving/vllm/README.md | 0 .../single-host-serving/vllm/inference-perf-config.yml | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename inference/g4/{ => qwen3_32b}/single-host-serving/vllm/README.md (100%) rename inference/g4/{ => qwen3_32b}/single-host-serving/vllm/inference-perf-config.yml (100%) diff --git a/inference/g4/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md similarity index 100% rename from inference/g4/single-host-serving/vllm/README.md rename to inference/g4/qwen3_32b/single-host-serving/vllm/README.md diff --git a/inference/g4/single-host-serving/vllm/inference-perf-config.yml b/inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml similarity index 100% rename from inference/g4/single-host-serving/vllm/inference-perf-config.yml rename to inference/g4/qwen3_32b/single-host-serving/vllm/inference-perf-config.yml From ef5a2ff00842a773e046f72722c24d042a1b1a12 Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Wed, 19 Aug 2026 05:36:55 +0000 Subject: [PATCH 4/7] Add inference-perf vLLM recipe for Llama-3.1-70B-FP8 on G4 Serves neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 tensor-parallel across 8 G4 GPUs and benchmarks with inference-perf at ISL/OSL 128/2048. Measured: 4,937 output tok/s, TTFT p50 3.70s, TPOT p50 182ms, 0 failures (~+13% throughput and ~3x lower TTFT vs the tp8 reference at the same shape). Documents the G4-specific requirements (NCCL_P2P_LEVEL=SYS, DeepGEMM disabled for block-quant FP8 on Blackwell) and the load.request_timeout override needed so long OSL=2048 requests are not dropped as timeouts. --- .../single-host-serving/vllm/README.md | 104 ++++++++++++++++++ .../vllm/inference-perf-config.yml | 39 +++++++ 2 files changed, 143 insertions(+) create mode 100644 inference/g4/llama3_1_70b/single-host-serving/vllm/README.md create mode 100644 inference/g4/llama3_1_70b/single-host-serving/vllm/inference-perf-config.yml diff --git a/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md new file mode 100644 index 00000000..e9a65648 --- /dev/null +++ b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md @@ -0,0 +1,104 @@ +# Serve Llama-3.1-70B-FP8 on G4 (NVIDIA RTX PRO 6000) with vLLM + +This recipe serves [`neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8`](https://huggingface.co/neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8) +on a single 8-GPU G4 instance with vLLM, and benchmarks it with +[inference-perf](https://github.com/kubernetes-sigs/inference-perf). + +The 70B model (~70 GB in FP8) does not fit on a single 96 GB RTX PRO 6000, so it is +served **tensor-parallel across all 8 GPUs** (`--tensor-parallel-size 8`). + +## Prerequisites + +- A G4 instance (`g4-standard-384`, 8× NVIDIA RTX PRO 6000). +- Docker with the NVIDIA container runtime. +- A Hugging Face token (`HF_TOKEN`) with access to the model. + +## Serving on 8 GPUs (tensor-parallel) + +G4 instances accelerate multi-GPU workloads with direct GPU +[peer-to-peer](https://cloud.google.com/blog/products/compute/g4-vms-p2p-fabric-boosts-multi-gpu-workloads/) +communication over the PCIe bus. To use it, set `NCCL_P2P_LEVEL=SYS` **before** starting +the server. Without it, tensor-parallel all-reduces fall back to host memory and decode +throughput drops by roughly an order of magnitude. + +```bash +sudo docker run \ + --runtime nvidia \ + --gpus all \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ + --env "NCCL_P2P_LEVEL=SYS" \ + --env "VLLM_USE_DEEP_GEMM=0" \ + --env "VLLM_MOE_USE_DEEP_GEMM=0" \ + -p 8000:8000 \ + --ipc=host \ + vllm/vllm-openai:latest \ + --model neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \ + --kv-cache-dtype fp8 \ + --max-model-len 2560 \ + --gpu-memory-utilization 0.90 \ + --max-num-seqs 1024 \ + --max-num-batched-tokens 8192 \ + --no-enable-prefix-caching \ + --block-size 256 \ + --tensor-parallel-size 8 +``` + +Notes on the arguments: +- `NCCL_P2P_LEVEL=SYS` — **required** on G4 to use the PCIe P2P fabric for TP all-reduces. +- `VLLM_USE_DEEP_GEMM=0` / `VLLM_MOE_USE_DEEP_GEMM=0` — block-quantized FP8 checkpoints + crash at startup on Blackwell with DeepGEMM enabled (`Unknown SF transformation`); + disabling DeepGEMM avoids the crash with no measurable throughput cost. +- `--kv-cache-dtype fp8` — halves KV-cache footprint, leaving ample headroom at 128/2048. +- `--max-model-len 2560` — covers the target 128 + 2048 workload with a small buffer. +- `--max-num-seqs 1024` — decode batch cap; the 128/2048 shape is KV-light so the server + can hold a large decode batch. + +## Benchmark with inference-perf + +[inference-perf](https://github.com/kubernetes-sigs/inference-perf) is a +model-server-agnostic benchmarking tool that reports standardized throughput and latency +metrics. + +```bash +pip install inference-perf +inference-perf --config_file inference-perf-config.yml +``` + +The provided [`inference-perf-config.yml`](./inference-perf-config.yml) drives 4000 +requests at concurrency 960 with ISL/OSL 128/2048. + +> **Important:** OSL=2048 requests take ~360 s end-to-end at high concurrency, which +> exceeds inference-perf's default 300 s request timeout. The config sets +> `load.request_timeout: 900` so long requests complete and are counted instead of being +> dropped as timeouts. + +### Results (Llama-3.1-70B-FP8, 8× G4, tp8, ISL/OSL 128/2048) + +| Metric | Value | +| --- | --- | +| Output throughput | **4,937 tokens/s** | +| Output throughput / GPU | 617 tokens/s | +| Request throughput | 2.52 req/s | +| TTFT (p50) | 3.70 s | +| TPOT (p50) | 182 ms | +| Successful requests | 4000 / 4000 (0 failures) | + +For comparison, a reference tensor-parallel run at the same 128/2048 shape reports +~4,356 output tokens/s at ~11.1 s TTFT (p50). This operating point delivers +**~13% higher output throughput with ~3× lower TTFT**. + +#### Latency-optimized operating point + +Lowering the offered load trades throughput for much lower latency. At a constant arrival +rate of ~2.4 req/s (set `load.type: constant` with `rate: 2.4` in the config), the same +server reaches ~4,300 output tokens/s at **TTFT p50 ~0.20 s** and **TPOT p50 ~62 ms** — +throughput on par with the reference but with dramatically lower latency. + +## Clean up + +Delete the GCE instance and its disks when finished: + +```bash +gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID?} --quiet --delete-disks=all +``` diff --git a/inference/g4/llama3_1_70b/single-host-serving/vllm/inference-perf-config.yml b/inference/g4/llama3_1_70b/single-host-serving/vllm/inference-perf-config.yml new file mode 100644 index 00000000..9deac350 --- /dev/null +++ b/inference/g4/llama3_1_70b/single-host-serving/vllm/inference-perf-config.yml @@ -0,0 +1,39 @@ +# inference-perf benchmark config for Llama-3.1-70B-FP8 on G4 (8 GPUs, tensor-parallel). +# Throughput-maximizing operating point: concurrency 960, 4000 requests, ISL/OSL 128/2048. +# NOTE: OSL=2048 produces long (~360s) requests. inference-perf's default request +# timeout is 300s, which would drop long requests; raise it via load.request_timeout. +load: + type: concurrent + request_timeout: 900 + stages: + - concurrency_level: 960 + num_requests: 4000 +api: + type: completion + streaming: true +server: + type: vllm + model_name: neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 + base_url: http://0.0.0.0:8000 + ignore_eos: true +tokenizer: + pretrained_model_name_or_path: neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 +data: + type: random + input_distribution: + min: 128 + max: 128 + mean: 128 + std_dev: 0 + total_count: 4000 + output_distribution: + min: 2048 + max: 2048 + mean: 2048 + std_dev: 0 + total_count: 4000 +report: + request_lifecycle: + summary: true + per_stage: true + per_request: false From 3c299554cd204b1dd7297c9c6d16b9d5e071c779 Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Wed, 19 Aug 2026 23:50:47 +0000 Subject: [PATCH 5/7] Remove perf result numbers from G4 vLLM READMEs (Qwen3-32B, Llama-3.1-70B) Drop the measured throughput/latency result tables and comparison figures pending internal vetting; keep the serving recipe, config, and tuning notes. --- .../single-host-serving/vllm/README.md | 21 ------------------- .../single-host-serving/vllm/README.md | 12 ----------- 2 files changed, 33 deletions(-) diff --git a/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md index e9a65648..9121bf39 100644 --- a/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md +++ b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md @@ -73,27 +73,6 @@ requests at concurrency 960 with ISL/OSL 128/2048. > `load.request_timeout: 900` so long requests complete and are counted instead of being > dropped as timeouts. -### Results (Llama-3.1-70B-FP8, 8× G4, tp8, ISL/OSL 128/2048) - -| Metric | Value | -| --- | --- | -| Output throughput | **4,937 tokens/s** | -| Output throughput / GPU | 617 tokens/s | -| Request throughput | 2.52 req/s | -| TTFT (p50) | 3.70 s | -| TPOT (p50) | 182 ms | -| Successful requests | 4000 / 4000 (0 failures) | - -For comparison, a reference tensor-parallel run at the same 128/2048 shape reports -~4,356 output tokens/s at ~11.1 s TTFT (p50). This operating point delivers -**~13% higher output throughput with ~3× lower TTFT**. - -#### Latency-optimized operating point - -Lowering the offered load trades throughput for much lower latency. At a constant arrival -rate of ~2.4 req/s (set `load.type: constant` with `rate: 2.4` in the config), the same -server reaches ~4,300 output tokens/s at **TTFT p50 ~0.20 s** and **TPOT p50 ~62 ms** — -throughput on par with the reference but with dramatically lower latency. ## Clean up diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md index 2e2164c1..7bf07c66 100644 --- a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md +++ b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md @@ -246,18 +246,6 @@ inference-perf --config_file inference-perf-config.yml The provided [`inference-perf-config.yml`](./inference-perf-config.yml) drives 4000 requests at concurrency 2000 with ISL/OSL 1024/1024. See the [inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) to sweep other shapes or concurrency levels. -### Results (Qwen3-32B-FP8, 8× G4, ISL/OSL 1024/1024) - -| Metric | Value | -| --- | --- | -| Output throughput | **16,181 tokens/s** | -| Total throughput | 32,820 tokens/s | -| Request throughput | 16.25 req/s | -| TTFT (p50) | 9.7 s | -| TPOT (p50) | 95 ms | -| Successful requests | 4000 / 4000 (0 failures) | -> This is a throughput-maximizing operating point (concurrency 2000). For latency-sensitive serving, lower the `concurrency_level` in the config to trade throughput for lower TTFT. - ## Clean up From 748959f9562f5cac6ec832f37e5e794cda87bcce Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Thu, 20 Aug 2026 00:22:23 +0000 Subject: [PATCH 6/7] Align G4 vLLM serving commands with tuned ground-truth configs Add the env vars from the tuned ubench configs that were missing from the serving recipes so they reproduce the benchmarked setup: - Qwen3-32B dp8: VLLM_ATTENTION_BACKEND=FLASHINFER, VLLM_USE_DEEP_GEMM=0, VLLM_MOE_USE_DEEP_GEMM=0. - Llama-3.1-70B tp8: PYTORCH_CUDA_ALLOC_CONF, VLLM_ATTENTION_BACKEND=FLASHINFER. --- inference/g4/llama3_1_70b/single-host-serving/vllm/README.md | 3 +++ inference/g4/qwen3_32b/single-host-serving/vllm/README.md | 3 +++ 2 files changed, 6 insertions(+) diff --git a/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md index 9121bf39..428a1fc9 100644 --- a/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md +++ b/inference/g4/llama3_1_70b/single-host-serving/vllm/README.md @@ -28,6 +28,8 @@ sudo docker run \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ --env "NCCL_P2P_LEVEL=SYS" \ + --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ + --env "VLLM_ATTENTION_BACKEND=FLASHINFER" \ --env "VLLM_USE_DEEP_GEMM=0" \ --env "VLLM_MOE_USE_DEEP_GEMM=0" \ -p 8000:8000 \ @@ -46,6 +48,7 @@ sudo docker run \ Notes on the arguments: - `NCCL_P2P_LEVEL=SYS` — **required** on G4 to use the PCIe P2P fabric for TP all-reduces. +- `VLLM_ATTENTION_BACKEND=FLASHINFER` — attention backend used for the tuned config. - `VLLM_USE_DEEP_GEMM=0` / `VLLM_MOE_USE_DEEP_GEMM=0` — block-quantized FP8 checkpoints crash at startup on Blackwell with DeepGEMM enabled (`Unknown SF transformation`); disabling DeepGEMM avoids the crash with no measurable throughput cost. diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md index 7bf07c66..091f5d81 100644 --- a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md +++ b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md @@ -220,6 +220,9 @@ sudo docker run \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ + --env "VLLM_ATTENTION_BACKEND=FLASHINFER" \ + --env "VLLM_USE_DEEP_GEMM=0" \ + --env "VLLM_MOE_USE_DEEP_GEMM=0" \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ From 1c14b2d6f3799ec56b71d3a42900504fc50ff86c Mon Sep 17 00:00:00 2001 From: Joey Wang Date: Thu, 20 Aug 2026 00:27:33 +0000 Subject: [PATCH 7/7] Restructure Qwen3-32B G4 vLLM recipe to ground-truth dp8 configs Drop the pre-existing single-GPU and basic tp8 serving sections and the old vllm-bench section; present only the tuned data-parallel (dp8) 1024/1024 recipe that matches the ubench ground-truth configs. Add both FP8 and NVFP4 serving variants (Qwen/Qwen3-32B-FP8 and RedHatAI/Qwen3-32B-NVFP4). --- .../single-host-serving/vllm/README.md | 223 +++++------------- 1 file changed, 59 insertions(+), 164 deletions(-) diff --git a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md index 091f5d81..152ad40e 100644 --- a/inference/g4/qwen3_32b/single-host-serving/vllm/README.md +++ b/inference/g4/qwen3_32b/single-host-serving/vllm/README.md @@ -1,26 +1,32 @@ -# Single host inference benchmark of Qwen3-32B-FP8 with vLLM on G4 +# Serve and benchmark Qwen3-32B on G4 (NVIDIA RTX PRO 6000) with vLLM -This recipe shows how to serve and benchmark Qwen3-32B-FP8 model using [vLLM](https://github.com/vllm-project/vllm) on a single GCP VM with G4 GPUs. vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. For more information on G4 machine types, see the [GCP documentation](https://cloud.google.com/compute/docs/accelerator-optimized-machines#g4-machine-types). +This recipe shows how to serve and benchmark Qwen3-32B on a single 8-GPU G4 instance +with [vLLM](https://github.com/vllm-project/vllm), using +[inference-perf](https://github.com/kubernetes-sigs/inference-perf) for standardized +throughput/latency measurement. Both **FP8** (`Qwen/Qwen3-32B-FP8`) and **NVFP4** +(`RedHatAI/Qwen3-32B-NVFP4`) checkpoints are covered. + +The 32B model fits on a single 96 GB RTX PRO 6000, so for maximum aggregate throughput it +is served **data-parallel** (one replica per GPU, `--data-parallel-size 8`). ## Before you begin ### 1. Create a GCP VM with G4 GPUs -First, we will create a Google Cloud Platform (GCP) Virtual Machine (VM) that has the necessary GPU resources. - Make sure you have the following prerequisites: * [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) is initialized. -* You have a project with a GPU quota. See [Request a quota increase](https://cloud.google.com/docs/quota/view-request#requesting_higher_quota). +* A project with GPU quota. See [Request a quota increase](https://cloud.google.com/docs/quota/view-request#requesting_higher_quota). * [Enable required APIs](https://console.cloud.google.com/flows/enableapi?apiid=compute.googleapis.com). -The following commands set up environment variables and create a GCE instance. The `MACHINE_TYPE` is set to `g4-standard-48` for a single GPU VM, More information on different machine types can be found in the [GCP documentation](https://docs.cloud.google.com/compute/docs/accelerator-optimized-machines#g4-machine-types). The boot disk is set to 200GB to accommodate the models and dependencies. +The following commands set up environment variables and create a GCE instance. +`MACHINE_TYPE` is set to `g4-standard-384` for an 8-GPU VM. The boot disk is set to 200GB +to accommodate the models and dependencies. ```bash export VM_NAME="${USER}-g4-test" export PROJECT_ID="your-project-id" export ZONE="your-zone" -# g4-standard-48 is for a single GPU VM. For a multi-GPU VM (e.g., 8 GPUs), you can use g4-standard-384. -export MACHINE_TYPE="g4-standard-48" +export MACHINE_TYPE="g4-standard-384" export IMAGE_PROJECT="ubuntu-os-accelerator-images" export IMAGE_FAMILY="ubuntu-accelerator-2404-amd64-with-nvidia-570" @@ -36,85 +42,33 @@ gcloud compute instances create ${VM_NAME} \ ### 2. Connect to the VM -Use `gcloud compute ssh` to connect to the newly created instance. - ```bash gcloud compute ssh ${VM_NAME?} --project=${PROJECT_ID?} --zone=${ZONE?} -``` - -``` -# Run NVIDIA smi to verify the driver installation and see the available GPUs. +# Verify the driver installation and available GPUs. nvidia-smi ``` -## Install Dependencies +## Install dependencies ### 1. Install Docker -Before you can serve the model, you need to have Docker installed on your VM. You can follow the official documentation to install Docker on Ubuntu: -[Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/) - -After installing Docker, make sure the Docker daemon is running. +Follow the official documentation to install Docker on Ubuntu: +[Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/). Make +sure the Docker daemon is running. ### 2. Install NVIDIA Container Toolkit -To enable Docker containers to access the GPU, you need to install the NVIDIA Container Toolkit. This toolkit allows the container to interact with the NVIDIA driver on the host machine, making the GPU resources available within the container. - -You can follow the official NVIDIA documentation to install the container toolkit: -[NVIDIA Container Toolkit Install Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) - -## Serve a Model - -### 1. Serving on Single-Chip (1 GPU): - -To run the vLLM server, you can use the following command: - -```bash -sudo docker run \ - --runtime nvidia \ - --gpus all \ - -v ~/.cache/huggingface:/root/.cache/huggingface \ - --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ - -p 8000:8000 \ - --ipc=host \ - vllm/vllm-openai:latest \ - --model Qwen/Qwen3-32B-FP8 \ - --kv-cache-dtype fp8 \ - --max-num-batched-tokens 4096 \ - --max-num-seqs 256 \ - --max-model-len 2300 \ - --gpu-memory-utilization 0.95 \ - --tensor-parallel-size 1 -``` - -For the 32B model on a G4 (1 chip) instance, we recommend `--max-num-batched-tokens 4096`, `--max-num-seqs 256`, and `--max-model-len 2300` for a `2048/128` workload. - -Here's a breakdown of the arguments: -- `--runtime nvidia --gpus all`: This makes the NVIDIA GPUs available inside the container. -- `-v ~/.cache/huggingface:/root/.cache/huggingface`: This mounts the Hugging Face cache directory from the host to the container. This is useful for caching downloaded models. -- `--env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN"`: This sets the Hugging Face Hub token as an environment variable in the container. This is required for downloading models that require authentication. -- `-p 8000:8000`: This maps port 8000 on the host to port 8000 in the container. -- `--ipc=host`: This allows the container to share the host's IPC namespace, which can improve performance. -- `vllm/vllm-openai:latest`: This is the name of the official vLLM docker image. -- `--model Qwen/Qwen3-32B-FP8`: The model to be served from Hugging Face. -- `--kv-cache-dtype fp8`: Sets the data type for the key-value cache to FP8 to save GPU memory. -- `--max-num-batched-tokens 4096`: Maximum number of tokens to be processed in a single iteration. -- `--max-num-seqs 256`: This sets the maximum number of concurrent requests (sequences) the VLLM scheduler keeps actively running in the GPU's KV cache. -- `--max-model-len 2300`: This limits the total sequence length to 2300 tokens. This is sufficient to cover our target workload (Input 2048 + Output 128) while leaving a small buffer for prompt variations. -- `--gpu-memory-utilization 0.95`: The fraction of GPU memory to be used by vLLM. -- `--tensor-parallel-size 1`: It specifies the number of gpu's to use. - -### 2. Serving on Multi-Chip (8 GPU): +Follow the official NVIDIA documentation: +[NVIDIA Container Toolkit Install Guide](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). +This lets the container access the host NVIDIA driver. -G4 instances enhance multi-GPU workload performance by using direct GPU [peer-to-peer](https://cloud.google.com/blog/products/compute/g4-vms-p2p-fabric-boosts-multi-gpu-workloads/) communication. This capability allows GPUs that attach to the same G4 instance to exchange data directly over the PCIe bus, bypassing the need to transfer data through the CPU's main memory. +## Serve the model (8 GPUs, data-parallel) +For maximum aggregate throughput on the balanced `1024/1024` workload, serve the model +data-parallel — one replica per GPU — so all 8 GPUs are saturated with independent request +streams. -``` -To configure NCCL, before you run your workloads, set the NCCL_P2P_LEVEL on your G4 instance by: -export NCCL_P2P_LEVEL=SYS -``` - -we will use the official vllm/vllm-openai:latest Docker image to run the server. +### FP8 (`Qwen/Qwen3-32B-FP8`) ```bash sudo docker run \ @@ -122,96 +76,29 @@ sudo docker run \ --gpus all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ + --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ + --env "VLLM_ATTENTION_BACKEND=FLASHINFER" \ + --env "VLLM_USE_DEEP_GEMM=0" \ + --env "VLLM_MOE_USE_DEEP_GEMM=0" \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ --model Qwen/Qwen3-32B-FP8 \ --kv-cache-dtype fp8 \ - --gpu-memory-utilization 0.95 \ - --tensor-parallel-size 8 -``` - -Here's a breakdown of the arguments: -- `gpus all`: Exposes all 8 available GPUs to the Docker container. -- `tensor-parallel-size 8`: This is the crucial setting, enabling Tensor Parallelism to split the 32B model weights and operations across all 8 GPUs. - -For more information on the available engine arguments, you can refer to the [official vLLM documentation](https://docs.vllm.ai/en/latest/configuration/engine_args/), which includes different parallelism strategies that can be used with multi GPU setup. - -After running the command, the model will be served. To run the benchmark, you will need to either run the server in the background by appending `&` to the command, or open a new terminal to run the benchmark command. - -## Run Benchmarks for Qwen3-32B-FP8 - -### 1. Server Output - -When the server is up and running, you should see output similar to the following. - -``` -(APIServer pid=XXXXXX) INFO XX-XX XX:XX:XX [launcher.py:XX] Route: /metrics, Methods: GET -(APIServer pid=XXXXXX) INFO: Started server process [XXXXXX] -(APIServer pid=XXXXXX) INFO: Waiting for application startup. -(APIServer pid=XXXXXX) INFO: Application startup complete. -``` - -### 2. Run the benchmarks - -To run the benchmark, you can use the following command: - -```bash -sudo docker run \ - --runtime nvidia \ - --gpus all \ - --network="host" \ - --entrypoint vllm \ - vllm/vllm-openai:latest bench serve \ - --model Qwen/Qwen3-32B-FP8 \ - --dataset-name random \ - --random-input-len 2048 \ - --random-output-len 128 \ - --request-rate inf \ - --num-prompts 1000 \ - --ignore-eos -``` -Here's a breakdown of the arguments: -- `--model Qwen/Qwen3-32B-FP8`: The model to benchmark. -- `--dataset-name random`: The dataset to use for the benchmark. `random` will generate random prompts. -- `--random-input-len 2048`: The length of the random input prompts. -- `--random-output-len 128`: The length of the generated output. -- `--request-rate inf`: The number of requests per second to send. `inf` sends requests as fast as possible. -- `--num-prompts 1000`: The total number of prompts to send. -- `--ignore-eos`: A flag to ignore the end-of-sequence token and generate a fixed number of tokens. - -### 3. Example output - -The output shows various performance metrics of the model, such as throughput and latency. - -```bash -============ Serving Benchmark Result ============ -Successful requests: XX -Request rate configured (RPS): XX -Benchmark duration (s): XX -Total input tokens: XX -Total generated tokens: XX -Request throughput (req/s): XX -Output token throughput (tok/s): XX -Total Token throughput (tok/s): XX ----------------Time to First Token---------------- -Mean TTFT (ms): XX -Median TTFT (ms): XX -P99 TTFT (ms): XX ------Time per Output Token (excl. 1st token)------ -Mean TPOT (ms): XX -Median TPOT (ms): XX -P99 TPOT (ms): XX ----------------Inter-token Latency---------------- -Mean ITL (ms): XX -Median ITL (ms): XX -P99 ITL (ms): XX -================================================== + --max-model-len 2560 \ + --gpu-memory-utilization 0.92 \ + --max-num-seqs 512 \ + --max-num-batched-tokens 8192 \ + --no-enable-prefix-caching \ + --block-size 256 \ + --tensor-parallel-size 1 \ + --data-parallel-size 8 ``` -## High-throughput serving (8 GPUs, data-parallel) +`VLLM_USE_DEEP_GEMM=0` / `VLLM_MOE_USE_DEEP_GEMM=0` are required for block-quantized FP8 +checkpoints, which otherwise crash at startup on Blackwell (`Unknown SF transformation`). -For maximum aggregate throughput on the balanced `1024/1024` workload, serve the model **data-parallel** (one replica per GPU) instead of tensor-parallel. This saturates all 8 GPUs with independent request streams: +### NVFP4 (`RedHatAI/Qwen3-32B-NVFP4`) ```bash sudo docker run \ @@ -221,12 +108,10 @@ sudo docker run \ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ --env "PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True" \ --env "VLLM_ATTENTION_BACKEND=FLASHINFER" \ - --env "VLLM_USE_DEEP_GEMM=0" \ - --env "VLLM_MOE_USE_DEEP_GEMM=0" \ -p 8000:8000 \ --ipc=host \ vllm/vllm-openai:latest \ - --model Qwen/Qwen3-32B-FP8 \ + --model RedHatAI/Qwen3-32B-NVFP4 \ --kv-cache-dtype fp8 \ --max-model-len 2560 \ --gpu-memory-utilization 0.92 \ @@ -238,23 +123,33 @@ sudo docker run \ --data-parallel-size 8 ``` +Common flags: +- `--kv-cache-dtype fp8` — halves the KV-cache footprint. +- `--max-model-len 2560` — covers the 1024 + 1024 workload with a small buffer. +- `--tensor-parallel-size 1 --data-parallel-size 8` — one independent replica per GPU. +- `VLLM_ATTENTION_BACKEND=FLASHINFER` — attention backend used for the tuned config. + ## Benchmark with inference-perf -[inference-perf](https://github.com/kubernetes-sigs/inference-perf) is a model-server-agnostic benchmarking tool that reports standardized throughput and latency metrics. Install it and run against the server above: +[inference-perf](https://github.com/kubernetes-sigs/inference-perf) is a +model-server-agnostic benchmarking tool that reports standardized throughput and latency +metrics. Install it and run against the server above: ```bash pip install inference-perf inference-perf --config_file inference-perf-config.yml ``` -The provided [`inference-perf-config.yml`](./inference-perf-config.yml) drives 4000 requests at concurrency 2000 with ISL/OSL 1024/1024. See the [inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) to sweep other shapes or concurrency levels. - +The provided [`inference-perf-config.yml`](./inference-perf-config.yml) drives 4000 +requests at concurrency 2000 with ISL/OSL 1024/1024 against `Qwen/Qwen3-32B-FP8`. For the +NVFP4 server, set `server.model_name` and `tokenizer.pretrained_model_name_or_path` to +`RedHatAI/Qwen3-32B-NVFP4`. See the +[inference-perf configuration guide](https://github.com/kubernetes-sigs/inference-perf/blob/main/docs/config.md) +to sweep other shapes or concurrency levels. ## Clean up -### 1. Delete the VM - -This command will delete the GCE instance and all its disks. +This command deletes the GCE instance and all its disks. ```bash gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID} --quiet --delete-disks=all