Add inference-perf vLLM recipes for Qwen3-32B-FP8 and Llama-3.1-70B-FP8 on G4

[joeywan-google]

Summary

Adds inference-perf-based vLLM serving recipes for G4 (NVIDIA RTX PRO 6000), with measured, reproducible throughput/latency.

1. Qwen3-32B-FP8 (inference/g4/qwen3_32b/single-host-serving/vllm/)

• 8× G4, data-parallel (dp8), ISL/OSL 1024/1024.
• 16,181 output tok/s, 0 failures.

2. Llama-3.1-70B-FP8 (inference/g4/llama3_1_70b/single-host-serving/vllm/)

• 8× G4, tensor-parallel (tp8), ISL/OSL 128/2048.
• 4,937 output tok/s, TTFT p50 3.70 s, TPOT p50 182 ms, 0 failures.
• ~+13% output throughput and ~3× lower TTFT vs the tp8 reference at the same shape.
• Documents G4-specific requirements: NCCL_P2P_LEVEL=SYS, DeepGEMM disabled for block-quant FP8 on Blackwell, and the load.request_timeout override so long OSL=2048 requests are not dropped as timeouts.

Each recipe ships a README (serving + benchmark commands) and an inference-perf-config.yml.

[gemini-code-assist]

Code Review

This pull request introduces a benchmarking recipe for serving the Qwen3-32B model in NVFP4 precision using vLLM on a single GCP VM with G4 GPUs, including a detailed README and an inference-perf configuration file. Feedback on the README recommends prefixing Docker commands with sudo, mounting a standard cache directory, setting up a Python virtual environment to prevent PEP 668 installation errors, and ensuring environment variable safety in the cleanup command. Additionally, the reviewer suggests using localhost instead of 0.0.0.0 for the server's base URL in the configuration file.

[gemini-code-assist]

On standard GCP Ubuntu VMs, the default user does not have permission to run Docker commands without sudo unless explicitly added to the docker group. To ensure the command runs successfully out-of-the-box, prefix it with sudo (consistent with other recipes in this repository).

Additionally, mounting /scratch/cache is non-standard since no local SSDs are attached to the VM in the creation step, meaning /scratch does not exist by default. It is cleaner and safer to mount ~/.cache/huggingface to avoid polluting the root directory and prevent potential permission issues.

Suggested change

	docker run --gpus all --ipc=host --network=host \
	-v /scratch/cache:/root/.cache \
	-e VLLM_ATTENTION_BACKEND=FLASHINFER \
	-e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \
	vllm/vllm-openai:latest \
	--model RedHatAI/Qwen3-32B-NVFP4 \
	--tensor-parallel-size 1 \
	--data-parallel-size 8 \
	--kv-cache-dtype fp8 \
	--max-model-len 2560 \
	--gpu-memory-utilization 0.92 \
	--max-num-seqs 512 \
	--max-num-batched-tokens 8192 \
	--no-enable-prefix-caching \
	--block-size 256
	```
	sudo docker run --gpus all --ipc=host --network=host \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	-e VLLM_ATTENTION_BACKEND=FLASHINFER \
	-e PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True" \
	vllm/vllm-openai:latest \
	--model RedHatAI/Qwen3-32B-NVFP4 \
	--tensor-parallel-size 1 \
	--data-parallel-size 8 \
	--kv-cache-dtype fp8 \
	--max-model-len 2560 \
	--gpu-memory-utilization 0.92 \
	--max-num-seqs 512 \
	--max-num-batched-tokens 8192 \
	--no-enable-prefix-caching \
	--block-size 256

[gemini-code-assist]

The VM image family used (ubuntu-accelerator-2404-amd64-with-nvidia-570) is based on Ubuntu 24.04, which enforces PEP 668 (externally managed environments). Running pip install directly on the host system will fail with an externally-managed-environment error.

To prevent this, it is recommended to create and activate a Python virtual environment before installing inference-perf.

Suggested change

	pip install inference-perf
	python3 -m venv venv
	source venv/bin/activate
	pip install inference-perf

[gemini-code-assist]

For safety and consistency with other commands in this recipe (such as the SSH command on line 52), use ${PROJECT_ID?} instead of ${PROJECT_ID} to ensure the command fails immediately if the environment variable is not set.

Suggested change

	gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID} --quiet --delete-disks=all
	gcloud compute instances delete ${VM_NAME?} --zone=${ZONE?} --project=${PROJECT_ID?} --quiet --delete-disks=all

[gemini-code-assist]

Using 0.0.0.0 as a destination address in a client configuration is non-standard and can fail on certain operating systems or strict HTTP clients, as 0.0.0.0 is meant for binding/listening rather than routing. It is safer and more portable to use localhost or 127.0.0.1 for client connections.

  base_url: http://localhost:8000