feat(examples): add OpenAI-compatible benchmark sample

examples/README.md

@@ -21,6 +21,7 @@ Functions are long-running services that respond to HTTP or gRPC invocations.
 | [Ray Serve Helm Chart](function-samples/helmchart-samples/ray-serve-sample/) | Helm chart that deploys a Ray Serve application as an NVCF function. |
 | [Dynamo Operator Sample](function-samples/helmchart-samples/dynamo-operator-sample/) | Helm chart for a vLLM disaggregated router deployed through NVCF. |
 | [Load Tester Supreme](function-samples/load-tester-supreme/) | HTTP and gRPC echo servers designed for load and throughput testing. |
+| [OpenAI-compatible](function-samples/openai-compatible-sample/) | Controllable OpenAI-compatible LLM endpoint target for SDK and load testing. |
 | [gRPC Streaming ASR Client](function-samples/grpc-streaming-asr-client/) | Invoke a Nemotron ASR Streaming NIM over the NVCF gRPC gateway using bidirectional streaming. |
 
 ## Task Samples

examples/function-samples/load-tester-supreme/Dockerfile

@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.22 AS http-server-build
+FROM golang:1.23 AS http-server-build
 
 WORKDIR /go/src/http-app
 COPY http-server .
@@ -43,4 +43,4 @@ COPY grpc-server/grpc_echo_server.py /app/
 ENV GIN_MODE=release
 COPY --from=http-server-build /go/bin/http-app /app/
 
-CMD python3 /app/grpc_echo_server.py & /app/http-app
+CMD python3 /app/grpc_echo_server.py & /app/http-app

examples/function-samples/load-tester-supreme/README.md

@@ -2,7 +2,7 @@
 
 A dual-protocol echo server (HTTP + gRPC) purpose-built for load and throughput
 testing of NVCF deployments. Use this instead of the simpler `grpc-echo-sample`
-for any real load testing — it ships with 500 gRPC worker threads (vs 10) and
+for real load testing. It ships with 500 gRPC worker threads (vs 10) and
 exposes tunable response behaviour.
 
 ## What's included
@@ -19,7 +19,7 @@ exposes tunable response behaviour.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `message` | string | — | Payload content to echo back |
+| `message` | string | Required for HTTP | Payload content to echo back. gRPC accepts an omitted value and returns an empty string. |
 | `repeats` | int | 1 | Number of times to repeat the response |
 | `delay` | float | ~0 | Seconds to sleep between responses |
 | `size` | int | 0 | Generate a random string of this length instead of echoing `message` (HTTP only) |

examples/function-samples/openai-compatible-sample/Dockerfile

@@ -0,0 +1,33 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM golang:1.23 AS build
+
+WORKDIR /src
+COPY http-server/go.mod ./
+RUN go mod download
+COPY http-server/ ./
+RUN CGO_ENABLED=0 go build -o /openai-compatible-sample .
+
+FROM alpine:3.20
+
+COPY --from=build /openai-compatible-sample /app/openai-compatible-sample
+RUN adduser -D -H app && chown app:app /app/openai-compatible-sample
+
+EXPOSE 8000
+
+USER app
+
+CMD ["/app/openai-compatible-sample"]

examples/function-samples/openai-compatible-sample/README.md

@@ -0,0 +1,204 @@
+# OpenAI-compatible sample
+
+A controllable HTTP target for testing NVCF LLM functions with OpenAI client
+libraries and load generators. It returns synthetic text and deterministic
+embedding vectors. It is not a model server and does not implement the full
+OpenAI API.
+
+## Supported routes
+
+| Route | Behavior |
+|-------|----------|
+| `POST /v1/chat/completions` | Chat completions with JSON or SSE output |
+| `POST /v1/completions` | Legacy text completions with JSON or SSE output |
+| `POST /v1/responses` | Responses API JSON or SSE output |
+| `POST /v1/embeddings` | Embeddings for one string or an array of strings |
+| `GET /v1/models` | Lists the sample model |
+| `GET /v1/models/{id}` | Returns metadata for a model ID |
+| `GET /health` | Health check returning 200 |
+
+The sample accepts normal OpenAI request envelopes. It reads the standard
+fields needed to select streaming and embedding output, but does not inspect
+prompt or message content for benchmark tuning. Images, files, tools,
+multimodal input, stored responses, retrieval, and token-array embeddings are
+not implemented.
+
+## Benchmark controls
+
+Benchmark controls use either HTTP headers or top-level JSON body fields. Body
+controls may appear anywhere in the JSON object. The normal OpenAI fields,
+including `model`, `stream`, `input`, `messages`, and `prompt`, are still
+decoded normally.
+
+If any request header starts with `X-Load-Tester-`, all body controls are
+ignored. Header controls and defaults apply instead. The default output chunk
+is `xxxx`. Text controls apply to Chat Completions, Responses, and legacy
+Completions. Queue delay, TTFT, status injection, and concurrency limits apply
+to all POST routes.
+
+| Control | Header | JSON body field | Default | Behavior |
+|---------|--------|-----------------|---------|----------|
+| Queue delay | `X-Load-Tester-Queue-Delay-Ms` | `x_load_tester_queue_delay_ms` | `0` | Delay before processing the request. |
+| TTFT | `X-Load-Tester-TTFT-Ms` | `x_load_tester_ttft_ms` | `0` | Delay before the first response byte. |
+| TTFT jitter | `X-Load-Tester-TTFT-Jitter-Ms` | `x_load_tester_ttft_jitter_ms` | `0` | Random extra delay from 0 through this value. |
+| ITL | `X-Load-Tester-ITL-Ms` | `x_load_tester_itl_ms` | `0` | Delay between streamed output chunks. |
+| ITL jitter | `X-Load-Tester-ITL-Jitter-Ms` | `x_load_tester_itl_jitter_ms` | `0` | Random extra delay between chunks. |
+| Chunk text | `X-Load-Tester-Chunk` | `x_load_tester_chunk` | `xxxx` | Text returned in each output chunk. |
+| Chunk bytes | `X-Load-Tester-Chunk-Bytes` | `x_load_tester_chunk_bytes` | `0` | Generate a random chunk of this byte length. |
+| Output chunks | `X-Load-Tester-Output-Chunks` | `x_load_tester_output_chunks` | `1` | Number of text chunks to return, capped by the startup limit. |
+| Status injection | `X-Load-Tester-Status-Code` | `x_load_tester_status_code` | unset | Return an OpenAI-shaped HTTP error. |
+| Stream error | `X-Load-Tester-Stream-Error-After-Chunks` | `x_load_tester_stream_error_after_chunks` | unset | End a stream with an OpenAI-shaped error after this many chunks. |
+| Stream truncate | `X-Load-Tester-Stream-Truncate-After-Chunks` | `x_load_tester_stream_truncate_after_chunks` | unset | Close a stream without its completion event after this many chunks. |
+| Concurrency limit | `X-Load-Tester-Max-Concurrency` | `x_load_tester_max_concurrency` | `0` | Return 429 when the global in-flight request count exceeds this value. |
+
+Timing values are non-negative integer milliseconds and are capped at five
+minutes. Output is capped at 1 MiB and the startup chunk limit. The
+`LOAD_TESTER_MAX_OUTPUT_CHUNKS` environment variable defaults to 6000 and
+accepts values from 1 through 60000. It is read once when the server starts.
+`Chunk` and `Chunk-Bytes` cannot be combined. Stream error and truncate controls
+are mutually exclusive.
+For a deterministic concurrency test, send the same concurrency limit header
+on every request in the load test.
+
+## Build and run
+
+```bash
+cd examples/function-samples/openai-compatible-sample
+docker build --platform linux/amd64 -t openai-compatible-sample .
+docker run --rm -p 18000:8000 openai-compatible-sample
+```
+
+## Smoke test
+
+```bash
+curl --request POST \
+  --url http://localhost:18000/v1/chat/completions \
+  --header 'Content-Type: application/json' \
+  --header 'X-Load-Tester-Chunk: token' \
+  --header 'X-Load-Tester-Output-Chunks: 3' \
+  --data '{
+    "model": "test-model",
+    "messages": [{"role": "user", "content": "hello"}]
+  }'
+
+curl --request POST \
+  --url http://localhost:18000/v1/responses \
+  --header 'Content-Type: application/json' \
+  --data '{
+    "model": "test-model",
+    "input": "hello",
+    "stream": true,
+    "x_load_tester_ttft_ms": 200,
+    "x_load_tester_itl_ms": 50,
+    "x_load_tester_chunk": "token",
+    "x_load_tester_output_chunks": 3
+  }'
+
+curl --request POST \
+  --url http://localhost:18000/v1/responses \
+  --header 'Content-Type: application/json' \
+  --header 'X-Load-Tester-TTFT-Ms: 200' \
+  --header 'X-Load-Tester-ITL-Ms: 50' \
+  --header 'X-Load-Tester-Chunk: token' \
+  --header 'X-Load-Tester-Output-Chunks: 3' \
+  --data '{
+    "model": "test-model",
+    "input": "hello",
+    "stream": true
+  }'
+
+curl --request POST \
+  --url http://localhost:18000/v1/embeddings \
+  --header 'Content-Type: application/json' \
+  --data '{"model": "test-model", "input": ["one", "two"], "encoding_format": "float"}'
+```
+
+## OpenAI Python client
+
+Use a normal OpenAI client with the sample URL as its `base_url`:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="not-needed",
+    base_url="http://localhost:18000/v1",
+    _strict_response_validation=True,
+)
+
+response = client.responses.create(
+    model="test-model",
+    input="hello",
+    extra_headers={
+        "X-Load-Tester-Chunk": "token",
+        "X-Load-Tester-Output-Chunks": "3",
+    },
+)
+assert response.output_text == "tokentokentoken"
+
+chat = client.chat.completions.create(
+    model="test-model",
+    messages=[{"role": "user", "content": "hello"}],
+)
+assert chat.choices[0].message.content == "xxxx"
+
+embedding = client.embeddings.create(
+    model="test-model",
+    input=["one", "two"],
+    encoding_format="float",
+)
+assert len(embedding.data) == 2
+```
+
+Run the included client compatibility check after starting the Go server:
+
+```bash
+cd examples/function-samples/openai-compatible-sample/http-server
+go run .
+
+# In another terminal, with the openai package installed:
+python3 openai_client_check.py
+```
+
+## 60-second SSE capacity run
+
+Use the matching pinned xk6 binary from `examples/load-tests`. The command
+opens two approximately 60-second streams per VU at 5 ms ITL. Start the sample
+with `LOAD_TESTER_MAX_OUTPUT_CHUNKS=12000` so it accepts the requested output
+shape. Raise the generator file-descriptor limit before using high concurrency.
+
+```bash
+cd examples/load-tests
+ulimit -n 65536
+
+./k6 run functions/oai_compatible_responses_sse_load_test.js \
+  -e OAI_COMPAT_URL=$OAI_COMPAT_URL \
+  -e OPENAI_RESPONSES_PROFILE=calibration \
+  -e OPENAI_RESPONSES_VUS=1024 \
+  -e OPENAI_RESPONSES_ITERATIONS=2 \
+  -e OPENAI_RESPONSES_MAX_DURATION=5m \
+  -e OPENAI_RESPONSES_EXPECTED_DELTAS=12000 \
+  -e OPENAI_RESPONSES_CALIBRATION_TOLERANCE_MS=1 \
+  -e LOAD_TESTER_QUEUE_DELAY_MS=0 \
+  -e LOAD_TESTER_TTFT_MS=1 \
+  -e LOAD_TESTER_TTFT_JITTER_MS=0 \
+  -e LOAD_TESTER_ITL_MS=5 \
+  -e LOAD_TESTER_ITL_JITTER_MS=0 \
+  -e LOAD_TESTER_CHUNK=xxxx \
+  -e LOAD_TESTER_OUTPUT_CHUNKS=12000
+```
+
+## NVCF LLM functions
+
+Expose port 8000 and configure the function inference URL as `/`. Declare the
+routes required by the workload, such as:
+
+```text
+/v1/chat/completions
+/v1/completions
+/v1/responses
+/v1/embeddings
+```
+
+See the [LLM Gateway guide](../../../docs/user/llm-gateway.md) for function
+model configuration and invocation flow.

examples/function-samples/openai-compatible-sample/http-server/go.mod

@@ -0,0 +1,3 @@
+module nvcf-openai-compatible-sample
+
+go 1.23.0