Skip to content

refactor(nvidia): run multi-node fixed-sequence recipes on the InferenceX client #3415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
cquil11 merged 10 commits into from
Sep 25, 2026
Merged

refactor(nvidia): run multi-node fixed-sequence recipes on the InferenceX client #3415

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
9e58409
refactor(nvidia): run multi-node fixed-sequence recipes on the Infere…
cquil11 Sep 24, 2026
a5df7d9
fix(power): expect windows for the matrix concurrencies on fixed-sequ…
cquil11 Sep 24, 2026
5a1c189
fix(nvidia): omit --publish-metrics on Dynamo TRT-LLM recipes
cquil11 Sep 24, 2026
4214f78
fix(nvidia): give H100 etcd and NATS time to start
cquil11 Sep 24, 2026
2701ca1
refactor(nvidia): let the client read the served model name
cquil11 Sep 24, 2026
83c85b1
fix(nvidia): give H100 workers an hour to become healthy
cquil11 Sep 25, 2026
826adc8
fix(nvidia): run H100 etcd and NATS in the nginx image
cquil11 Sep 25, 2026
71392fe
fix(nvidia): let TRT-LLM pick the UCX interface on B300
cquil11 Sep 25, 2026
183b207
fix(nvidia): prewarm H100 weights so SGLang TP ranks load in step
cquil11 Sep 25, 2026
2957ea0
Revert "fix(nvidia): let TRT-LLM pick the UCX interface on B300"
cquil11 Sep 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions ...ode/srt-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-1p1d-p-tp4-d-tp8-b512-c32-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "32"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...pes/dsr1/sglang/b200-fp4/8k1k/disagg-1p1d-p-tp4-ep4-dp4-d-tp8-ep8-dp8-b2048-c512-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "512"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions .../srt-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-1p3d-p-tp4-d-tp8-b512-c32x64-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "32x64"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-1p5d-p-tp4-d-tp8-b512-c4x8x16x32-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "4x8x16x32"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...pes/dsr1/sglang/b200-fp4/8k1k/disagg-2p1d-p-tp4-ep4-dp4-d-tp8-ep8-dp8-b2048-c768-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "768"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...es/dsr1/sglang/b200-fp4/8k1k/disagg-3p1d-p-tp4-ep4-dp4-d-tp8-ep8-dp8-b2048-c1024-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "1024"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...pes/dsr1/sglang/b200-fp4/8k1k/disagg-4p1d-p-tp4-ep4-dp4-d-tp8-ep8-dp8-b2048-c512-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "512"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions ...es/dsr1/sglang/b200-fp4/8k1k/disagg-5p1d-p-tp4-ep4-dp4-d-tp8-ep8-dp8-b2048-c2048-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,9 +139,9 @@ health_check:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: 300
concurrencies: "2048"
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
12 changes: 6 additions & 6 deletions benchmarks/multi_node/srt-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-mtp-4p1d-c2048.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@ health_check:
max_attempts: 360
interval_seconds: 10
benchmark:
type: sa-bench
req_rate: inf
isl: 8192
osl: 1024
concurrencies: '2048'
use_chat_template: true
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
24 changes: 6 additions & 18 deletions ...marks/multi_node/srt-slurm-recipes/dsr1/sglang/b200-fp4/8k1k/disagg-stp-mtp-variants.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,12 @@ base:
interval_seconds: 10

benchmark:
type: "sa-bench"
isl: 8192
osl: 1024
req_rate: "inf"
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'


# STP TP4 prefill mode: TP4 (DP=1, EP=1) instead of dep4 — low-latency single-node
Expand All @@ -181,8 +183,6 @@ override_stp_tp4:
decode:
args:
expert-parallel-size: 1
benchmark:
concurrencies: "4x8x16x64"


# MTP TP4 prefill mode: same as STP tp4 but adds EAGLE speculative decoding
Expand All @@ -208,8 +208,6 @@ override_mtp_tp4:
speculative-num-steps: 2
speculative-eagle-topk: 1
speculative-num-draft-tokens: 3
benchmark:
concurrencies: "4x8x16x64"


# STP low-latency: dep4 prefill + tep8 decode (DP=1), scale sweep 1p1d/1p5d/2p5d
Expand All @@ -226,8 +224,6 @@ zip_override_stp_lowlat:
decode:
nodes: [1, 5, 5]
workers: [1, 5, 5]
benchmark:
concurrencies: ["64x128", "8", "4x128"]


# MTP low-latency: same scales as STP, adds EAGLE speculative decoding
Expand All @@ -253,8 +249,6 @@ zip_override_mtp_lowlat:
speculative-num-steps: 2
speculative-eagle-topk: 1
speculative-num-draft-tokens: 3
benchmark:
concurrencies: ["64x128", "8", "4x128"]


# STP max-throughput 7p2d: dep4 prefill + dep8 decode, flashinfer_cutlass backend
Expand Down Expand Up @@ -283,8 +277,6 @@ override_stp_maxtpt_7p2d:
max-running-requests: 2048
cuda-graph-max-bs: 1024
fp4-gemm-backend: "flashinfer_cutlass"
benchmark:
concurrencies: "1024x2048"


# MTP max-throughput 7p2d: same as STP but adds EAGLE speculative decoding
Expand Down Expand Up @@ -320,8 +312,6 @@ override_mtp_maxtpt_7p2d:
speculative-num-steps: 2
speculative-eagle-topk: 1
speculative-num-draft-tokens: 3
benchmark:
concurrencies: "1024x2048"


# MTP-only: 4p1d, no frontends, SGLANG_FLASHINFER_FP4_GEMM_BACKEND env var (fp4-gemm-backend: null
Expand Down Expand Up @@ -365,5 +355,3 @@ override_mtp_maxtpt_4p1d:
speculative-num-steps: 2
speculative-eagle-topk: 1
speculative-num-draft-tokens: 3
benchmark:
concurrencies: "1024"
11 changes: 6 additions & 5 deletions ...ipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p1d-p-tp8-dp8-d-tp8-ep8-dp8-b256-c160x288-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: 160x288
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...ipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p1d-p-tp8-dp8-d-tp8-ep8-dp8-b256-c160x288-stp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: 160x288
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p2d-p-tp8-dp8-d-tp8-ep8-dp8-b128-c288-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '288'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p2d-p-tp8-dp8-d-tp8-ep8-dp8-b128-c288-stp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '288'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...srt-slurm-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p3d-p-tp8-dp8-d-tp8-b32-c128-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '128'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...srt-slurm-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p3d-p-tp8-dp8-d-tp8-b32-c128-stp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '128'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...srt-slurm-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p4d-p-tp8-dp8-d-tp8-b32-c128-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '128'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...srt-slurm-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p4d-p-tp8-dp8-d-tp8-b32-c128-stp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '128'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...ecipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p6d-p-tp8-dp8-d-tp8-b22-c8x16x32x64x128-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,8 +138,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: 8x16x32x64x128
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...ecipes/dsr1/sglang/b200-fp8/8k1k/disagg-1p6d-p-tp8-dp8-d-tp8-b22-c8x16x32x64x128-stp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,8 +136,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: 8x16x32x64x128
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
11 changes: 6 additions & 5 deletions ...-recipes/dsr1/sglang/b200-fp8/8k1k/disagg-2p1d-p-tp8-dp8-d-tp8-ep8-dp8-b512-c512-mtp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,8 +141,9 @@ health_check:
interval_seconds: 10

benchmark:
type: sa-bench
isl: 8192
osl: 1024
req_rate: inf
concurrencies: '512'
type: custom
command: bash /infmax-workspace/benchmarks/multi_node/srt_fixed_sequence.sh
env:
TOKENIZER: /model
ISL: '8192'
OSL: '1024'
Loading