refactor: migrate fixed-sequence recipes to SRT-Slurm / 将定长配方迁移至 SRT-Slurm

[adibarra]

Converts all 50 active Slurm single-node fixed-sequence configs to native SRT recipes: 35 SGLang, eight TRT-LLM, and seven ATOM, covering all 407 existing points. This includes DeepSeek-R1 and Qwen3.5 on H100/H200/B200/B300 and MI300X/MI325X/MI355X. Models, images, concurrency ranges, topology, and serving settings are preserved. Every converted entry exists in current main's active master configs; the deprecation cleanup in #3349 is included. The two Docker-only RTX PRO 6000 fixed-sequence configs are now retired into the existing NVIDIA archive, with original settings and scripts preserved.

Fixed-sequence jobs in the eleven Slurm pools now require an SRT recipe. There is no fallback to the former single-node Bash implementation; the 50 replaced scripts are removed. Active single-node fixed-sequence coverage is SRT-only: the two RTX Docker configs and their scripts are archived, and their unused runner mappings, launcher, and runtime settings are removed. There is no active fixed-sequence Bash/Docker fallback. Changelog planning preserves entries naming archived configs without scheduling them; unknown keys and retired append-only selections still fail. AgentX, existing multi-node paths, and explicitly selected SPEED-Bench collectors retain their current execution paths. AgentX migration will be a separate PR.

Recipes own serving and workload settings; pool profiles own scheduling, mounts, model locations, and image caches. The connector selects exactly one native variant, validates it against the matrix, and applies runtime inputs. Submission, allocation state, cancellation, result/power staging, and native provenance are checked explicitly. Native eval dispatch uses the existing InferenceX eval implementation and propagates eval or staging failures.

Concurrency-dependent settings use existing upstream SRT zip_override_* support. Lists within a group pair by index and override base; they are not a Cartesian product. For example, DeepSeek-R1 H200 TRT MTP uses MTP3 without DP attention at c4–32, then MTP1 with DP attention at c64–256, with corresponding batch/token budgets. Qwen3.5 H200 SGLang varies CUDA graph batch size with concurrency. InferenceX selects the indexed variant matching the job's concurrency and topology.

The SRT submodule now uses NVIDIA upstream main 3cbc5dd256af2bfd2fed09b724628c3f5456c85f, including merged AMD, MoRI, ATOM/AToMesh, and native TRT-LLM served-model-name support. This migration no longer depends on fork #27. The seven ATOM recipes select native frontend.type: atomesh with a single aggregate worker. Their historical worker images remain unchanged; the router alone uses official rocm/atom-dev:nightly_202609161445, pinned by digest sha256:f00a9dc588a380038fdad54cd57a956043cb316ddda54ebb1b62a24f8caa343b through frontend.container_image. The cached May worker image does not include AToMesh; the official router image tag/digest and cached executable were verified. TRT recipes use native engine.served_model_name forwarding, removing the now-duplicate extra_args flag. No worker image, model argument, concurrency range, topology, or benchmark setting was tuned, and no Bash branches were added. The pre-existing TileRT legacy checkout remains outside this migration.

Upstream-switch validation at 66781f6c0: 152 focused CPU tests passed (122 InferenceX integration checks and 30 upstream ATOM/TRT/SGLang checks). Two stubbed-launcher tests required an unsandboxed rerun because macOS denied /dev/stderr; both passed there. All 50 configs / 407 points passed native schema, topology, single-point submission and worker-command generation for throughput and eval-only modes (814 checks), including 73 AToMesh variants and 63 TRT variants with exactly one native served-name argument. The before/after recipe comparison confirms port-only changes. Changelog matrix generation and historical byte preservation passed. Fresh GPU validation of this upstream pin and AToMesh request path is pending: all ten MI355X compute nodes were allocated at the capacity check, so no new job was queued. The hardware measurements below remain evidence for their explicitly recorded earlier pins, not for this change.

Validation:

• Local CPU suite at 8f3f0d551 after retiring the remaining Docker configs and fixing archived changelog selection: 1,991 passed, 1 skipped, and 187 subtests passed. Ruff, Bash syntax, changelog validation, and historical changelog byte preservation passed.

• All 407 points passed native schema, topology, submission selection, and expanded legacy/native command comparisons. Eval-only generation also passed for all 407 points. These checks do not establish hardware qualification.

• Qwen3.5 H200 c4, runtime a88a27a7cdf04d9f7b9e87492e6abdeadd38fc2c: 40/40 requests, 432.42 output tokens/s, about -0.2% versus the matching published baseline.

• Qwen3.5 H200 c16, same runtime: 160/160 requests, 1002.34 output tokens/s, about +0.6% versus the matching published baseline. Both runs use SGLang v0.5.14-cu130, FP8, TP8/EP8, 8k1k, and pass required eight-GPU power validation.

• Qwen3.5 H200 c64, runtime 2763e8d002efd1786d7c52e30c73410c0d0803c2: 640/640 requests, 1741.50 output tokens/s, about -1.5% versus the matching published baseline. Required eight-GPU power validation passed (maximum gap 1.183 s); Slurm completed with 0:0, and result collection succeeded.

• DeepSeek-R1 H200 c4, runtime 348f773eb6fba29cdb4216d41240cff122a4522e: 40/40 requests, 344.78 output tokens/s, about -0.04% versus the matching published baseline; required power validation passed.

• DeepSeek-R1 TRT MTP H200 c4, runtime b4724b08bb1488105ff6fba32a46ad37470d6417, SRT 2ac4eb1367dd2a78f597a72ca91afe4211d76b38: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, MTP3, 8k1k, concurrency 4. Native E2E passed: 40/40 measured requests, 416.27 output tokens/s, correct served model name, eight-GPU power validation, result collection, and allocation cleanup. This exercises the existing recipe extra_args path without [NVIDIA] Add GB200 DSR1 FP4 TRT #26. A sequential legacy control at main 7e257acec661842d9b7f5651f92b4c29ea162fd7 completed 40/40 requests at 414.86 output tokens/s with identical input/output token totals: native throughput was +0.34%. The control workflow failed result discovery; its raw JSON was later included in the power-audit artifact. Offline validation of the unmodified telemetry in the verified cluster UTC timezone passed for all eight GPUs. That recovery does not change the failed workflow conclusion. This is one point on different H200 nodes, not full performance or accuracy qualification. Both allocations are released.

• DeepSeek-R1 TRT MTP H200 c64, runtime b4724b08, SRT 2ac4eb1: FP8, TRT-LLM 1.3.0rc14, TP8/EP8 with DP attention, MTP1, 8k1k, 640/640 measured requests, 1622.04 output tokens/s. E2E, eight-GPU power validation, result collection, and cleanup passed. This verifies the higher-concurrency recipe variant; no fresh c64 control has been completed.

• Qwen3.5 H200 eval integration, runtime a67435c0f8d2cb2e2d0c61c829ca4197a1368e64: SGLang v0.5.14-cu130, FP8, TP8/EP8, concurrency 4, GSM8K first 16 samples. All requests completed; strict and flexible exact match were 0.9375 (15/16). The workflow score gate failed against Qwen's 0.94 threshold. Native eval receipt was 0, metadata/artifacts staged, and Slurm completed 0:0. This proves the exercised execution/artifact path, not accuracy qualification; the limited-sample failure remains visible and the threshold is unchanged.

• MI300X Qwen3.5 c4 startup check, runtime 571fa51b3, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, 8k1k. The allocation failed before server startup because the cluster preamble ended with a newline before the native && join; zero measured requests. The allocation was released. Commit 0bd3bf5f7 fixes both affected AMD preambles and adds controller-based terminal verification for pools without working accounting; native shell execution and negative status cases passed locally. The hardware retry at 8e279188e reached server readiness but failed before benchmark requests because pip --user is incompatible with the ROCm image virtual environment. Commit c430090a8 removes the forced user-site install. The AMD c4 hardware retry at 6f60a8bf2, SRT c29ef7c, passed E2E: 40/40 measured requests, 238.58 output tokens/s versus 241.17 in the matching 2026-05-17 InferenceX API baseline (-1.07%). Mean TTFT was 1.0813 s versus 0.9888 s (+9.35%); mean TPOT was 15.227 ms versus 15.143 ms (+0.56%). Required power passed on exactly eight GPUs (maximum sample gap 2 s); artifacts were collected and Slurm completed 0:0, with the allocation released. This historical comparison does not establish latency parity or accuracy qualification.

• DeepSeek-R1 B200 SGLang MTP c1, runtime c430090a8, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP1, 8k1k. 10/10 requests completed, 297.44 output tokens/s versus 296.69 in the matching InferenceX API baseline dated 2026-08-06 (+0.25%); mean TTFT -3.59%, mean TPOT -0.02%. Slurm completed 0:0 and released the allocation. The workflow failed required power validation because the exclusive eight-GPU allocation exposed eight devices to the client instead of four; this is measured throughput evidence, not a green smoke. Commit 6f60a8bf2 limits native server/client steps to the validated serving GPU count while retaining node exclusivity. The verification run at 6f60a8bf2 passed E2E: 10/10 requests, 297.74 output tokens/s (+0.36% against the same API baseline), mean TTFT -3.79%, mean TPOT -0.11%. Power validation observed exactly four GPUs (maximum sample gap 1.071 s), artifacts were collected, and Slurm completed 0:0 with its allocation released. The five-recipe, 16-point representative batch is paced one allocation at a time. H200 c4 has now passed, completing initial low-concurrency coverage for all five recipes. The B200 Qwen TRT-MTP c4 run completed all three TP2/TP4/TP8 variants serially with required power validation. MI355X ATOM c4 now passes required power and cleanup after the sampler fix, as detailed below. The H200 c4 smoke passed at 77961ed42 with required power and no evals. MI300X c16 has since passed, as detailed below. H200 c64 has also passed, including its DP attention/MTP1 settings, as detailed below. B200 c32 has also passed with exactly four power-valid GPUs, as detailed below. H200 c256 and MI300X c64 have also passed, completing both recipes’ selected low/medium/high points. B200 SGLang c256 has now passed as well, completing its selected c1/c32/c256 points. B200 Qwen TRT c128 has also passed. MI355X ATOM c32 and c256 have now passed, completing the selected batch at 16/16. The final MI355X ATOM c256 point has passed. All 16 selected points across five recipes are complete: 11,290/11,290 measured requests, required power validation and cleanup passed, and all task-owned allocations are released. The batch always ran at most one task-owned GPU allocation. These no-eval smokes do not qualify all 50 converted configs or establish full performance or accuracy parity. The Qwen TP8 c4 historical delta remains unresolved: current exact CLI selection expands all three TP variants and rejects their shared experiment name as ambiguous, so no three-variant rerun was dispatched for a single follow-up point.

• Qwen3.5 B200 TRT-MTP c4, TP2, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP2/EP1, DP attention off, 8k1k. The benchmark job passed with 40/40 measured requests and 770.39 output tokens/s versus 769.23 in the matching 2026-06-24 API baseline (+0.15%). Mean TTFT +2.80%; mean TPOT -0.68%. Required power passed on exactly two GPUs (maximum sample gap 1.040 s); Slurm 16906 completed 0:0 and released its allocation. TP4 and TP8 also passed as recorded below; the full three-variant workflow completed successfully. No accuracy eval was requested.

• Qwen3.5 B200 TRT-MTP c4, TP8, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP8/EP8, DP attention off, 8k1k. The benchmark job passed with 40/40 measured requests and 1042.85 output tokens/s versus 1081.62 in the matching 2026-06-24 API baseline (-3.58%). Mean TTFT +3.55%; mean TPOT +2.17%. Required power passed on eight GPUs (maximum sample gap 1.001 s); Slurm 16907 completed 0:0 and released its allocation before TP4 started. This throughput decrease is flagged for a fresh comparison; a single historical-baseline smoke cannot identify its cause or establish performance parity.

• Qwen3.5 B200 TRT-MTP c4, TP4, runtime 6f60a8bf2, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP4/EP4, DP attention off, 8k1k. 40/40 requests passed, 942.20 output tokens/s versus 917.70 in the matching 2026-06-24 API baseline (+2.67%); mean TTFT -6.64%, mean TPOT -1.68%. Required power passed on exactly four GPUs (maximum sample gap 1.001 s); Slurm 16908 completed 0:0 and released its allocation. All three c4 TP variants and result collection finished successfully.

• MI355X DSR1 ATOM-MTP c4 startup failure, runtime b3a5a7f39, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, 8k1k. Python 3.12 raised IndexError in importlib.cache_from_source while importing a PyTorch generated module from / with PYTHONPYCACHEPREFIX enabled. Zero benchmark requests; no throughput or accuracy result. Slurm 45548 failed 1:0 and released its allocation. The same import-path failure was reproduced locally. Commit 161aa243a restores the legacy container working-directory behavior through native srun_options.container-workdir=/infmax-workspace, using the existing repository mount and preserving caller overrides. All 69 focused CPU tests, native command generation, Ruff and changelog validation passed. The retry at 838607d8a completed 40/40 requests, confirming the startup fix; its separate power-validation failure is recorded below.

• MI355X DSR1 ATOM-MTP c4 measurement, runtime 838607d8a, SRT c29ef7c: FP4, TP8/EP1, DP attention off, 8k1k. 40/40 requests completed, 611.90 output tokens/s versus 605.22 in the matching 2026-05-20 API baseline (+1.10%); mean TTFT -2.56%, mean TPOT -0.35%. Slurm 45549 completed 0:0 and released its allocation. The workflow failed required power validation: all eight GPUs were observed, but CSV samples stopped 3.68–4.68 s before the benchmark ended (benchmark_window_not_bracketed). This is measured throughput, not a green smoke or accuracy qualification. The existing CSV filter was reproduced buffering live pipe input with mawk despite fflush(). Commit dd955705c replaces it with a shared Bash line reader, preserving validation and benchmark settings. All 237 focused CPU tests, a Linux streaming check, Ruff, shell syntax and append-only changelog validation passed. The c4 verification run at dd955705c passed E2E: 40/40 requests, 610.71 output tokens/s (+0.91%) against the same May 20 API baseline; mean TTFT -0.36%, mean TPOT -1.39%. Required power passed on all eight GPUs with a maximum sample gap of 2 s. Slurm 45550 completed 0:0 and released its allocation. No accuracy eval was requested. That MI355X allocation and the subsequent H200 c4 allocation are released.

H200 exact selection exposed stale single-digit entries in configs/runners.yaml: _7 matched no registered _07 runner. Commit 77961ed42 aligns both H200 labels with all 14 live registered two-digit names. Eight runner-filter CPU tests, exact one-point generation and append-only changelog validation passed. Benchmark settings are unchanged.

• H200 DSR1 TRT-MTP c4, previous fork pin, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention off, MTP3, 8k1k. E2E passed, 40/40 requests, 413.62 output tokens/s versus 413.01 in the matching 2026-05-18 API baseline (+0.15%); mean TTFT -1.97%, mean TPOT -0.36%. Required power passed on all eight GPUs (maximum sample gap 1.164 s); source revisions and artifacts were verified. Slurm 88762 completed 0:0 and released its allocation. No accuracy eval was requested.

• MI300X Qwen3.5 SGLang c16, runtime 77961ed42, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, DP attention off, no speculation, 8k1k. E2E passed, 160/160 measured requests, 650.52 output tokens/s versus 652.14 in the matching 2026-05-17 API baseline (-0.25%); mean TTFT +0.89%, mean TPOT +0.21%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions and artifacts were verified. The native terminal-state gate and workflow cleanup passed; Slurm allocation 1159 was independently confirmed absent from the live queue. No accuracy eval; the historical comparison does not establish full performance parity.

• H200 DSR1 TRT-MTP c64, previous fork pin, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention enabled, MTP1, 8k1k. E2E passed, 640/640 measured requests, 1629.27 output tokens/s versus 1628.59 in the matching 2026-05-18 API baseline (+0.04%); mean TTFT -3.64%, mean TPOT +0.29%. Required power passed on all eight GPUs (maximum sample gap 1.154 s). The emitted TRT config confirms the concurrency-dependent settings. Source revisions, artifacts and cleanup were verified; Slurm 88777 completed 0:0 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• B200 DSR1 SGLang-MTP c32, runtime 77961ed42, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP1, DP attention off, 8k1k. E2E passed, 320/320 measured requests, 1699.06 output tokens/s versus 1717.36 in the matching 2026-08-06 API baseline (-1.07%); mean TTFT -2.17%, mean TPOT +1.20%. Required power passed on exactly four GPUs (maximum sample gap 1.072 s); native submission uses srun_options.gpus-per-node=4. Source revisions, artifacts and cleanup were verified; Slurm 16910 completed 0:0 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• H200 DSR1 TRT-MTP c256, runtime 77961ed42, SRT c29ef7c: FP8, TRT-LLM 1.3.0rc14, TP8/EP8, DP attention enabled, MTP1, 8k1k. E2E passed, 2560/2560 measured requests, 2639.48 output tokens/s versus 2636.47 in the matching 2026-05-18 API baseline (+0.11%); mean TTFT -1.69%, mean TPOT -0.02%. Required power passed on all eight GPUs (maximum sample gap 1.169 s); emitted TRT settings include max_batch_size: 32. Source revisions, artifacts and cleanup were verified; Slurm 88778 completed 0:0 and released its allocation. The selected H200 c4/c64/c256 points have all passed; no accuracy eval or full performance qualification is claimed.

• MI300X Qwen3.5 SGLang c64, runtime 77961ed42, SRT c29ef7c: FP8, SGLang v0.5.12-rocm720-mi30x, TP8/EP1, DP attention off, no speculation, 8k1k. E2E passed, 640/640 measured requests, 1385.23 output tokens/s versus 1387.16 in the matching 2026-05-17 API baseline (-0.14%); mean TTFT +4.29% (2.0803 s versus 1.9947 s), mean TPOT -0.09%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 1160 completed 0:0 and released its allocation. MI300X selected c4/c16/c64 are complete. No accuracy eval; the historical comparison does not establish full performance parity.

• B200 DSR1 SGLang-MTP c256, runtime 77961ed42, SRT c29ef7c: FP4, SGLang v0.5.16-cu130, TP4/EP4, DP attention enabled, 8k1k. E2E passed, 2560/2560 measured requests, 3843.97 output tokens/s versus 3904.54 in the matching 2026-08-06 API baseline (-1.55%); mean TTFT +1.89%, mean TPOT +1.68%. Server visibility and required power both covered exactly four GPUs (maximum sample gap 1.003 s). Source revisions, artifacts and cleanup were verified; Slurm 16911 completed 0:0 and released its allocation. Selected B200 SGLang c1/c32/c256 are complete, including the EP4/DP-attention transition. No accuracy eval; historical comparisons do not establish full performance parity.

• B200 Qwen3.5 TRT-MTP c128, runtime 77961ed42, SRT c29ef7c: FP4, TRT-LLM 1.3.0rc18, TP8/EP8, DP attention enabled, 8k1k. E2E passed, 1280/1280 measured requests, 8977.02 output tokens/s versus 8801.95 in the matching 2026-06-24 API baseline (+1.99%); mean TTFT -4.04%, mean TPOT +2.34%. Required power passed on all eight GPUs (maximum sample gap 1.140 s). Source revisions, artifacts and cleanup were verified; Slurm 16912 completed 0:0 and released its allocation. This covers the selected DP-attention transition; it does not resolve the separate TP8 c4 historical throughput decrease or qualify accuracy.

• MI355X DSR1 ATOM-MTP c32, runtime 77961ed42, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, DP attention off, 8k1k. E2E passed, 320/320 measured requests, 2002.39 output tokens/s versus 1978.91 in the matching 2026-05-20 API baseline (+1.19%); mean TTFT +1.28%, mean TPOT -1.14%. Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 45584 completed 0:0 on mia1-p01-g14 and released its allocation. No accuracy eval; the historical comparison does not establish full performance parity.

• MI355X DSR1 ATOM-MTP c256, runtime 77961ed42, SRT c29ef7c: FP4, ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511, TP8/EP1, DP attention off, 8k1k. E2E passed, 2560/2560 measured requests, 3492.79 output tokens/s versus 3471.59 in the matching 2026-05-20 API baseline (+0.61%); mean TTFT +5.88% (2.8468 s), mean TPOT -0.91% (69.409 ms). Required power passed on all eight GPUs (maximum sample gap 2 s). Source revisions, artifacts and cleanup were verified; Slurm 45587 completed 0:0 on mia1-p01-g14 and released its allocation. This completes the 16-point representative batch. No accuracy eval; the historical comparison does not establish latency or full performance parity.

Keep draft. The completed representative batch passed 16/16 fixed-sequence points across five recipes on the previous fork pin c29ef7c, with 11,290/11,290 measured requests, required power and cleanup; six additional NVIDIA points used older pins. None of those measurements qualify the new upstream/AtoMesh path. The B200 Qwen TRT TP8 c4 historical throughput delta remains -3.58% versus the June 24 API baseline. Broader recipe and pool coverage, accuracy evaluation, fresh performance comparison and cancellation under load remain incomplete. Future GPU checks stay bounded to one task-owned allocation at a time. This replaces closed #3351.

中文

将全部 50 个活跃的 Slurm 单节点定长配置切换为原生 SRT 配方：35 个 SGLang、8 个 TRT-LLM、7 个 ATOM，覆盖全部 407 个现有测试点。包含 H100/H200/B200/B300 和 MI300X/MI325X/MI355X 上的 DeepSeek-R1 与 Qwen3.5。模型、镜像、并发范围、拓扑和服务设置保持一致。每个迁移条目均存在于当前 main 的活跃 master 配置中；已包含 #3349 的弃用清理。两个仅使用 Docker 的 RTX PRO 6000 定长配置现已退役至现有 NVIDIA 归档，原始设置与脚本均予保留。

十一个 Slurm 池的定长作业现在必须提供 SRT 配方，不再回退到旧版单节点 Bash 实现；50 个已替代脚本已删除。活跃的单节点定长覆盖仅使用 SRT：两个 RTX Docker 配置及其脚本已归档，不再使用的 runner 映射、启动器和运行时设置已移除，不保留活跃的定长 Bash/Docker 回退。Changelog 规划保留引用归档配置的历史条目，但不生成对应作业；未知键名及 append-only 选择已退役配置仍会报错。AgentX、现有多节点路径及显式选择的 SPEED-Bench 采集器继续使用当前执行路径。AgentX 迁移将单独提交 PR。

配方负责服务与工作负载设置；集群配置负责调度、挂载、模型位置及镜像缓存。连接器精确选择一个原生变体，与矩阵核对后再注入运行时参数。提交、分配状态、取消、结果与功耗产物及原生来源信息均有显式检查。原生 eval 调度复用现有 InferenceX 实现，并将评测或产物准备失败传递给作业。

随并发变化的设置使用 SRT 上游已有的 zip_override_* 功能。同一组内的列表按索引配对并覆盖 base，不会生成笛卡尔积。例如，DeepSeek-R1 H200 TRT MTP 在 c4–32 使用 MTP3、不启用 DP attention，在 c64–256 改用 MTP1 和 DP attention，并相应调整 batch/token 预算。Qwen3.5 H200 SGLang 则随并发调整 CUDA graph batch size。InferenceX 根据作业的并发与拓扑选择对应索引的变体。

SRT 子模块现使用 NVIDIA 上游 main 3cbc5dd256af2bfd2fed09b724628c3f5456c85f，包含已合入的 AMD、MoRI、ATOM/AToMesh 及原生 TRT-LLM 服务模型名称支持。本迁移不再依赖分叉 #27。七个 ATOM 配方选择原生 frontend.type: atomesh，保留一个聚合 worker。历史 worker 镜像保持不变；仅路由器通过 frontend.container_image 使用官方 rocm/atom-dev:nightly_202609161445，并固定摘要 sha256:f00a9dc588a380038fdad54cd57a956043cb316ddda54ebb1b62a24f8caa343b。已确认缓存的五月 worker 镜像不包含 AToMesh，并验证官方路由器镜像的 tag、摘要及缓存中的可执行文件。TRT 配方使用原生 engine.served_model_name 转发，移除现已重复的 extra_args 参数。未调整 worker 镜像、模型参数、并发范围、拓扑或基准设置，也未新增 Bash 分支。原有 TileRT 旧版独立检出路径不属于本次迁移。

上游切换在 66781f6c0 上的验证：152 项定向 CPU 测试通过，其中 122 项为 InferenceX 集成检查，30 项为上游 ATOM/TRT/SGLang 检查。两个使用模拟 launcher 的测试因 macOS 沙箱禁止访问 /dev/stderr 而在沙箱外重跑，均通过。全部 50 个配置 / 407 个测试点通过吞吐及 eval-only 模式的原生 schema、拓扑、单点提交与 worker 命令生成检查，共 814 项检查，涵盖 73 个 AToMesh 变体以及仅传入一次原生服务模型名称参数的 63 个 TRT 变体。配方前后对比确认仅做移植适配。Changelog 矩阵生成及历史字节保留检查通过。此上游固定版本和 AToMesh 请求路径的新实机验证仍待完成：检查时 MI355X 十个计算节点全部占用，因此未提交新排队作业。下方实机测量仅证明其明确记录的较早版本，不能作为本次切换的验证。

验证：

• 8f3f0d551 退役剩余 Docker 配置并修复归档 Changelog 选择后的本地 CPU 测试：1,991 项通过、1 项跳过，另有 187 个子测试通过。Ruff、Bash 语法、changelog 校验及历史字节保留检查通过。

• 全部 407 个测试点通过原生 schema、拓扑、提交变体选择和展开后的旧版/原生命令对比；407 个 eval-only 配置生成检查也通过。这些检查不代表硬件验收完成。

• Qwen3.5 H200 c4，运行时 a88a27a7cdf04d9f7b9e87492e6abdeadd38fc2c：40/40 请求完成，输出吞吐 432.42 tokens/s，相对匹配的已发布基线约 -0.2%。

• Qwen3.5 H200 c16，相同运行时：160/160 请求完成，输出吞吐 1002.34 tokens/s，相对匹配的已发布基线约 +0.6%。两次运行均使用 SGLang v0.5.14-cu130、FP8、TP8/EP8、8k1k，八张 GPU 的强制功耗校验通过。

• Qwen3.5 H200 c64，运行时 2763e8d002efd1786d7c52e30c73410c0d0803c2：640/640 请求完成，输出吞吐 1741.50 tokens/s，相对匹配的已发布基线约 -1.5%。八张 GPU 的强制功耗校验通过（最大采样间隔 1.183 s）；Slurm 以 0:0 完成，结果汇总成功。

• DeepSeek-R1 H200 c4，运行时 348f773eb6fba29cdb4216d41240cff122a4522e：40/40 请求完成，输出吞吐 344.78 tokens/s，相对匹配的已发布基线约 -0.04%；强制功耗校验通过。

• DeepSeek-R1 TRT MTP H200 c4，运行时 b4724b08bb1488105ff6fba32a46ad37470d6417、SRT 2ac4eb1367dd2a78f597a72ca91afe4211d76b38：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、MTP3、8k1k、并发 4。原生 E2E 通过：40/40 个计量请求完成，输出吞吐 416.27 tokens/s，服务模型名称正确，八张 GPU 的功耗校验、结果汇总和资源清理均通过。这验证了现有配方 extra_args 路径，无需 [NVIDIA] Add GB200 DSR1 FP4 TRT #26。随后串行运行的旧版对照使用 main 7e257acec661842d9b7f5651f92b4c29ea162fd7，40/40 请求完成，输出吞吐 414.86 tokens/s；输入和输出 token 总数完全一致，原生吞吐高 0.34%。对照工作流因未找到结果文件而失败，但原始 JSON 随后包含在上传的功耗审计产物中。按已核实的集群 UTC 时区离线校验未经修改的遥测，八张 GPU 均通过。恢复产物不会改变工作流失败的结论。这只是不同 H200 节点上的一个测试点，不代表完整性能或准确性验收。两个资源分配均已释放。

• DeepSeek-R1 TRT MTP H200 c64，运行时 b4724b08、SRT 2ac4eb1：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、DP attention、MTP1、8k1k，640/640 个计量请求完成，输出吞吐 1622.04 tokens/s。E2E、八张 GPU 的功耗校验、结果收集及清理均通过。验证了高并发配方变体；尚未完成新的 c64 对照运行。

• Qwen3.5 H200 eval 集成检查，运行时 a67435c0f8d2cb2e2d0c61c829ca4197a1368e64：SGLang v0.5.14-cu130、FP8、TP8/EP8、并发 4，GSM8K 前 16 个样本。全部请求完成，strict/flexible exact match 均为 0.9375（15/16）。工作流分数门槛未通过，Qwen 要求 0.94。原生 eval 退出记录为 0，元数据及产物已准备，Slurm 以 0:0 完成。这只验证实际执行和产物路径，不代表准确性验收；保留小样本失败记录，阈值未变。

• MI300X Qwen3.5 c4 启动检查，运行时 571fa51b3、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、8k1k。集群前置命令尾部的换行与原生 && 拼接冲突，资源分配在服务启动前失败；计量请求数为零。资源已释放。0bd3bf5f7 修复了两个受影响的 AMD 前置命令，并在 accounting 不可用的集群上通过控制器核验终态；原生 shell 实际执行和失败状态用例已在本地通过。基于 8e279188e 的实机重试已到达服务就绪阶段，但 pip --user 与 ROCm 镜像的虚拟环境不兼容，因而在发送基准请求前失败。c430090a8 移除了强制用户目录安装参数。基于 6f60a8bf2、SRT c29ef7c 的 AMD c4 实机重试 E2E 通过：40/40 个计量请求完成，输出吞吐 238.58 tokens/s；匹配的 2026-05-17 InferenceX API 基线为 241.17 tokens/s，变化 -1.07%。平均 TTFT 为 1.0813 s，基线为 0.9888 s（+9.35%）；平均 TPOT 为 15.227 ms，基线为 15.143 ms（+0.56%）。八张 GPU 的强制功耗校验通过，最大采样间隔 2 s；产物已收集，Slurm 以 0:0 完成并释放资源。历史基线比较不能证明延迟完全一致，也不代表准确性验收。

• DeepSeek-R1 B200 SGLang MTP c1，运行时 c430090a8、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP1、8k1k。10/10 请求完成，输出吞吐 297.44 tokens/s；匹配的 InferenceX API 基线日期为 2026-08-06，吞吐为 296.69 tokens/s，变化 +0.25%；平均 TTFT -3.59%，平均 TPOT -0.02%。Slurm 以 0:0 完成并释放资源。节点独占分配向客户端暴露了八张 GPU，而预期为四张，因此工作流未通过必需的功耗校验；这是已测吞吐证据，不是通过的 smoke。6f60a8bf2 在保留节点独占的同时，将原生服务和客户端步骤限制为已验证的推理 GPU 数量。基于 6f60a8bf2 的验证运行 E2E 通过：10/10 请求完成，输出吞吐 297.74 tokens/s（相同 API 基线相比 +0.36%），平均 TTFT -3.79%，平均 TPOT -0.11%。功耗校验恰好观测到四张 GPU，最大采样间隔 1.071 s；产物已收集，Slurm 以 0:0 完成并释放资源。五个配方的 16 个代表点每次仅运行一个分配。H200 c4 现已通过，五个配方均已完成首轮低并发验证。B200 Qwen TRT-MTP c4 运行已串行完成 TP2/TP4/TP8，强制功耗校验均通过。修复采样器后，MI355X ATOM c4 已通过必需的功耗校验和资源清理，详情见下文。基于 77961ed42 的 H200 c4 smoke 已通过必需功耗校验，本次未运行 eval。MI300X c16 现已通过，详情见下文。H200 c64 也已通过，包括 DP attention/MTP1 配置，详情见下文。B200 c32 也已通过，功耗校验恰好覆盖四张 GPU，详情见下文。H200 c256 和 MI300X c64 均已通过，这两个配方选定的低、中、高并发测试点已全部完成。B200 SGLang c256 现也已通过，其选定的 c1/c32/c256 均已完成。B200 Qwen TRT c128 现也已通过。MI355X ATOM c32 和 c256 均已通过，选定批次已全部完成，达到 16/16。最后一个 MI355X ATOM c256 测试点已通过。五个配方的全部 16 个选定测试点均已完成：11,290/11,290 个计量请求完成，必需功耗校验及清理通过，本任务的全部资源分配均已释放。整个批次始终最多仅有一个 GPU 资源分配。这些未运行 eval 的 smoke 不代表全部 50 个已转换配置通过验收，也不能证明完整性能或准确性一致。Qwen TP8 c4 相对历史基线的差异仍未查明：当前 CLI 会展开三个 TP 变体，且因它们共享实验名称而拒绝精确名称选择，因此未为单个复测点重跑三个变体。

• Qwen3.5 B200 TRT-MTP c4、TP2，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP2/EP1、不启用 DP attention、8k1k。基准作业通过，40/40 个计量请求完成，输出吞吐 770.39 tokens/s；匹配的 2026-06-24 API 基线为 769.23 tokens/s，变化 +0.15%。平均 TTFT +2.80%，平均 TPOT -0.68%。功耗校验恰好观测到两张 GPU，最大采样间隔 1.040 s；Slurm 16906 以 0:0 完成并释放资源。TP4 和 TP8 也已通过，结果如下；三个变体的完整工作流已成功结束。本次未运行准确性评测。

• Qwen3.5 B200 TRT-MTP c4、TP8，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP8/EP8、不启用 DP attention、8k1k。基准作业通过，40/40 个计量请求完成，输出吞吐 1042.85 tokens/s；匹配的 2026-06-24 API 基线为 1081.62 tokens/s，变化 -3.58%。平均 TTFT +3.55%，平均 TPOT +2.17%。八张 GPU 的强制功耗校验通过，最大采样间隔 1.001 s；Slurm 16907 以 0:0 完成，并在 TP4 启动前释放资源。这一下降已列为后续新鲜对照的关注项；单次历史基线 smoke 不能确定原因或证明性能一致。

• Qwen3.5 B200 TRT-MTP c4、TP4，运行时 6f60a8bf2、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP4/EP4、不启用 DP attention、8k1k。40/40 请求通过，输出吞吐 942.20 tokens/s；匹配的 2026-06-24 API 基线为 917.70 tokens/s，变化 +2.67%；平均 TTFT -6.64%，平均 TPOT -1.68%。四张 GPU 的强制功耗校验通过，最大采样间隔 1.001 s；Slurm 16908 以 0:0 完成并释放资源。三个 c4 TP 变体及结果收集均已成功结束。

• MI355X DSR1 ATOM-MTP c4 启动失败，运行时 b3a5a7f39、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、8k1k。启用 PYTHONPYCACHEPREFIX 且从 / 启动时，Python 3.12 在导入 PyTorch 动态生成模块的 importlib.cache_from_source 路径抛出 IndexError。基准请求数为零，无吞吐或准确性结果。 Slurm 45548 以 1:0 失败并释放资源。已在本地复现相同导入路径错误。161aa243a 通过原生 srun_options.container-workdir=/infmax-workspace 恢复旧版容器工作目录行为，使用现有仓库挂载并保留调用方覆盖。69 项定向 CPU 测试、原生命令生成、Ruff 和 changelog 校验均通过。基于 838607d8a 的重试完成了 40/40 个请求，确认启动问题已修复；另一个功耗校验失败记录如下。

• MI355X DSR1 ATOM-MTP c4 测量，运行时 838607d8a、SRT c29ef7c：FP4、TP8/EP1、不启用 DP attention、8k1k。40/40 请求完成，输出吞吐 611.90 tokens/s；匹配的 2026-05-20 API 基线为 605.22 tokens/s，变化 +1.10%；平均 TTFT -2.56%，平均 TPOT -0.35%。Slurm 45549 以 0:0 完成并释放资源。工作流未通过必需的功耗校验：八张 GPU 均有数据，但 CSV 样本在基准结束前 3.68–4.68 s 停止（benchmark_window_not_bracketed）。这是吞吐测量结果，不是通过的 smoke 或准确性验收。已复现现有 CSV 过滤器在 mawk 下即使调用 fflush() 仍缓冲管道输入的问题。dd955705c 将其替换为共享 Bash 逐行读取器，保留功耗校验和基准设置。237 项定向 CPU 测试、Linux 流式检查、Ruff、shell 语法及追加式 changelog 校验均通过。基于 dd955705c 的 c4 验证运行 E2E 通过：40/40 请求完成，输出吞吐 610.71 tokens/s（+0.91%），对比相同的 5 月 20 日 API 基线；平均 TTFT -0.36%，平均 TPOT -1.39%。八张 GPU 的必需功耗校验通过，最大采样间隔为 2 s。Slurm 45550 以 0:0 完成并释放资源。本次未运行准确性评测，该 MI355X 分配及后续 H200 c4 分配均已释放。

H200 精确选择暴露了 configs/runners.yaml 中过期的单数字条目：_7 无法匹配实际注册的 _07 runner。77961ed42 将两个 H200 标签清单与 14 个在线注册的两位数字名称对齐。8 项 runner 过滤 CPU 测试、精确单点生成及追加式 changelog 校验均通过，基准设置未变。

• H200 DSR1 TRT-MTP c4，此前分叉固定版本，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、不启用 DP attention、MTP3、8k1k。E2E 通过，40/40 请求完成，输出吞吐 413.62 tokens/s；匹配的 2026-05-18 API 基线为 413.01 tokens/s，变化 +0.15%；平均 TTFT -1.97%，平均 TPOT -0.36%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.164 s；源版本和产物均已核实。Slurm 88762 以 0:0 完成并释放资源，本次未运行准确性评测。

• MI300X Qwen3.5 SGLang c16，运行时 77961ed42、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、不启用 DP attention、无推测解码、8k1k。E2E 通过，160/160 个计量请求完成，输出吞吐 650.52 tokens/s；匹配的 2026-05-17 API 基线为 652.14 tokens/s，变化 -0.25%；平均 TTFT +0.89%，平均 TPOT +0.21%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本及产物已核实；原生终态校验与工作流清理均通过，另行检查实时队列确认 Slurm 分配 1159 已释放。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• H200 DSR1 TRT-MTP c64，此前分叉固定版本，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、启用 DP attention、MTP1、8k1k。E2E 通过，640/640 个计量请求完成，输出吞吐 1629.27 tokens/s；匹配的 2026-05-18 API 基线为 1628.59 tokens/s，变化 +0.04%；平均 TTFT -3.64%，平均 TPOT +0.29%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.154 s。实际生成的 TRT 配置确认了随并发变化的设置。源版本、产物与清理均已核实；Slurm 88777 以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 DSR1 SGLang-MTP c32，运行时 77961ed42、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP1、不启用 DP attention、8k1k。E2E 通过，320/320 个计量请求完成，输出吞吐 1699.06 tokens/s；匹配的 2026-08-06 API 基线为 1717.36 tokens/s，变化 -1.07%；平均 TTFT -2.17%，平均 TPOT +1.20%。必需功耗校验恰好覆盖四张 GPU，最大采样间隔 1.072 s；原生提交使用 srun_options.gpus-per-node=4。源版本、产物与清理均已核实；Slurm 16910 以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• H200 DSR1 TRT-MTP c256，运行时 77961ed42、SRT c29ef7c：FP8、TRT-LLM 1.3.0rc14、TP8/EP8、启用 DP attention、MTP1、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 2639.48 tokens/s；匹配的 2026-05-18 API 基线为 2636.47 tokens/s，变化 +0.11%；平均 TTFT -1.69%，平均 TPOT -0.02%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.169 s；实际 TRT 设置包含 max_batch_size: 32。源版本、产物与清理均已核实；Slurm 88778 以 0:0 完成并释放资源。选定的 H200 c4/c64/c256 均已通过；本次未运行准确性评测，也不代表完整性能验收。

• MI300X Qwen3.5 SGLang c64，运行时 77961ed42、SRT c29ef7c：FP8、SGLang v0.5.12-rocm720-mi30x、TP8/EP1、不启用 DP attention、无推测解码、8k1k。E2E 通过，640/640 个计量请求完成，输出吞吐 1385.23 tokens/s；匹配的 2026-05-17 API 基线为 1387.16 tokens/s，变化 -0.14%；平均 TTFT +4.29%（2.0803 s 对比 1.9947 s），平均 TPOT -0.09%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 1160 以 0:0 完成并释放资源。选定的 MI300X c4/c16/c64 已全部通过。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 DSR1 SGLang-MTP c256，运行时 77961ed42、SRT c29ef7c：FP4、SGLang v0.5.16-cu130、TP4/EP4、启用 DP attention、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 3843.97 tokens/s；匹配的 2026-08-06 API 基线为 3904.54 tokens/s，变化 -1.55%；平均 TTFT +1.89%，平均 TPOT +1.68%。服务可见设备与必需功耗校验均恰好覆盖四张 GPU，最大采样间隔 1.003 s。源版本、产物与清理均已核实；Slurm 16911 以 0:0 完成并释放资源。选定的 B200 SGLang c1/c32/c256 已全部完成，覆盖 EP4/DP attention 配置切换。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• B200 Qwen3.5 TRT-MTP c128，运行时 77961ed42、SRT c29ef7c：FP4、TRT-LLM 1.3.0rc18、TP8/EP8、启用 DP attention、8k1k。E2E 通过，1280/1280 个计量请求完成，输出吞吐 8977.02 tokens/s；匹配的 2026-06-24 API 基线为 8801.95 tokens/s，变化 +1.99%；平均 TTFT -4.04%，平均 TPOT +2.34%。八张 GPU 的必需功耗校验通过，最大采样间隔 1.140 s。源版本、产物与清理均已核实；Slurm 16912 以 0:0 完成并释放资源。本次覆盖选定的 DP attention 配置切换，但不能解释 TP8 c4 相对历史基线的吞吐下降，也不代表准确性验收。

• MI355X DSR1 ATOM-MTP c32，运行时 77961ed42、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、不启用 DP attention、8k1k。E2E 通过，320/320 个计量请求完成，输出吞吐 2002.39 tokens/s；匹配的 2026-05-20 API 基线为 1978.91 tokens/s，变化 +1.19%；平均 TTFT +1.28%，平均 TPOT -1.14%。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 45584 在 mia1-p01-g14 上以 0:0 完成并释放资源。本次未运行准确性评测，历史基线比较不代表完整性能验收。

• MI355X DSR1 ATOM-MTP c256，运行时 77961ed42、SRT c29ef7c：FP4、ATOM rocm7.2.3_ubuntu24.04_py3.12_pytorch_release_2.10.0_atom20260511、TP8/EP1、不启用 DP attention、8k1k。E2E 通过，2560/2560 个计量请求完成，输出吞吐 3492.79 tokens/s；匹配的 2026-05-20 API 基线为 3471.59 tokens/s，变化 +0.61%；平均 TTFT +5.88%（2.8468 s），平均 TPOT -0.91%（69.409 ms）。八张 GPU 的必需功耗校验通过，最大采样间隔 2 s。源版本、产物与清理均已核实；Slurm 45587 在 mia1-p01-g14 上以 0:0 完成并释放资源。至此 16 个代表性测试点全部完成。本次未运行准确性评测，历史基线比较不能证明延迟或完整性能一致。

保持草稿。此前分叉固定版本 c29ef7c 上的代表性批次已通过五个配方的 16/16 个固定序列长度测试点，11,290/11,290 个计量请求完成，必需功耗及清理检查通过；另六个 NVIDIA 测试点使用更早版本。这些测量均不能验收新的上游/AtoMesh 路径。B200 Qwen TRT TP8 c4 相对 6 月 24 日 API 历史基线的吞吐差异仍为 -3.58%。更广的配方与集群池覆盖、准确性评测、新鲜性能对照以及负载下取消验证仍未完成。后续 GPU 检查继续限制为同一时间至多一个本任务分配。本 PR 替代已关闭的 #3351。

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Claude finished @cquil11's task in 9m 35s —— View job

---

Review of PR #3352

• Gather context (diff vs origin/main, changed files, repo guidance)
• Review connector / runtime changes (benchmark_lib.sh, infx/srt_slurm/single_node.py, slurm_utils.sh, launchers, workflow template)
• Spot-check recipes against the deleted Bash scripts (11 recipes across TRT/SGLang/ATOM on NVIDIA and AMD: settings, per-concurrency derived values, env vars, MTP chat template, EP flags, images)
• Check master-config / perf-changelog rules
• Post findings

Summary: No blocking issues found. One 🟡 warning posted inline: the native single-node path only reuses an existing .sqsh and never imports one, so uncached images (every image bump, and always on b200-nb) are pulled by Pyxis per srun step inside the GPU allocation, and the reproducibility guarantee of the enroot import docker:// pattern is lost (runners/slurm_utils.sh:161-166). Everything else checked out:

• All 11 spot-checked recipe/script pairs match, including hand-rederived per-concurrency max_batch_size/max_num_tokens, cuda-graph-max-bs, scheduler-recv-interval, context lengths, exported env vars, and gpus/TP/EP per zip group. The MI300X HSA_NO_SCRATCH_RECLAIM firmware probe moved correctly into the mi300x-amd cluster profile preamble. Eval-only context (MAX_MODEL_LEN = ISL+OSL+256 = 9472) equals what setup_eval_context produced in the legacy scripts.
• Master configs: only srt-recipe additions and flow→block reformatting; no concurrency range, image, model, or topology changed. RTX PRO 6000 entries archived verbatim. All images are public registries (Docker Hub digest pin for the AToMesh router included).
• perf-changelog.yaml: entries appended at the tail, none append-only, master-config edits are documented.
• Line-count report for infx/matrix/generate.py posted inline (+2).

Limitations: the utils/srt-slurm submodule is not checked out in this environment and network fetches were not permitted, so I could not verify the upstream 3cbc5dd contract (atomesh frontend, engine.served_model_name, post_eval.passthrough_env, srun_options leaf overrides) directly; I relied on the repo's tests and the PR's E2E evidence, which as the description notes does not yet cover this pin on GPUs. Sandbox restrictions also prevented running pytest locally.
· Branch

[claude]

🟡 WARNING: The native single-node path never populates the squash cache. It only reuses an existing valid SRT_SQUASH_FILE; otherwise it hands the raw docker://-style reference to Pyxis, and launch_b200-nb.sh unsets SRT_SQUASH_FILE so it always does this.

Why it matters: Unlike the retained enroot import -o $SQUASH_FILE docker://$IMAGE blocks in every legacy launcher, nothing on this path ever writes the .sqsh, so a fresh image tag (i.e. every image bump PR) is pulled from the registry by Pyxis independently for each srun step in the allocation (worker, frontend, benchmark client, eval) on every run, inside the GPU allocation. The unsquashfs -s probe also runs on the runner/login host, so on pools where the squash dir is node-local (e.g. B200_SQUASH_DIR on Nscale) it silently falls back to the pull path even when a cache exists on the compute node. This also removes the reproducibility guarantee the enroot-import pattern gives (the exact same imported image used across steps and runs).

Fix: Either import once per job before submission, reusing the existing per-pool import helpers, e.g.

if [[ -n "${SRT_SQUASH_FILE:-}" ]] && ! unsquashfs -s "$SRT_SQUASH_FILE" >/dev/null 2>&1; then
    srun -N 1 -A "$SLURM_ACCOUNT" -p "$SLURM_PARTITION" --time="$ENROOT_IMPORT_TIME_LIMIT" \
        bash -c "enroot import -o '$SRT_SQUASH_FILE' 'docker://${IMAGE/nvcr.io\//nvcr.io#}'"
fi

or document here (and in the launchers that unset SRT_SQUASH_FILE) why per-step Pyxis pulls are acceptable for these pools.

[cquil11]

this is gonna be deprecated in later cleanup anyway so I don't think this matters?

[claude]

I reviewed this PR and didn't find any bugs. Because it's a large (151-file) migration of single-node benchmark execution to a new YAML-recipe-driven SRT-Slurm path, touching CI workflows, benchmark scripts, runner launchers, master configs, and a submodule pin, a human look would still be worthwhile.

What was reviewed: the new srt_fixed_sequence.sh/srt_eval.sh launcher scripts for AGENTS.md bash conventions (env-var validation via check_env_vars, no silent fallback defaults); a sample recipe YAML (dsr1/atom/mi355x-fp4-mtp/8k1k.yaml) for structure and the synthetic-acceptance/served-model-name handling described in the PR; and the overall diff shape (deleted per-model bash scripts, new recipe tree, connector module, runner/launcher and master-config updates).

Extended reasoning...

This PR is a large, multi-system migration (151 files, ~6.6k/~5.1k line diff) spanning CI workflow config, dozens of new/removed benchmark scripts and YAML recipes, runner launcher scripts, master configs, a connector module, and an upstream submodule pin bump — none of which is auth/crypto-sensitive but all of which affects benchmark correctness and CI behavior broadly. The bug hunter ran to a dry streak with no findings, and spot checks of the new launcher script and a sample recipe followed the repo's documented bash/config conventions. There is also an inline reviewer comment thread from a third party (cquil11) on one recipe file whose resolution status cannot be independently confirmed from available metadata, and no prior review from this session exists on this PR, so approval is not appropriate given the size and open thread.

[cquil11]

lgtm