feat(srt): check srt-slurm recipes against master images before sweep dispatch / feat(srt)：在派发 sweep 前检查 srt-slurm 配方与主配置镜像是否一致

[chunfangamd]

Description

Follow-up to #3567; the two PRs belong together. #3567 aligns the two srt-slurm recipes whose container had drifted from the master-config image. This PR adds the check that would have caught the drift: each sweep now binds every planned srt-slurm point to the recipe variant it would launch, and stops before dispatching any job if a point doesn't bind.

Problem. An srt-slurm point names its container in two places: the master config's image and the recipe's model.container. An image bump has to edit both, as #3446 did, but nothing compares them before benchmark jobs start.

• Single-node points: infx/srt_slurm/single_node.py rejects a mismatch (Single-node SRT image: recipe/matrix ...) only after the benchmark job has started on its cluster runner, prepared an srt-slurm checkout and installed srtctl. [AMD] Update GLM-5.2 MI355X image to 20260924 daily / 更新 GLM-5.2 MI355X 镜像至 20260924 daily #3446's first sweep failed this way. So would every point of the two keys fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 fixes, because feat(srt): run AgentX on native srt-slurm #3428 ported their recipes from the legacy scripts as they were before the image bumps in config(dsv41flash): re-sweep MI355X on the first ROCm 10.0 nightly carrying vllm#58510 / 在首个包含 vllm#58510 的 ROCm 10.0 nightly 上重新扫描 MI355X #3420 and [Klaud Cold] Update dsv4-fp4-b200-sglang-agentic-hicache-mtp SGLang image to v0.5.20-cu130 / 将 dsv4-fp4-b200-sglang-agentic-hicache-mtp 的 SGLang 镜像更新至 v0.5.20-cu130 #3334.
• Multi-node points: nothing checks them. srtctl uses a recipe container that isn't a configured alias as a literal image, so a stale recipe benchmarks a different image from the one the master config records.

#3567 first added a pytest scan for this. Review found four problems with it:

• It couldn't gate run-sweep, which is a separate workflow.
• It compared image sets per recipe instead of the variant each point selects, so it missed an image swap inside the shared Qwen3.5 H200 recipe.
• It skipped EVAL_CONFIG_FILE.
• It pinned checked-in config, against the AGENTS.md test rules.

#3567 dropped the scan, and this PR replaces it.

Changes

1. Validator. infx/srt_slurm/preflight.py (python -m infx.srt_slurm.preflight) reads the matrix on stdin. If every srt-slurm point binds, it writes the matrix unchanged to stdout; otherwise it prints each problem once with the points it affects and exits 1.
   • Single-node: it builds the environment that benchmark-tmpl.yml exports for the point and calls the runtime's own select_recipe. Exactly one variant must match on engine, model, image, precision, parallelism, GPU count, speculative decoding, concurrency and KV offloading. The check runs per point, so a recipe shared by keys with different images is checked against each key's image.
   • Multi-node: it expands CONFIG_FILE and EVAL_CONFIG_FILE with srtctl's own override expansion. Each variant's model.container must be the point's image (in either registry spelling) or a container alias that configs/runners.yaml defines for the runner's cluster. A declared identity.container.image must name the same image. TileRT points are skipped because they pair their own containers.
2. Workflows.
   • run-sweep.yml: setup initializes the srt-slurm submodule and runs the validator after benchmark_schema --plan, before ci_priority. If the validator fails, setup fails, so canary-select and the benchmark jobs, which all require a successful setup, never start.
   • e2e-tests.yml (which trusted-external-sweep.yml and claude.yml also dispatch) and profile.yml: the trusted tooling's validator checks the measured tree's recipes, runner config and srt-slurm submodule. The step is skipped when either tree predates the module, so measuring an older revision is unaffected.
3. Tests. infx/tests/srt_slurm/test_preflight.py uses small temporary recipes and runner inventories, per the AGENTS.md test rules. It covers:
   • a one-sided image update through the CLI;
   • a variant swap in a shared recipe;
   • an EVAL_CONFIG_FILE mismatch;
   • configured and misspelled aliases, and both registry spellings;
   • runner labels, pools and bare cluster ids;
   • TileRT points and a missing recipe.

A sweep that selects the B200 key without #3567's fix stops at setup with:

srt-slurm recipe preflight found 1 problem(s) affecting 13 point(s):
  srt-recipe=benchmarks/single_node/srt-slurm-recipes/dsv4/sglang/b200-fp4-mtp/agentic.yaml: Expected exactly one matching single-node SRT recipe; override_dep8_hicache_c128: Single-node SRT image: recipe/matrix 'lmsysorg/sglang:v0.5.19-cu130' != 'lmsysorg/sglang:v0.5.20-cu130'; ...
    - dsv4_tp8_conc1_kvnone_spec-draft_model on cluster:b200-nscale
    ...
    - dsv4_tp8_conc160_kvdram-hicache_spec-draft_model on cluster:b200-nscale | eval-only

Validation

• test_preflight.py: 14 passed. Each of eight deliberate regressions in the validator fails at least one test:
  • skipping EVAL_CONFIG_FILE, single-node points, or the alias lookup for bare cluster ids;
  • ignoring identity.container.image, or rejecting the pyxis spelling;
  • checking TileRT points, or accepting any container;
  • dropping the fallback for an override-format recipe without overrides.
• ci.yml commands, run locally: Lint is clean, and Tests has 2,349 passed. Two test_slurm_cli cases fail only on the test host because it has a real sacct; they fail the same way on clean main.
• Full generated matrix (1,712 points, about 10 s): 3 problems affecting 35 points, with no false positives.
  • dsv4-fp4-b200-sglang-agentic-hicache-mtp (12 points) and dsv41flash-fp4-mi355x-vllm-agentic-dspark (16 points) are fixed by fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567.
  • glm5.2-fp8-mi325x-sglang-agentic-mtp (7 points) still drifts on main.
• run-sweep setup replayed in a throwaway worktree with the workflow's exact commands under bash -e:
  • With a changelog entry for the B200 key and no recipe fix, it exits 1 at the validator, before ci_priority.
  • With fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's recipe, the matrix passes through byte-identical.

Notes for reviewers

• Please merge fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 first. Until it merges, a sweep that selects either of its keys fails at setup; it would fail at runtime anyway.
• glm5.2-fp8-mi325x-sglang-agentic-mtp still drifts on main, so a sweep that selects it fails at setup until its recipe is fixed. Only the points a sweep plans are checked, so drift in other keys doesn't block unrelated PRs.
• single_node_environment mirrors how benchmark-tmpl.yml exports matrix fields to the job. A change to that mapping needs the same change here.
• run-sweep only triggers on PRs that edit perf-changelog.yaml, so this PR's own checks don't run the new setup step. The replay above stands in for that.
• There is no perf-changelog.yaml entry, because no benchmark config or recipe changes.
• All changed files are owned by @SemiAnalysisAI/core.

AI model disclosure

• Model/version: Claude Opus 5.5 (the identifier exposed by the Cursor agent runtime). No delegated agents were used.
• Role: designed and wrote the validator, workflow steps and tests to address the review findings on fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567's earlier static test, ran the local validation, and drafted this description. @chunfangamd reviewed the changes and chose the design.

Related Issue

No issue. Follow-up to #3567. Related: #3428, #3446.

Type of Change

• Bug fix
• New feature
• Configuration change
• Documentation update
• Other (please describe)

Checklist

• I have completed the AI model disclosure and kept it current
• I have tested my changes locally
• I have updated documentation if necessary
• For every change that can affect benchmark performance and every recipe addition or modification, I have appended a new entry to the physical end of inferencex-e2e/perf-changelog.yaml and have not edited historical entries
• Before merging via reuse, an authorized maintainer (OWNER/MEMBER/COLLABORATOR) has commented /use <run_id> (or the legacy /reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.

中文

改动说明

本 PR 是 #3567 的后续，两者是一个整体。#3567 修正了两个 container 与主配置 image 不一致的 srt-slurm 配方；本 PR 加上本可以发现这种不一致的检查：每次 sweep 都会把每个计划中的 srt-slurm 点绑定到它实际会启动的配方 variant，任何一个点绑定失败，就在派发任何任务之前停止。

问题： srt-slurm 点在两个地方声明 container：主配置的 image 和配方的 model.container。升级镜像必须同时修改两处（如 #3446），但在 benchmark 任务开始之前，没有任何检查比较这两处。

• 单节点点： infx/srt_slurm/single_node.py 要等 benchmark 任务已在集群 runner 上启动、准备好 srt-slurm checkout 并安装 srtctl 之后，才会拒绝不一致的点（Single-node SRT image: recipe/matrix ...）。[AMD] Update GLM-5.2 MI355X image to 20260924 daily / 更新 GLM-5.2 MI355X 镜像至 20260924 daily #3446 的第一次 sweep 就是这样失败的。fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 修复的两个 key 的每个点也会这样失败，因为 feat(srt): run AgentX on native srt-slurm #3428 移植这些配方时，依据的是 config(dsv41flash): re-sweep MI355X on the first ROCm 10.0 nightly carrying vllm#58510 / 在首个包含 vllm#58510 的 ROCm 10.0 nightly 上重新扫描 MI355X #3420 和 [Klaud Cold] Update dsv4-fp4-b200-sglang-agentic-hicache-mtp SGLang image to v0.5.20-cu130 / 将 dsv4-fp4-b200-sglang-agentic-hicache-mtp 的 SGLang 镜像更新至 v0.5.20-cu130 #3334 升级镜像之前的旧脚本。
• 多节点点： 完全没有检查。srtctl 会把不是已配置 alias 的配方 container 当作字面镜像使用，所以过期的配方会测试一个与主配置记录不同的镜像。

#3567 最初为此加了一个 pytest 扫描。Review 指出它有四个问题：

• 它无法拦住独立的 run-sweep workflow。
• 它按配方比较镜像集合，而不是每个点实际选中的 variant，因此漏掉了共享的 Qwen3.5 H200 配方内部的镜像交换。
• 它没有检查 EVAL_CONFIG_FILE。
• 它在测试里固定了 checked-in 配置，违反 AGENTS.md 的测试规则。

#3567 已移除该扫描，由本 PR 取代。

改动：

1. Validator： infx/srt_slurm/preflight.py（python -m infx.srt_slurm.preflight）从 stdin 读入 matrix。所有 srt-slurm 点都能绑定时，原样输出到 stdout；否则每个问题只打印一次并列出受影响的点，然后以退出码 1 结束。
   • 单节点： 按 benchmark-tmpl.yml 为该点导出的环境变量，调用运行时自己的 select_recipe。必须在引擎、模型、镜像、精度、并行配置、GPU 数、投机解码、并发和 KV offloading 上恰好匹配一个 variant。由于逐点检查，被多个不同镜像的 key 共享的配方会分别对照每个 key 的镜像。
   • 多节点： 用 srtctl 自己的 override 展开逻辑展开 CONFIG_FILE 和 EVAL_CONFIG_FILE。每个 variant 的 model.container 必须是该点的镜像（两种 registry 写法均可），或 configs/runners.yaml 为该 runner 所在 cluster 配置的 container alias。声明了 identity.container.image 时，它必须是同一个镜像。TileRT 点使用自己配对的 container，因此跳过。
2. Workflow：
   • run-sweep.yml：setup 先初始化 srt-slurm submodule，在 benchmark_schema --plan 之后、ci_priority 之前运行 validator。validator 失败时 setup 失败，canary-select 和所有要求 setup 成功的 benchmark 任务都不会启动。
   • e2e-tests.yml（trusted-external-sweep.yml 和 claude.yml 也通过它派发）和 profile.yml：用可信工具树里的 validator 检查被测代码树的配方、runner 配置和 srt-slurm submodule。任一棵树还没有这个模块时跳过该步骤，因此测量旧版本不受影响。
3. 测试： infx/tests/srt_slurm/test_preflight.py 按 AGENTS.md 的测试规则，只使用临时目录中的小型配方和 runner 配置，覆盖：
   • 通过 CLI 的单边镜像更新；
   • 共享配方中的 variant 交换；
   • EVAL_CONFIG_FILE 不一致；
   • 已配置和拼错的 alias，以及两种 registry 写法；
   • runner label、pool 和裸 cluster id；
   • TileRT 点和缺失的配方。

上方的示例输出，是一个选中 B200 key、但没有 #3567 修复的 sweep 在 setup 停止时打印的内容。

验证：

• test_preflight.py：14 个通过。对 validator 故意做的 8 种破坏，每一种都会让至少一个测试失败：
  • 跳过 EVAL_CONFIG_FILE、单节点点，或裸 cluster id 的 alias 查找；
  • 忽略 identity.container.image，或不接受 pyxis 写法；
  • 检查 TileRT 点，或接受任意 container；
  • 去掉对没有 override 的 override 格式配方的兜底。
• 在本地运行 ci.yml 的命令：Lint 干净，Tests 2,349 个通过。两个 test_slurm_cli 用例只在测试机上失败，因为它装有真实的 sacct；在干净的 main 上同样失败。
• 完整生成的 matrix（1,712 个点，约 10 秒）：3 个问题影响 35 个点，没有误报。
  • dsv4-fp4-b200-sglang-agentic-hicache-mtp（12 个点）和 dsv41flash-fp4-mi355x-vllm-agentic-dspark（16 个点）由 fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 修复。
  • glm5.2-fp8-mi325x-sglang-agentic-mtp（7 个点）在 main 上仍不一致。
• 在一次性 worktree 中用 workflow 的原样命令和 bash -e 重放 run-sweep 的 setup：
  • changelog 选中 B200 key 而配方未修复时，在 validator 处以退出码 1 退出，没有走到 ci_priority。
  • 带上 fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 的配方时，matrix 逐字节不变地通过。

审阅注意事项：

• 请先合入 fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567。在它合入之前，选中其两个 key 的 sweep 会在 setup 失败；这些 sweep 在运行时本来也会失败。
• glm5.2-fp8-mi325x-sglang-agentic-mtp 在 main 上仍不一致，选中它的 sweep 在其配方修复之前会在 setup 失败。validator 只检查 sweep 计划中的点，其他 key 的不一致不会挡住无关的 PR。
• single_node_environment 复刻了 benchmark-tmpl.yml 把 matrix 字段导出给任务的方式；那边的映射改动时，这里也要同步修改。
• run-sweep 只在修改 perf-changelog.yaml 的 PR 上触发，因此本 PR 自己的检查不会运行新的 setup 步骤，由上面的重放代替。
• 没有 perf-changelog.yaml 记录，因为本 PR 不改任何 benchmark 配置或配方。
• 所有改动文件都归 @SemiAnalysisAI/core。

AI 模型使用说明

• 模型/版本：Claude Opus 5.5（Cursor agent 运行环境提供的标识），未使用其他委派 agent。
• 工作内容：针对 fix(srt): align B200 and MI355X recipe images with master configs / fix(srt)：对齐 B200 与 MI355X 配方镜像与主配置 #3567 早期静态测试的 review 意见，设计并编写 validator、workflow 步骤和测试，运行本地验证，并起草本说明；@chunfangamd 审阅了全部改动并确定了设计。

关联 issue

无。#3567 的后续；相关 PR：#3428、#3446。

改动类型

新功能。

[claude]

🔴 This runs the "trusted" preflight check with srtctl imported from the PR's own utils/srt-slurm submodule checkout, letting a malicious PR execute code inside the trusted-tooling process instead of only the sandboxed measured tree. check_multi_node -> selected_recipes does from srtctl.core.config import generate_override_configs, and PYTHONPATH is $PRIORITY_ROOT:$MEASURED_ROOT/utils/srt-slurm/src (e2e-tests.yml:336) while only $MEASURED_ROOT ever gets git submodule update --init utils/srt-slurm (line 335) — $PRIORITY_ROOT never does, so srtctl can only resolve from the PR-controlled submodule commit. -P doesn't block PYTHONPATH imports. Fix: give PRIORITY_ROOT its own trusted srt-slurm checkout (or vendor/pin the srtctl it imports) so the trusted interpreter never imports a package whose source lives only in the untrusted MEASURED_ROOT tree; same pattern in profile.yml:111-116.

Why this was flagged

The whole point of checking out .ci-priority/PRIORITY_ROOT separately from MEASURED_ROOT in this job is to run decision logic with code the PR cannot modify. This diff defeats that: infx.srt_slurm.preflight (loaded from PRIORITY_ROOT) calls infx.srt_slurm.synthetic_acceptance.selected_recipes, which lazily imports srtctl.core.config.generate_override_configs; that package is only ever present via $MEASURED_ROOT/utils/srt-slurm/src (e2e-tests.yml:335-336), which is the PR's own branch/submodule pointer. A PR that points its utils/srt-slurm submodule gitlink at a malicious commit gets that code executed inside the trusted PRIORITY_ROOT python process when get-jobs runs (same for profile.yml:111-116). python -P only disables automatic sys.path prepending, not PYTHONPATH, so this import is not blocked. The output of this step feeds CONFIG_JSON/job outputs that downstream benchmark-dispatch jobs (which do use secrets.INFERENCEX_OFFICIAL_RO_HF_TOKEN, secrets.MODAL_TOKEN_ID/SECRET) consume, so compromising this step can also poison what those privileged jobs run.

Verification: e2e-tests.yml:335 runs git -C "$MEASURED_ROOT" submodule update --init utils/srt-slurm (PR-controlled submodule), and :336 sets PYTHONPATH="$PRIORITY_ROOT:$MEASURED_ROOT/utils/srt-slurm/src", so srtctl resolves only from the measured tree.