[SRT runtime] repair setup and required recipe power / 修复安装并按配方校验功耗

docs/configuration-procedures.md

@@ -35,6 +35,8 @@ git submodule update --init
 
 To upgrade, fetch and check out the desired commit inside the relevant submodule, then commit the updated submodule pointer in InferenceX. Benchmark workflows already initialize submodules. Slurm launchers make a local Git clone for each job so recipe staging and runtime writes do not modify the submodule, and record the actual commit for result provenance. NVIDIA setup clones locally; TileRT setup fetches its pinned fork commit over the network.
 
+B200, GB200, and GB300 resolve the job checkout's actual Hatch package version on local disk before installing it. This avoids repeated `git describe --dirty` scans of shared storage. The launcher preserves that version for both login and compute editable builds, uses a Python 3.12 login environment, and stops on environment or installation failures. The version is retained in `srt-slurm-version.txt` and the uploaded `srt-setup.log`; it supplements the pinned `srt-slurm-sha.txt` and does not replace source or patch provenance. Native compute jobs without a preserved version keep their existing version lookup.
+
 Single-node fixed-sequence recipes use NVIDIA upstream srt-slurm. ATOM recipes use
 the native `atomesh` frontend with one aggregate worker and
 `enable_multiple_frontends: false`. The router's pinned official image belongs in
@@ -160,6 +162,20 @@ Sources: [`configs/CONFIGS.md`](../configs/CONFIGS.md), [`validation.py`](../uti
 
 Fixed-sequence `8192/1024` scenarios may set `require-power: true` to opt into validated measured power. The matrix passes this flag to standard sweeps and manual E2E throughput jobs; eval-only and AgentX rows do not inherit it. Omit the field to preserve existing behavior. Enable it only alongside the corresponding runtime and result adapter, then qualify the complete selected scope.
 
+For native AgentX on B200 Nscale, GB200 and GB300, enable `telemetry.enabled` and
+`telemetry.required` in the SRT recipe, use `storage_subdir: power` and the
+`dcgm-exporter` container alias. The launchers inspect the selected recipe after
+native overrides, require the shared `agentic_srt.sh` window/result contract and
+forward the matrix concurrency to both replay and power validation. An invalid
+enabled contract fails before submission. Eval-only jobs retain real verification
+and do not produce performance power results.
+
+Qualification requires the strict power adapter to accept the recorded serving
+window, all participating GPUs identified by hostname and UUID, samples bracketing
+both boundaries and no adjacent sample gap over three seconds. Retain the raw
+telemetry, window and configuration audit plus producer SHA. Recipe enablement or
+passing local checks alone does not qualify measured power or publish results.
+
 ## Register and set up a runner
 
 Setup source: [`utils/runner_setup/RUNNER_SETUP.md`](../utils/runner_setup/RUNNER_SETUP.md). Config source: [`configs/CONFIGS.md#runners`](../configs/CONFIGS.md#runners).

docs/configuration-procedures_zh.md

@@ -35,6 +35,8 @@ git submodule update --init
 
 升级时，在对应子模块中获取并检出目标提交，再将更新后的子模块指针提交到 InferenceX。基准测试工作流已配置为自动初始化子模块。Slurm 启动器为每个作业创建本地 Git 克隆，避免配方准备和运行时写入修改子模块，并记录实际提交以供结果溯源。NVIDIA 启动器使用本地克隆；TileRT 启动器通过网络获取固定的分支提交。
 
+B200、GB200 和 GB300 在本地磁盘上解析作业检出副本的实际 Hatch 包版本，再安装运行环境，避免在共享存储上反复执行 `git describe --dirty` 扫描。启动器为登录节点和计算节点的可编辑构建保留同一版本，登录环境使用 Python 3.12；环境创建或安装失败时立即停止。版本写入 `srt-slurm-version.txt` 和上传的 `srt-setup.log`，作为固定 `srt-slurm-sha.txt` 的补充，不替代源码或补丁来源记录。没有保留版本文件的原生计算作业继续使用原有版本解析方式。
+
 单节点固定序列长度配方使用 NVIDIA 上游 srt-slurm。ATOM 配方使用原生 `atomesh`
 frontend、一个聚合 worker，并设置 `enable_multiple_frontends: false`。旧版基准 worker
 镜像不包含 AToMesh，因此通过 `frontend.container_image` 单独固定路由器的官方镜像。
@@ -111,6 +113,16 @@ STP（Single Token Prediction，单 Token 预测）是每次前向传播生成
 
 固定序列 `8192/1024` 场景可设置 `require-power: true`，要求经过验证的实测功耗。矩阵将此标记传递给标准 sweep 和手动 E2E 吞吐作业；eval-only 和 AgentX 行不继承该标记。省略此字段可保留现有行为。仅在对应 runtime 和结果适配器同时交付时启用，然后验证完整选定范围。
 
+对于 B200 Nscale、GB200 和 GB300 的原生 AgentX，在 SRT 配方中启用
+`telemetry.enabled` 和 `telemetry.required`，设置 `storage_subdir: power`，并使用
+`dcgm-exporter` 容器别名。launcher 在应用原生覆盖项后检查选定配方，要求使用共享
+`agentic_srt.sh` 的测量窗口和结果契约，并将矩阵并发同时传给回放和功耗验证。
+已启用但不符合契约的配置会在提交前失败。eval-only 作业保留真实验证，不生成性能功耗结果。
+
+验收要求严格功耗适配器接受记录的服务窗口，按 hostname 和 UUID 标识所有参与 GPU，
+采样覆盖窗口两端，且相邻样本间隔不超过三秒。保留原始遥测、窗口、配置审计和
+producer SHA。仅启用配方或通过本地检查不代表实测功耗合格，也不代表结果已发布。
+
 ## 注册并设置 runner
 
 设置来源：[`utils/runner_setup/RUNNER_SETUP.md`](../utils/runner_setup/RUNNER_SETUP.md)。配置来源：[`configs/CONFIGS.md#runners`](../configs/CONFIGS.md#runners)。

infx/srt_slurm/synthetic_acceptance.py

@@ -9,6 +9,7 @@
 import math
 import os
 import re
+import shlex
 import subprocess
 import sys
 from collections.abc import Mapping
@@ -241,15 +242,10 @@ def plan_commands(
     from srtctl.core.overrides import apply_overrides_to_recipe, parse_overrides
 
     path, _, selector = config.partition(":")
-    raw = yaml.safe_load(Path(path).read_text())
-    if not isinstance(raw, dict):
-        raise ValueError("Recipe must be a mapping")
+    raw, caller_overrides = recipe_with_overrides(path, arguments)
     parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
     parser.add_argument("--set", action="append")
     parser.add_argument("--unset", action="append")
-    existing, _ = parser.parse_known_args(arguments)
-    caller_overrides = parse_overrides(existing.set, existing.unset)
-    apply_overrides_to_recipe(raw, caller_overrides)
     commands = []
     for variant, recipe in selected_recipes(raw, selector or None):
         arguments_to_add = build_overrides(recipe, framework, environment, golden_dir=golden_dir)
@@ -271,17 +267,90 @@ def plan_commands(
     return commands
 
 
+def recipe_with_overrides(path: str, arguments: list[str]) -> tuple[dict[str, Any], list[Any]]:
+    """Use the same native caller-override ordering for inspection and submission."""
+    from srtctl.core.overrides import apply_overrides_to_recipe, parse_overrides
+
+    raw = yaml.safe_load(Path(path).read_text())
+    if not isinstance(raw, dict):
+        raise ValueError("Recipe must be a mapping")
+    parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False)
+    parser.add_argument("--set", action="append")
+    parser.add_argument("--unset", action="append")
+    existing, _ = parser.parse_known_args(arguments)
+    overrides = parse_overrides(existing.set, existing.unset)
+    apply_overrides_to_recipe(raw, overrides)
+    return raw, overrides
+
+
+def inspect_power(config: str, arguments: list[str], environment: Mapping[str, str]) -> str:
+    """Check the native power artifact/window contract before submission."""
+    from marshmallow import ValidationError
+    from srtctl.core.config import expand_engine_config_defaults, resolve_config_with_defaults
+    from srtctl.core.schema import SrtConfig
+
+    path, _, selector = config.partition(":")
+    raw, _ = recipe_with_overrides(path, arguments)
+    variants = selected_recipes(raw, selector or None)
+    if len(variants) != 1:
+        # Preserve existing non-AgentX, non-power multi-variant submissions.
+        # Their lifecycle is separate from the single-job PowerX result contract.
+        if (
+            variants
+            and environment["IS_AGENTIC"] != "1"
+            and all(not recipe.get("telemetry", {}).get("enabled", False) for _, recipe in variants)
+        ):
+            return "none"
+        raise ValueError("SRT power launcher requires exactly one selected recipe per job")
+    recipe = variants[0][1]
+    if not recipe.get("telemetry", {}).get("enabled", False):
+        return "none"
+    resolved = resolve_config_with_defaults(recipe, None)
+    expand_engine_config_defaults(resolved)
+    try:
+        typed = SrtConfig.Schema().load(resolved)
+    except ValidationError as error:
+        raise ValueError(f"Invalid power recipe: {error}") from error
+    if not typed.telemetry.enabled:
+        return "none"
+    if typed.telemetry.dcgm_exporter is None:
+        raise ValueError("SRT PowerX requires telemetry.dcgm_exporter")
+    if typed.telemetry.dcgm_exporter.container_image != "dcgm-exporter":
+        raise ValueError("SRT PowerX requires the dcgm-exporter container alias")
+    if typed.telemetry.storage_subdir != "power":
+        raise ValueError("SRT PowerX requires telemetry.storage_subdir: power")
+    if environment["IS_AGENTIC"] != "1":
+        return "dcgm"
+    benchmark = typed.benchmark
+    if (
+        benchmark.type != "custom"
+        or shlex.split(benchmark.command or "")
+        != ["bash", "/infmax-workspace/benchmarks/multi_node/agentic_srt.sh"]
+        or benchmark.env.get("RESULT_DIR") != "/logs/agentic"
+        or benchmark.env.get("INFMAX_CONTAINER_WORKSPACE") != "/infmax-workspace"
+        or benchmark.env.get("IS_MULTINODE") != "true"
+    ):
+        raise ValueError("AgentX PowerX requires the shared agentic_srt.sh window/result contract")
+    if not typed.telemetry.required:
+        raise ValueError("AgentX PowerX requires telemetry.required: true")
+    return "agentx"
+
+
 def main(argv: list[str] | None = None) -> int:
     parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--inspect-power", action="store_true")
     parser.add_argument("config")
     parser.add_argument("framework")
     parser.add_argument("arguments", nargs=argparse.REMAINDER)
     args = parser.parse_args(argv)
     arguments = args.arguments[1:] if args.arguments[:1] == ["--"] else args.arguments
     try:
+        if args.inspect_power:
+            print(inspect_power(args.config, arguments, os.environ))
+            return 0
         commands = plan_commands(args.config, args.framework, arguments, os.environ)
     except (OSError, KeyError, ValueError, TypeError, yaml.YAMLError) as error:
-        print(f"ERROR: golden acceptance: {error}", file=sys.stderr)
+        print(f"ERROR: SRT recipe: {error}", file=sys.stderr)
         return 1
     for command in commands:
         result = subprocess.run(command, check=False)

perf-changelog.yaml

@@ -8887,3 +8887,92 @@
     - "Add DeepSeek-V4-Pro-0813 golden AL for draft lengths 4, 5, 7 and 8 (3.36 / 3.61 / 3.73 / 3.47)."
     - "Agentic PD router: pin --decode-policy round_robin so decode no longer inherits the prefill --policy (consistent_hashing)."
   pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3256
+
+- config-keys:
+    - dsr1-fp4-b200-dynamo-trt
+    - dsr1-fp8-b200-dynamo-trt
+    - dsr1-fp4-b300-dynamo-trt
+    - dsr1-fp8-b300-dynamo-trt
+    - kimik3-fp4-h200-vllm-agentic-latency
+    - kimik3-fp4-h200-vllm-agentic-balanced
+    - kimik3-fp4-h200-vllm-agentic-simple
+    - dsv4-fp8-h200-dynamo-sglang-agentic-agg
+    - dsr1-fp8-h200-dynamo-trt
+    - dsr1-fp8-h100-dynamo-trt
+    - dsr1-fp8-h100-dynamo-sglang
+    - dsr1-fp4-gb200-dynamo-trt
+    - dsr1-fp8-gb200-dynamo-trt
+    - dsr1-fp8-gb200-dynamo-sglang
+    - dsr1-fp8-gb300-dynamo-sglang
+    - dsr1-fp4-gb200-dynamo-sglang
+    - dsr1-fp4-gb300-dynamo-trt
+    - dsr1-fp4-gb300-dynamo-sglang
+    - dsr1-fp8-gb300-dynamo-trt
+    - dsr1-fp8-h200-dynamo-sglang
+    - dsr1-fp4-b200-dynamo-sglang
+    - dsr1-fp8-b200-dynamo-sglang
+    - dsr1-fp8-b200-dynamo-sglang-mtp
+    - dsr1-fp4-b200-dynamo-sglang-mtp
+    - qwen3.5-fp8-gb200-dynamo-sglang
+    - qwen3.5-fp4-gb300-dynamo-sglang
+    - qwen3.5-fp4-gb300-dynamo-trt
+    - qwen3.5-fp4-gb300-dynamo-trt-mtp
+    - qwen3.5-fp4-gb300-dynamo-trt-agentic-disagg
+    - qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg
+    - qwen3.5-fp4-gb300-dynamo-sglang-agentic-disagg
+    - qwen3.5-fp4-gb300-dynamo-sglang-agentic-pp-pareto
+    - qwen3.5-fp4-gb300-dynamo-sglang-agentic-agg-pareto
+    - dsv4-fp4-gb300-dynamo-vllm-agentic
+    - qwen3.5-fp4-gb200-dynamo-sglang-agentic-mtp
+    - minimaxm3-fp4-gb300-dynamo-vllm-agentic-mtp-disagg
+    - minimaxm3-fp4-gb200-dynamo-vllm-agentic-agg-mtp
+    - minimaxm3-fp4-gb200-dynamo-vllm-agentic-disagg-mtp
+    - minimaxm3-fp4-gb200-dynamo-trt-agentic-agg-mtp
+    - dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-agg
+    - dsv4-fp4-gb200-dynamo-vllm-agentic-mtp-disagg
+    - dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-agg
+    - dsv4-fp4-gb300-dynamo-vllm-agentic-mtp-disagg
+    - dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-agg
+    - dsv4-fp4-gb200-dynamo-vllm-agentic-mtp2-disagg
+    - kimik3-fp4-gb200-dynamo-vllm-agentic
+    - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-dcp16-agg
+    - kimik3-fp4-gb200-dynamo-vllm-agentic-mooncake-dcp16-agg
+    - dsv4-fp4-gb300-dynamo-trt-agentx
+    - kimik3-fp4-gb200-dynamo-vllm-agentic-dspark-mooncake-tp8pp2
+    - dsv4-fp4-gb300-dynamo-sglang-agentic-agg
+    - dsv4-fp4-gb300-dynamo-sglang-agentic-disagg
+    - qwen3.5-fp8-gb300-dynamo-sglang
+    - glm5.2-fp8-h200-dynamo-sglang-agentic-mtp-2p2d
+    - glm5.2-fp8-h200-dynamo-sglang-agentic-mtp-1p1d-hicache
+    - glm5.2-fp8-h200-dynamo-sglang-agentic-mtp-agg
+    - glm5.2-fp4-b200-dynamo-sglang-agentic-agg
+    - glm5.2-fp4-b200-dynamo-sglang-agentic-disagg
+    - glm5.2-fp4-gb200-dynamo-sglang-agentic-agg
+    - glm5.2-fp4-gb200-dynamo-sglang-agentic-disagg
+    - glm5.2-fp4-gb200-dynamo-sglang-agentic-mtp
+    - glm5.2-fp4-gb200-dynamo-sglang-agentic-mtp-agg
+    - glm5.2-fp4-gb300-dynamo-sglang-agentic-agg
+    - glm5.2-fp4-gb300-dynamo-sglang-agentic-disagg
+    - glm5.2-fp4-gb300-dynamo-trt-agentic-disagg-mtp
+    - qwen3.5-fp8-gb200-dynamo-sglang-mtp
+    - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-disagg
+    - kimik3-fp4-gb300-dynamo-vllm-agentic-dspark-mooncake-dcp8-agg
+    - kimik3-fp4-b200-dynamo-vllm-agentic-dspark
+    - qwen3.5-fp8-gb300-dynamo-sglang-mtp
+    - glm5.1-fp8-b200-tilert-agentic
+    - dsv4-fp4-b200-dynamo-sglang-agentic-agg
+    - dsv4-fp4-b200-dynamo-sglang-agentic-disagg
+    - qwen3.5-fp8-b200-dynamo-sglang-agentic-disagg-mtp
+    - qwen3.5-fp8-b300-dynamo-sglang-agentic-disagg
+    - qwen3.5-fp8-mi355x-sglang-disagg
+    - qwen3.5-fp4-mi355x-sglang-disagg
+    - dsr1-fp8-mi355x-sglang-disagg
+    - dsr1-fp8-mi355x-sglang-disagg-mtp
+    - dsr1-fp4-mi355x-sglang-disagg
+    - dsr1-fp4-mi355x-sglang-disagg-8k1k-mtp
+    - dsr1-fp4-mi355x-sglang-disagg-mtp
+  description:
+    - "Resolve required AgentX power from selected native recipes on B200, GB200 and GB300; preserve replay concurrency and strict power audits."
+    - "Resolve srtctl package versions on local disk for those pools and stop native compute setup on dependency-sync failure; shared runtime scope is retained for qualification."
+    - "B200、GB200 和 GB300 根据选定原生配方启用必需功耗，保留回放并发和严格功耗审计；在本地磁盘解析包版本，原生计算节点依赖安装失败即停止。验收范围保留共享 runtime 的影响。"
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3429