[AMD][MI35X] 0926 DSV4 sglang mtp agentic

[1am9trash]

Summary:

• Update image to lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926.
• Switch attention to fp8 unified KV; enable grouped decode on TP8-only arms at conc >= 16.
• Enable breakable CUDA-graph prefill (BCG) at conc <= 4.
• Enable EP8 + MegaMoE for DP-attention arms at conc >= 128 (including conc 384).

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps only run on labeled PRs. Add full-sweep-fail-fast (strongly recommended); use full-sweep-enabled only when matrix jobs should continue after a failure.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**扫描仅在带有标签的 PR 上运行。强烈建议添加 full-sweep-fail-fast；仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

I reviewed this PR and didn't find any bugs. Because it introduces a new EP8+MegaMoE/EPLB code path, several new decode/prefill tuning flags (unified KV fp8, grouped-decode ASM, breakable CUDA-graph prefill), and shifts the DP-attention search-space arm from EP1 to EP8 with a new 384-concurrency point, a human (ideally one of the CODEOWNERS already on configs/amd-master.yaml) should still confirm the perf numbers before merge.

What was reviewed:

• New BCG_ARGS/EP_SIZE-branch env vars in the script — these are recipe-local tuning constants derived from already-validated CONC/EP_SIZE, not unvalidated caller config.
• perf-changelog.yaml entry — correctly appended at the tail; its pull/XXX placeholder matches an existing untouched entry elsewhere in the file, so it's not a new deviation.
• Image bump scope — recipe is multinode: false, so the multi-node model.container == image cross-check doesn't apply.
• New EP8 path writes expert-distribution stat files to shared /tmp rather than $RESULT_DIR (line 162) — a plausible minor collision risk on a reused AMD runner, but low severity since it's only diagnostic EPLB stat output, not benchmark results.

Extended reasoning...

The diff is confined to one AMD MI355X SGLang recipe: an image bump, a DP-attention search-space change (ep:1→ep:8, +conc 384), and a script restructure that adds a new EP8+MegaMoE/EPLB code path alongside fp8 unified KV, grouped decode, and breakable CUDA-graph prefill flags. No injection/auth/secrets surface is touched; the risk here is purely performance/behavioral correctness of a GPU serving recipe. The change is larger and more consequential than a mechanical config tweak (new code branch, several new env vars affecting memory fraction and expert placement), so despite the automated hunt finding nothing, a human with domain context should still confirm the new EP8 arm behaves as intended before it ships.

This review covers commit b0a5d2a, which is no longer the latest commit on this pull request; later commits are not covered by it.

[functionstackx]

InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest main into this PR: we have migrated single-node AgentX onto native srt-slurm (#3428), so AgentX configs are now declarative YAML recipes, not per-config 1000+ line bash slop scripts. Please also delete the old benchmarks/single_node/** scripts (see this recipe for the new format).

[1am9trash]

[Out of Date] /reuse-sweep-run 36289952547

[1am9trash]

https://inferencex.semianalysis.com/inference?unofficialRun=36289952547

[1am9trash]

[Out of Date]

• Eval: https://inferencex.semianalysis.com/evaluation?unofficialRun=36289952547
• Perf: https://inferencex.semianalysis.com/inference?unofficialRun=36289952547

[1am9trash]

The cookbook PR has been merged.
PR: sgl-project/sglang#41458

[functionstackx]

Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding

[1am9trash]

@functionstackx
I will do the rebase again. Thanks for the info.

[1am9trash]

Only path change in #3525. No config change in rebase.

[1am9trash]

/reuse-sweep-run 36401947630

[1am9trash]

After rebase:
https://inferencex.semianalysis.com/inference?unofficialRun=36401947630

[HaiShaw]

@functionstackx thanks for helping 👍

[HaiShaw]

@1am9trash Showing a conflict just now.

[1am9trash]

Conflict fixed.

[yctseng0211]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/inference?unofficialRun=36401947630
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://inferencex.semianalysis.com/evaluation?unofficialruns=36401947630
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here
• recipe : sgl-project/sglang#41458
• draft precision:
  • Everything is same to the last commit. No change on draft model side.
  • In-checkpoint DSpark head on deepseek-ai/DeepSeek-V4-Pro-0813; mtp.* attn / shared experts / main_proj stored F8_E4M3, routed experts MXFP4, norms/gate/markov BF16.
  • Pinned v0.5.20-rocm720-mi35x-20260926 defaults convert wo_a FP8->BF16 (Finished streaming dequant fp8 wo_a).
  • EP8 inherits MegaMoE a8w4 on MXFP4 experts (code default), other arms aiter MXFP4 with default bf16 acts; no draft path/quant override.
  • SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE unset (default false).
  • Effective = shipped default = baseline.

Signed: @yctseng0211

[github-actions]

✅✅✅ Verdict: PASS ✅✅✅

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @yctseng0211 is a named owner of inferencex-e2e/configs/amd-master.yaml; the recipe and perf-changelog.yaml fall only under the * catch-all. PR head is still c98a1c97, so no newer commit went unassessed.

✅ Check 1 (Passing sweep on in-PR commit): PASS — run 36401947630 ran on in-PR commit cb45b6da. All 9 agentic / jobs and the agentic eval / job succeeded (this is an agentic-only config, so there are no single-node */ jobs). The later commits only merge main: the recipe and the dsv4-fp4-mi355x-sglang-agentic-mtp master-config entry are byte-identical at the pinned SHA.

✅ Check 2 (Evals): PASS — GSM8K em_strict 0.973 ± 0.004 (n=1319) on the TP8/EP8/DPA c384 arm, run on lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926, the same image as the PR config.

✅ Check 3 (Recipe linked/merged/complete): PASS — sgl-project/sglang#41458 was MERGED 2026-09-27. Its DeepSeek-V4 cookbook §3.7 (MI355X FP4 DSpark agentic) matches the major args: model, TP8 / DP8 / EP8, DP attention + DP lm-head, attention-backend dsv4, fp4 indexer, kv-cache-dtype fp8_e4m3, SGLANG_DSV4_UNIFIED_KV_FP8=1, moe-a2a-backend megamoe with a8w4 + EPLB, BCG at conc ≤ 4, grouped ASM on TP-only conc ≥ 16, DSPARK block size 6, and the same image. Informational only: mem-fraction, chunked-prefill, max-running-requests, cuda-graph bs, and router/timeout knobs are InferenceX tuning.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 36401947630 was posted by 1am9trash (COLLABORATOR).

✅ Check 5 (Latest checklist template): PASS — every item in the current template is present and checked.

✅ Check 6 (Upstream images / engine-first): PASS — framework: sglang on MI355X uses the upstream lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260926.

✅ Check 7 (No deprecated models/scenarios): PASS — dsv4 agentic coding (including the DSpark arms) is active per MODELS.md as of 2026-09-29.

✅ Check 8 (No architecture hacks): PASS — no --hf-overrides, JSON model overrides, or model-file edits; the changes are kernel, backend, KV-dtype and parallelism choices only.

✅ Check 9 (Spec-decode via chat template): PASS — the agentic replay goes through /v1/chat/completions with --endpoint-type chat (benchmark_lib.sh:3272).

✅ Check 10 (No engine patches): PASS — the diff adds no patches, heredoc rewrites, monkey-patching, or engine wheel installs.

✅ Check 11 (Agentic golden AL): PASS — the harness injects SGLANG_SIMULATE_ACC_LEN=3.77 with match-expected / real-draft-token, confirmed in the run logs for all 9 points. That equals the golden dsv4-pro-0813-dspark.yaml thinking_on value for block size 6.

➖ Check 12 (Append-only): N/A — the new changelog entry has no append-only: true.

✅ Check 13 (Draft runs as shipped): PASS — the draft is the DSpark head bundled in deepseek-ai/DeepSeek-V4-Pro-0813, with no draft path, draft quantization, or dtype override. The logs show the image's default Finished streaming dequant fp8 wo_a. SGLANG_AMD_FLYDSL_MEGA_QUANT=a8w4 restates SGLang's code default (get() or "a8w4" in mega_moe_flydsl.py). SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is unset in the recipe and the run environment (default EnvBool(False)).

✅ Check 14 (Pareto coverage): PASS — curve dsv4 / agentic-coding / MI355X SGLang FP4 / p90 E2EL / v0.5.20-20260926 image: 9 measured points, 7 on the frontier (pareto_coverage.py), taken from the bmk_agentic_* artifacts of run 36401947630 attempt 1.

Assessed commit: c98a1c976bd98cc39cf2ec165def6f847ea6cf3e.

[1am9trash]

Hi, @Oseltamivir
May you help to review this PR? Thanks.
This PR enables megamoe, fp8 kv attn, prefill BCG on amd sglang dpsk-v4 model.
Run-sweep passed and YC helped to review the PR.