Conversation
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Claude finished @xinli-sw's task in 3m 48s —— View job Review of PR #3395
Summary: One blocking issue, posted inline. The new Everything else checks out: the image is a public Docker Hub tag, the changelog entry is appended at the tail, 中文摘要：发现一个阻塞问题，已内联评论。共享脚本 其余检查均通过：镜像为公开 Docker Hub 标签，changelog 条目追加在文件末尾， |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
xinli-sw
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=37046855925 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=37046855925 |
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline findings, I also checked whether doubling VLLM_ENGINE_READY_TIMEOUT_S to 7200s in runners/launch_gb200-nv.sh:113 could cost a sweep its Slurm wall-clock instead of a clean readiness-timeout error — the default SALLOC_TIME_LIMIT for these single-node sweeps is 480 minutes (8h, from the benchmark workflow), comfortably longer than the new 2h engine-ready timeout, so this isn't a practical risk.
Extended reasoning...
Confirmed via runners/runtime_settings.sh and .github/workflows/.yml that SALLOC_TIME_LIMIT for these sweeps defaults to 480 minutes, well above the new 7200s (120 min) VLLM_ENGINE_READY_TIMEOUT_S, so the candidate wall-clock-loss concern is not realistic under current defaults. Separately verified via git diff and runners/.sh that the new DSV41_BLACKWELL_ATTENTION gate is exported only by launch_gb200-nv.sh (not by launch_b200-nscale-slurm.sh, launch_gb300-nv.sh, or launch_h200-dgxc-slurm.sh), so the attention-config args no longer key off GPU_MEM_MIB alone.
|
|
||
| - config-keys: | ||
| - dsv41flash-fp4-b200-vllm-agentic-dspark |
There was a problem hiding this comment.
🟡 (optional) This entry's config-keys list only dsv41flash-fp4-b200-vllm-agentic-dspark and dsv41flash-fp4-gb200-vllm-agentic-dspark, but the fixed script (benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh) is also symlinked from dsv41flash_fp4_gb300_vllm_mtp.sh and dsv41flash_fp4_h200_vllm_mtp.sh, so dsv41flash-fp4-gb300-vllm-agentic-dspark and dsv41flash-fp4-h200-vllm-agentic-dspark get the same SIGPIPE fix but aren't listed. Anyone auditing perf-changelog.yaml by config-key for GB300 or H200 will miss this fix's context even though it changed their behavior too. Fix: include every config that resolves to the shared script (b200, gb200, gb300, h200) in config-keys, not just the two SKUs named in the PR title.
Why this was flagged
configs/nvidia-master.yaml:8210 (dsv41flash-fp4-gb300-vllm-agentic-dspark) and configs/nvidia-master.yaml:8239 (dsv41flash-fp4-h200-vllm-agentic-dspark) both point at benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh via symlink, the same script whose GPU_MEM_MIB probe is fixed by this diff. perf-changelog.yaml:8851-8852 lists only b200 and gb200 in config-keys for that fix entry (perf-changelog.yaml:8850-8858). AGENTS.md requires perf-changelog entries to reference the affected config keys as the source of truth for what changed. An engineer or tool filtering perf-changelog.yaml by dsv41flash-fp4-gb300-vllm-agentic-dspark or dsv41flash-fp4-h200-vllm-agentic-dspark would not find this SIGPIPE fix despite it changing their runs' behavior, unlike on base where the entry didn't exist for anyone.
Verification: nit. The SIGPIPE fix lives in the shared script benchmarks/single_node/agentic/dsv41flash_fp4_vllm_mtp.sh (diff replaces nvidia-smi ... | head -n1 | tr -d ' ' with GPU_MEM_MIB=$(nvidia-smi ...) + %%$'\n'* + numeric validation). ls -la confirms four vLLM symlinks point at this one file: dsv41flash_fp4_b200_vllm_mtp.sh, gb200, gb300, and h200 all -> dsv41flash_fp4_vllm_mtp.sh. All…
e8c965a to
e08edaa
Compare
|
InferenceX has switched away from unmaintainable bash scripts to YAML files that don't repeat the same stuff over and over again. Please merge the latest |
6b659e9 to
0485f33
Compare
|
Sorry, over the weekend, there was 2 major refactors to clean up the technical debt accumalated over the past 11 months of moving at the speed of light. We don't see any major refactors in the forthseeable future besides cleaning up AMD multinode AgentX pile of bash. As much, due to the refactors, u would need to ask your agent to rebase from remote main@latest. Thank you in advance for ur understanding |
0485f33 to
4d49128
Compare
xinli-sw
added
full-sweep-enabled
f42289c to
5a0af7e
Compare
wzhao18
left a comment
wzhao18
left a comment
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Assessed commit:
5a0af7e0ef64fe17fd5a9559634417e16b2e9c42. - PR validation: CI and full sweep, attempt 2 passed on the assessed commit. The completed sweep covers all 16 TP2/TP4 performance jobs at concurrency 1–128. The latest collected artifact contains 16 usable throughput/latency measurements plus one stale empty TP4 c128 record; that empty record is excluded, and the successful TP4 c128 rerun is used. Power data is valid for all eight TP4 measurements; all eight TP2 records report
power_valid: 0, so no validated TP2 power claim is made. - Evals: the same sweep and image passed GSM8K at TP4 c128, with
em_strict = 0.9727065959059894,n_eff = 1319, andinfrastructure_success: true. Evidence:eval_results_all. This is one evaluated configuration, not an eval at every performance point. - Reuse:
/use 36805208558was posted by CODEOWNERwzhao18on this PR to select the passing sweep for publication. - Recipe documentation: published DeepSeek-V4.1-Flash recipe, with the Blackwell sparse-attention settings documented in vllm-project/recipes#995, merged September 20, and GB200 TP2 Engram offload in vllm-project/recipes#1002, merged September 21.
- Image and serving stack: upstream
vllm/vllm-openai:nightly-ac68c3087215e0a4f3cdfa218508c6aada57235d, matching the recipe and master config. Performance and eval logs report vLLM0.30.1rc1.dev396+gac68c3087. No engine patch or replacement wheel is introduced. - Draft precision: inspected
deepseek-ai/DeepSeek-V4.1-Flash@2cba9e42aa026125f3ed06c6d98c1db82f7ca027. Safetensors headers contain 2,401 embeddedmtp.*tensors across shards 44–46: packed MXFP4 expert weights, FP8 dense weights and E8M0 scales, plus BF16/F32 auxiliary tensors. The pinned upstream DSpark loader loads these embedded weights through the checkpoint’s default quantization handling. The effective recipe adds no draft checkpoint substitution, dtype override, or quantization override.SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis absent and inapplicable to this vLLM submission. - AgentX acceptance and chat path: throughput uses five-token probabilistic DSpark drafting with synthetic AL
3.51, matchingthinking_onK5 in the committed golden curve. Runtime artifacts confirm synthetic acceptance; eval uses real block rejection without synthetic acceptance. Replay uses/v1/chat/completions. - Pareto evidence: run
36805208558, attempt 2, assessed SHA above,results_bmk, AgentX P90 E2EL versus total token throughput/GPU, one GB200 vLLM FP4 visual curve and one image. There are 16 usable measured throughput/latency points and 7 frontier points: TP4 c8/c16/c32/c128 and TP2 c8/c16/c32. The stale empty TP4 c128 record is excluded; power validity is not used as a throughput/E2EL filter, and the invalid TP2 power records are not presented as validated power measurements. The repository’spareto_coveragehelper reports PASS. append-onlyis N/A. This PR changes existing recipes and the image and uses a regular changelog entry. All historical changelog bytes are preserved, with the new entry appended at the tail.
Signed: wzhao18
|
@wzhao18 staged run 36805208558: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-10-01~r36805208558 This run remains available across future |
94a4b92 to
9100452
Compare
将 GB200 的 vLLM 镜像和服务参数与已通过的 B200 配置对齐。
9100452 to
5f0511d
Compare
This branch has not been deployed
3 participants
Pin GB200 DeepSeek-V4.1-Flash vLLM to nightly
ddd6fbca148a867aad1fcab7ec72f582b9977db4on the native srt-slurm AgentX recipe. Use FlashInfer sparse attention at TP2 and TP4 with MXFP4 indexer KV, sparse logits, fp8 KV cache, and Engram CPU offload. The engine readiness timeout is 7200 s. Topology and concurrency remain unchanged.中文
在原生 srt-slurm AgentX 配方中，将 GB200 DeepSeek-V4.1-Flash vLLM 固定到 nightly
ddd6fbca148a867aad1fcab7ec72f582b9977db4。TP2 和 TP4 均使用 FlashInfer 稀疏注意力，并启用 MXFP4 索引器 KV、稀疏 logits、fp8 KV 缓存及 Engram CPU 卸载。 引擎就绪超时为 7200 秒。 拓扑与并发设置不变。