Skip to content

fix(amd): enable DSpark speculative decoding for DeepSeek-V4-Pro-0813 MI355X disaggregated AgentX #3191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ichbinblau wants to merge 6 commits into from
Closed

fix(amd): enable DSpark speculative decoding for DeepSeek-V4-Pro-0813 MI355X disaggregated AgentX #3191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6060acb
fix(amd): use Pro-0813 golden acceptance for AgentX
cquil11 Sep 15, 2026
8fa6ee5
enable dspark
ichbinblau Sep 15, 2026
a6a97c3
align config and image with intenal sweep
ichbinblau Sep 16, 2026
ce07190
fix(amd): forward required AgentX client policy inputs
cquil11 Sep 16, 2026
5a5ad97
update perf-changelog.yaml
ichbinblau Sep 16, 2026
329e7ea
docs: point perf-changelog pr-link to #3191
ichbinblau Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions benchmarks/multi_node/amd_utils/models.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@
# Schema:
# <model-name>:
# base_flags: str # Common flags for both prefill and decode
# mtp_flags: str # Appended to decode when DECODE_MTP_SIZE > 0
# mtp_flags: str # Appended when DECODE_MTP_SIZE > 0 and spec-decoding is mtp
# dspark_flags: str # Replaces mtp_flags when spec-decoding is draft_model
# dp_flags: str # Appended when DP attention is enabled (prefill or decode)
# ep_flags: str # Appended when EP is enabled. EP-specific MoE knobs only
# # (a2a backend, deepep mode, ep-dispatch algorithm). With
Expand Down Expand Up @@ -354,9 +355,16 @@ DeepSeek-R1-0528-MXFP4-v2:

DeepSeek-V4-Pro-AgentX: &DeepSeek-V4-Pro-AgentX
base_flags: "--enable-deepseek-v4-fp4-indexer --watchdog-timeout 3600 --load-balance-method round_robin --kv-cache-dtype fp8_e4m3 --attention-backend dsv4 --page-size 256 --swa-full-tokens-ratio 0.1 --enforce-shared-experts-fusion --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 --disaggregation-transfer-backend mori --tokenizer-worker-num 8 --stream-interval 20 --log-level info --log-level-http error"
dp_flags: "--enable-dp-attention --swa-full-tokens-ratio 0.15 --enable-dp-attention-local-control-broadcast"
dp_flags: "--enable-dp-attention --enable-dp-lm-head --swa-full-tokens-ratio 0.15 --enable-dp-attention-local-control-broadcast"
ep_flags: "--ep-dispatch-algorithm fake --moe-a2a-backend mori --deepep-mode normal"
mtp_flags: "--speculative-algorithm EAGLE --speculative-eagle-topk 1"
# Selected instead of mtp_flags when the arm sets spec-decoding: draft_model.
# The DSpark draft head ships inside the DeepSeek-V4-Pro-0813 target checkpoint
# (dspark_block_size / dspark_markov_rank / dspark_target_layer_ids in
# config.json), so --speculative-draft-model-path defaults to --model-path and
# no separate draft checkpoint is needed. DECODE_MTP_SIZE carries the block
# size gamma; server_sglang.sh derives the verify window as gamma + 1.
dspark_flags: "--speculative-algorithm DSPARK --speculative-eagle-topk 1"
prefill:
disable_radix_cache: false
disable_cuda_graph: true
Expand Down
18 changes: 17 additions & 1 deletion benchmarks/multi_node/amd_utils/server_sglang.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ def parse_range(cuda_range, default_start, default_end):
# Output shell variables
print(f'MODEL_BASE_FLAGS=\"{m.get(\"base_flags\", \"\")}\"')
print(f'MODEL_MTP_FLAGS=\"{m.get(\"mtp_flags\", \"\")}\"')
print(f'MODEL_DSPARK_FLAGS=\"{m.get(\"dspark_flags\", \"\")}\"')
print(f'MODEL_DP_FLAGS=\"{m.get(\"dp_flags\", \"\")}\"')
print(f'MODEL_EP_FLAGS=\"{m.get(\"ep_flags\", \"\")}\"')

Expand Down Expand Up @@ -382,7 +383,18 @@ build_server_config() {
local specific_config=""

if [ "$decode_mtp_size" -gt 0 ]; then
mtp_config="${MODEL_MTP_FLAGS} --speculative-num-steps ${decode_mtp_size} --speculative-num-draft-tokens $((decode_mtp_size + 1))"
if [[ "${SPEC_DECODING:-}" == "draft_model" ]]; then
if [[ -z "${MODEL_DSPARK_FLAGS// }" ]]; then
echo "FATAL: SPEC_DECODING=draft_model but model '${model_name}' has no dspark_flags in models.yaml." >&2
exit 1
Comment on lines +386 to +389

@claude claude Bot Sep 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) The new fail-fast guard for missing dspark_flags does not actually stop the run: exit 1 at line 389 runs inside build_server_config, which is only ever invoked via command substitution (PREFILL_SERVER_CONFIG=$(build_server_config ...) at line 454). exit there only kills that subshell; with no set -e in this script and no $? check after the assignment, the script prints the FATAL line to stderr but keeps running with PREFILL_SERVER_CONFIG/DECODE_SERVER_CONFIG left empty (the function returns before reaching its final echo "$full_config"), so the server launches with a broken/missing config instead of aborting. Fix: after each build_server_config call, check the exit status (e.g. `PREFILL_SERVER_CONFIG=$(build_server_config ...) …

Extended reasoning...

…|| exit 1`) or have build_server_config return a status the caller checks, so a missing dspark_flags case (or any future error path in this function) actually halts the job for both the prefill and decode invocations at lines 454-455.

build_server_config is called via command substitution at lines 454-455, which bash always runs in a subshell. When SPEC_DECODING=draft_model and MODEL_DSPARK_FLAGS is empty, the function hits exit 1 at line 389 before it ever reaches the trailing echo "$full_config". That exit only terminates the subshell created for the command substitution; the parent server_sglang.sh script has no set -e and never inspects $? right after the assignment, so execution continues. PREFILL_SERVER_CONFIG (and likewise DECODE_SERVER_CONFIG from the second call) end up empty or truncated instead of containing --tp-size/base flags/etc. The script then proceeds to launch the sglang server with this incomplete config, producing a confusing downstream failure (or a badly configured server) instead of the clear early FATAL abort the PR intends for a model without…

Verification: nit. The candidate's mechanism is factually correct. The new fail-fast guard at server_sglang.sh:387-389 does exit 1 inside build_server_config(), but that function is invoked only via command substitution — PREFILL_SERVER_CONFIG=$(build_server_config ...) at line 454 and DECODE_SERVER_CONFIG=$(build_server_config ...) at line 455 — which bash runs in a subshell. The exit 1 therefore… | nit.…

fi
# DSpark proposes a whole block per step rather than walking a chain,
# so num-steps is pinned to 1 and decode_mtp_size is read as the block
# size gamma. The verify window is gamma + 1, same arithmetic as MTP.
mtp_config="${MODEL_DSPARK_FLAGS} --speculative-dspark-block-size ${decode_mtp_size} --speculative-num-steps 1 --speculative-num-draft-tokens $((decode_mtp_size + 1))"
else
mtp_config="${MODEL_MTP_FLAGS} --speculative-num-steps ${decode_mtp_size} --speculative-num-draft-tokens $((decode_mtp_size + 1))"
fi
fi

if [[ "$enable_dp" == "true" ]]; then
Expand Down Expand Up @@ -1382,6 +1394,10 @@ else
DeepSeek-V4-Pro-0813:1) DSV4_GOLDEN_AL=1.84 ;;
DeepSeek-V4-Pro-0813:2) DSV4_GOLDEN_AL=2.51 ;;
DeepSeek-V4-Pro-0813:3) DSV4_GOLDEN_AL=3.01 ;;
# The dspark curve does carry values out to length 8, but only
# 1-3 are wired here, matching upstream. Refuse rather than fall
# through to the original V4 curve, which would silently
# under-simulate this checkpoint.
DeepSeek-V4-Pro-0813:*)
echo "ERROR: Pro-0813 draft length ${DECODE_MTP_SIZE} has no golden AL wired here; refusing to use the original V4 curve." >&2
exit 1
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/runtime_settings.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,4 @@ export SGLANG_TORCH_PROFILER_DIR='/workspace'
export VLLM_TORCH_PROFILER_DIR='/workspace'

# Explicitly forward these settings across container boundaries.
export INFERENCEX_RUNTIME_ENV_VARS="OPENAI_API_KEY SWEBENCH_EXPECTED_INSTANCES SWEBENCH_AGENT_STEP_LIMIT SWEBENCH_AGENT_TIMEOUT SWEBENCH_AGENT_EXIT_GRACE SWEBENCH_WATCHDOG_POLL SWEBENCH_SANDBOX_SWEEP SWEBENCH_SKIP_SCORE SWEBENCH_EVAL_TIMEOUT SWEBENCH_SCORE_TIMEOUT SWEBENCH_MAX_WORKERS EVAL_ENDPOINT_READY_TIMEOUT_SECONDS EVAL_MODEL_STABILIZATION_SECONDS AIPERF_FAILED_REQUEST_THRESHOLD AIPERF_LIVE_FAILED_REQUEST_THRESHOLD AIPERF_TRACE_IDLE_GAP_CAP_SECONDS AIPERF_PYTHON_VERSION AIPERF_WARMUP_REQUESTS_PER_LANE AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES AGENTIC_WARMUP_GRACE_PERIOD AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS AIPERF_UNSAFE_OVERRIDE ENABLE_AGENTX_POWER VLLM_ENGINE_READY_TIMEOUT_S SGLANG_TORCH_PROFILER_DIR VLLM_TORCH_PROFILER_DIR"
export INFERENCEX_RUNTIME_ENV_VARS="OPENAI_API_KEY SWEBENCH_EXPECTED_INSTANCES SWEBENCH_AGENT_STEP_LIMIT SWEBENCH_AGENT_TIMEOUT SWEBENCH_AGENT_EXIT_GRACE SWEBENCH_WATCHDOG_POLL SWEBENCH_SANDBOX_SWEEP SWEBENCH_SKIP_SCORE SWEBENCH_EVAL_TIMEOUT SWEBENCH_SCORE_TIMEOUT SWEBENCH_MAX_WORKERS EVAL_ENDPOINT_READY_TIMEOUT_SECONDS EVAL_MODEL_STABILIZATION_SECONDS AIPERF_FAILED_REQUEST_THRESHOLD AIPERF_LIVE_FAILED_REQUEST_THRESHOLD AIPERF_TRACE_IDLE_GAP_CAP_SECONDS AIPERF_PYTHON_VERSION AIPERF_WARMUP_REQUESTS_PER_LANE AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES AGENTIC_WARMUP_GRACE_PERIOD AIPERF_USE_DYNAMO_CONV_AWARE_ROUTING AIPERF_HTTP_X_DYNAMO_SESSION_ID_FROM_CORRELATION_ID AIPERF_DYNAMO_SESSION_TIMEOUT_SECONDS AIPERF_EXPERIMENTAL_FAST AIPERF_UNSAFE_OVERRIDE ENABLE_AGENTX_POWER REQUIRE_POWER VLLM_ENGINE_READY_TIMEOUT_S SGLANG_TORCH_PROFILER_DIR VLLM_TORCH_PROFILER_DIR"
10 changes: 5 additions & 5 deletions configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1194,7 +1194,7 @@ minimaxm3-fp8-mi325x-vllm-agentic-mtp:
- { tp: 8, spec-decoding: mtp, kv-offloading: none, conc-list: [1, 2, 4, 8, 10, 12, 14, 16, 18] }

dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp:
image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260911
image: lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913
model: deepseek-ai/DeepSeek-V4-Pro-0813
model-prefix: dsv4
runner: cluster:mi355x-amds
Expand All @@ -1207,7 +1207,7 @@ dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp:
agentic-coding:
- dram-utilization: 0.80
search-space:
- spec-decoding: "mtp"
- spec-decoding: "draft_model"
conc-list: [ 4 ]
kv-offloading: none
prefill:
Expand All @@ -1226,7 +1226,7 @@ dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp:
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=3"
- spec-decoding: "mtp"
- spec-decoding: "draft_model"
conc-list: [ 16 ]
kv-offloading: none
prefill:
Expand All @@ -1245,7 +1245,7 @@ dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp:
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=3"
- spec-decoding: "mtp"
- spec-decoding: "draft_model"
conc-list: [ 32, 48 ]
kv-offloading: dram
kv-offload-backend: { name: hicache }
Expand All @@ -1266,7 +1266,7 @@ dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp:
additional-settings:
- "DECODE_NODES=1"
- "DECODE_MTP_SIZE=3"
- spec-decoding: "mtp"
- spec-decoding: "draft_model"
conc-list: [ 128, 192, 256 ]
kv-offloading: dram
kv-offload-backend: { name: umbp-linker }
Expand Down
9 changes: 9 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7923,3 +7923,12 @@
- "Bump image to lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260915."
- "Switch HiCache defaults to --hicache-io-backend kernel and --hicache-mem-layout page_first (from direct / page_first_direct). Ratio 1.5 and write_through are unchanged."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3118

- config-keys:
- dsv4-fp4-mi355x-sglang-disagg-agentic-hicache-mtp
scenario-type:
- agentic-coding
description:
- "Bump image to lmsysorg/sglang-rocm:v0.5.19-rocm720-mi35x-20260913."
- "Rerun the MI355X disaggregated MTP sweep with DeepSeek-V4-Pro-0813 + Dspark."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/3191
Loading