[zzwwrr]

Summary

Add Intern-Decision candidate scoring to LMDeploy's PyTorch backend. The service scores trained answer symbols from prefill logits without generating tokens.

• Support Intern-Decision-0.8B, 2B and 4B through a shared parser and engine.
• POST /v1/decisions: supported OpenAI Decisions subset, with input, a questions array, answers and token usage.
• POST /v1/systemone: Systemone state, questions object and optional image assets, including JSON and multipart uploads.
• Support text, images and JSON evidence, including OpenAI question content blocks, with request-local image reuse.
• Resolve calibration in order: explicit --decision-temperature → checkpoint metadata → exact known model-path name → startup error. Standalone answer decoding requires an explicit temperature.
• Validate candidate token IDs against the model vocabulary before submitting GPU work.
• Bound decision image HTTP downloads to 10 MiB and retain the Systemone route's 32 MiB request-body limit. Other media callers retain their existing download behavior unless they supply a budget.

Start with --task intern-decision and optionally --model-name. This task dedicates the server to decision inference: /v1/chat/completions, /v1/completions, /v1/messages, /v1/responses, /generate, /pooling, and /get_ppl reject inference POST requests with 400 before they enter the engine. Run ordinary generation on a separate server. Model listing, tokenization and management endpoints remain available.

The endpoints have distinct schemas; there is no /v1/jev route. Python Pipeline has no decision entry point.

The upstream --max-concurrent-requests limiter and ordinary-server middleware ordering are retained. Existing session/generator cleanup remains, but prompt cancellation on client disconnect is an unresolved limitation described below.

A100 text quality results

Single A100-SXM4-80GB, BF16/TP1, prefix cache OFF, concurrency/max batch 8, session/prefill limit 8192, piecewise CUDA graphs. Each size completed 10,751 text requests / 12,351 decision fields with zero failed requests. The harness calls the production Systemone scoring handler directly, without HTTP transport.

Seven-suite unweighted mean accuracy:

Model	Published official	LMDeploy A100	Difference (pp)
0.8B	79.38%	79.1617%	-0.2183
2B	84.68%	84.9129%	+0.2329
4B	90.02%	89.8408%	-0.1792

Per-suite LMDeploy accuracy (%):

Model	Jevbench Easy	Original	Hard	Typed Decision	ToolACE	AG News	WildJailbreak
0.8B	97.9167	79.1667	53.1532	77.4500	94.5161	88.6711	63.2579
2B	100.0000	84.7222	65.7658	79.3000	96.4516	89.9605	78.1900
4B	100.0000	98.6111	72.9730	80.3000	96.4516	90.7763	89.7738

Calibration on Jevbench-Hard only (111 fields); lower is better:

Model	Official Brier	LMDeploy Brier	Official ECE	LMDeploy ECE
0.8B	0.529519	0.529083	0.065710	0.091346
2B	0.437256	0.437578	0.100208	0.071510
4B	0.346762	0.347182	0.065267	0.081564

These are not the separate 96-case distribution-pilot metrics. Comparisons use the fixed official results; they do not establish numerical equivalence or isolate backend errors. Performance/speedup comparisons are outside this update.

Published evidence and reproduction instructions include all raw responses, suite metrics, metadata, source patch, evaluated source/harness, hashes, and GPU-free audit/replay commands.

Before publication, all 21 suite accuracies and three Hard Brier/ECE pairs were recomputed from saved responses with the pinned official scorer and matched. All 753 evaluated production Python file hashes were checked against the preserved source.

Validation

• Latest local regression run: 527 passed, 4 skipped.
• The run covers decision schemas/parser/HTTP conversion, startup and calibration, task isolation, image preparation/download budgets, error handling, generation configuration, existing /generate logprob behavior, ordinary Chat Completions and Anthropic endpoints.
• The real Intern-Decision-4B tokenizer/processor preprocessing cases ran with DECISION_TEST_CHECKPOINT set. These are CPU preprocessing checks and do not load model weights. The four skipped cases require an OpenAI SDK exposing Decisions; that SDK coverage remains unverified.
• Ruff checks on the edited Python files and Git diff checks passed. These are local validation results, not a claim that upstream CI has passed.
• No current-head full-weight accuracy, throughput/latency parity, or GPU cancellation validation is claimed. The historical A100 text run above bypasses HTTP transport. Additional archived H200 4B text/image results and their configuration limits are documented in the versioned guide.

Regression selection, in the configured test environment:

DECISION_TEST_CHECKPOINT=/path/to/Intern-Decision-4B python -m pytest \
  tests/test_lmdeploy/test_decision*.py \
  tests/test_lmdeploy/test_vl/test_media_download.py \
  tests/test_lmdeploy/serve/test_server_errors.py \
  tests/test_lmdeploy/serve/test_generation_config.py \
  tests/test_lmdeploy/serve/test_generate_logprobs.py \
  tests/test_lmdeploy/serve/openai/chat_completions \
  tests/test_lmdeploy/serve/anthropic/test_endpoints.py \
  -p no:cacheprovider -q

Scope and unresolved issues

• PyTorch hybrid engine only; speculative decoding and disaggregated roles are rejected. Python Pipeline and TurboMind decision entry points are unsupported.
• OpenAI Decisions is a subset: user messages only, up to 16 questions and eight image occurrences, image detail auto/null, no tool records or prompt-cache controls. Systemone does not support box.
• Local choice/score confidence is the maximum calibrated candidate probability. Output-token usage counts the assistant skeleton; no tokens are generated. cache_write_tokens=0 is an unmeasured compatibility placeholder.
• Prompt client-disconnect cancellation remains unresolved, with or without --max-concurrent-requests. The current endpoint checks disconnection only after the scoring generator produces a result; preprocessing, handle acquisition and scoring are not monitored concurrently. Disconnected requests may continue processing, and the upstream limiter does not remove queued requests on disconnect. Cancellation and disconnect-aware admission are deferred to separate work; this PR does not claim to resolve the disconnect review finding.
• Already-running image preparation threads are not forcibly interrupted by coroutine cancellation.
• The engine does not support mixed ordinary-generation/decision scoring batches. This PR prevents that combination through the decision-only HTTP task gate; it does not implement mixed-batch scheduling.
• Explicit raw_logprobs scoring retains a known BF16 normalization-rounding risk; the normal raw_logits path avoids that normalization.
• With DP > 1, use rank endpoints directly; the built-in DP proxy does not provide decision routing.

[Copilot]

🟡 Changes recommended

Critical issues remain: mixed scoring and generation batches prematurely stop generation requests, and unchecked out-of-vocabulary candidate IDs can trigger a CUDA assertion that disables the shared worker.

2 open issues

• Reject out-of-vocabulary candidate IDs before CUDA gather · New
• Mixed prefill batches incorrectly stop generation requests · New

What changed in this PR

Adds Intern-Decision scoring to LMDeploy’s PyTorch backend, using prefill candidate scores rather than token generation and reusing existing text and multimodal processing.

Changes:

• Adds requested-token input logprobs and decision prompt compilation, scoring, and calibration.
• Exposes native and OpenAI-compatible requests through /v1/decisions, with /v1/jev as a native alias.
• Adds startup validation, regression tests, bilingual documentation, and an HTTP example.

File	Description
tests/​test_lmdeploy/​test_decision.py	Tests native validation, scoring, image handling, and cleanup.
tests/​test_lmdeploy/​test_decision_startup.py	Tests task selection and startup validation.
tests/​test_lmdeploy/​test_decision_service.py	Tests native HTTP routes and service errors.
tests/​test_lmdeploy/​test_decision_preprocessing.py	Checks checkpoint preprocessing and protocol equivalence.
tests/​test_lmdeploy/​test_decision_openai.py	Tests OpenAI adaptation, HTTP behavior, and SDK compatibility.
tests/​test_lmdeploy/​test_decision_metadata.py	Tests safe temperature metadata parsing and fallback.
tests/​pytorch/​engine/​test_prefill_logprobs.py	Tests per-request candidate-column filtering.
tests/​pytorch/​engine/​test_model_agent.py	Tests gathering requested candidate scores.
lmdeploy/​serve/​utils/​server_utils.py	Applies sleeping-engine handling to decision routes.
lmdeploy/​serve/​parsers/​decision_parser.py	Adapts requests and decodes candidate scores.
lmdeploy/​serve/​openai/​endpoints/​router.py	Registers decision endpoints.
lmdeploy/​serve/​openai/​endpoints/​decisions.py	Runs scoring requests and manages session cleanup.
lmdeploy/​serve/​openai/​decision_protocol.py	Defines native and OpenAI-compatible schemas.
lmdeploy/​serve/​openai/​api_server.py	Initializes and validates decision task context.
lmdeploy/​pytorch/​strategies/​ar/​sampling.py	Builds the batch union of requested candidate IDs.
lmdeploy/​pytorch/​messages.py	Carries candidate IDs into sampling parameters.
lmdeploy/​pytorch/​engine/​model_agent/​agent.py	Gathers candidate scores from prefill logits.
lmdeploy/​pytorch/​engine/​logits_process.py	Adds candidate IDs to sampling inputs.
lmdeploy/​pytorch/​engine/​engine_loop.py	Filters candidate columns for each request.
lmdeploy/​messages.py	Exposes requested-token input-logprob configuration.
lmdeploy/​intern_decision_encoding.py	Compiles prompts and calibrates decision answers.
lmdeploy/​cli/​serve.py	Adds the task flag and selects PyTorch.
examples/​decision/​decision.py	Demonstrates HTTP scoring with optional images.
docs/​zh_cn/​llm/​intern_decision.md	Documents usage, protocols, and limitations in Chinese.
docs/​en/​llm/​intern_decision.md	Documents usage, protocols, and limitations in English.

_{Review effort: Balanced}

[zzwwrr]

fixed

[zzwwrr]

是否开放决策模型的自回归生成能力仍需评估。
当前打分路径沿用上游的 batch 级处理逻辑，会跳过采样并结束整批请求；
若同时支持生成与决策，需要隔离两类 prefill，或补齐混批处理。

[Copilot]

🟡 Changes recommended

Client disconnection is checked only after scoring completes, leaving preprocessing and inference running after disconnects or timeouts and allowing retries to accumulate duplicate work.

3 open issues

• Mixed prefill batches incorrectly stop generation requests
• Reject out-of-vocabulary candidate IDs before CUDA gather
• Client disconnects do not cancel preprocessing or inference · New

_{Review effort: Balanced}

[silencelamb]

Confirmed on 3df76b5b with an asynchronous regression harness using the real _score_request and session manager, a request whose is_disconnected() returns true, and a fake engine blocked before its first result:

While inference is blocked: disconnect_checks=0, task_completed=False, live_sessions=1
After the engine is released: disconnect_checks=1, live_sessions=0

Since this endpoint emits only a final result, the current check cannot interrupt preprocessing, waiting for an engine handle, or scoring. A timed-out caller's work continues until completion, and retries can add duplicate work.

Please monitor disconnects concurrently with request processing and cancel/abort the request through the existing session and generator lifecycle. The cleanup helper runs only when this task exits; it does not itself detect a disconnected client. Add regression cases for disconnects during preprocessing and before the first scoring output, verifying that the request/session is released promptly.

[zzwwrr]

We’ve added concurrent disconnect monitoring during preprocessing and inference. When the client disconnects, the scoring task is cancelled, and we wait for the existing session and generator cleanup to finish.

Cancellation within image preparation is not yet implemented: we stop awaiting the result, but work already running in a background thread may continue. We can consider adding finer-grained cancellation for image downloading and processing as a follow-up, though it would require additional implementation and testing.

[silencelamb]

Current-head status at bbe8cf1c: this is still the same unresolved cancellation issue, and the updated PR description/docs now explicitly defer it. The head no longer contains the concurrent watcher from e312b02f; the only disconnect check is again inside the final-result loop (current code).

A real-HTTP controlled-backend probe on both /v1/decisions and /v1/systemone, 450 ms after disconnect during preprocessing, showed disconnect_checks=0, one live session and no cancellation, both with and without the limiter. With --max-concurrent-requests=1, the only permit remained occupied until work exited. This is a serving cancellation/resource-management limitation to track in the follow-up; it does not invalidate the completed text-quality evaluation or establish a performance regression for requests that finish normally.

[RunningLeon]

can you update the docstring for this newly added argument?

[zzwwrr]

fixed

[silencelamb]

Reviewed at 3df76b5b. Please address the inline calibration and documentation findings and the disconnect/cancellation issue, and update the PR description and validation evidence below.

The PR description should match the implemented endpoints.

The Summary still says native and OpenAI requests share /v1/decisions, with /v1/jev as an alias. The actual route registrations are:

Endpoint	Request/response contract
POST /v1/decisions	OpenAI Decisions: model, input, and a questions array; predicate/choice/score answers
POST /v1/systemone	SystemOne: state, a questions object, and optional assets; noul/choice/score answers; JSON and multipart input

There is no /v1/jev route, and a native request sent to /v1/decisions returns 422. Please update the Summary accordingly and state support for 0.8B, 2B, and 4B.

Keep the supported-subset wording and describe the limits against the OpenAI Decisions reference and SystemOne reference. In particular, the current adapter accepts only user messages, at most 16 questions and 62 choice options, and only auto image detail; SystemOne box questions are unsupported. The documented local confidence and cache-write placeholder semantics should remain explicit.

Please include measured quality and latency comparisons in the PR.

Comparing the two LMDeploy protocol paths on 96 text inputs establishes adapter consistency, but does not establish parity with the reference implementation. The published pilot inputs are all text-only choice requests, so they also do not replace the seven-suite accuracy evaluation.

For reference, the official published results are:

Model	Seven-suite average accuracy	Jevbench-Hard Brier / ECE	RTX 4090 latency: mean / P50 / P95 (ms)
Intern-Decision-0.8B	79.38%	0.529519 / 0.065710	33.98 / 33.44 / 37.50
Intern-Decision-2B	84.68%	0.437256 / 0.100208	33.28 / 33.15 / 33.55
Intern-Decision-4B	90.02%	0.346762 / 0.065267	44.16 / 44.03 / 44.60

These are upstream baselines, not LMDeploy measurements. Please add LMDeploy results and deltas for all three sizes, with reproducible commands and raw results. Report the seven individual suite accuracies as well as their unweighted mean.

The separate 4B known-distribution pilot reports Brier 0.550 / ECE 0.089 after calibration. Keep those separate from Jevbench-Hard metrics. The evaluation guide specifies the datasets, fixed temperatures, checkpoint verification, and metric definitions. Its reported quality tables use XTuner; the latency table uses native HF + SDPA, BF16, one RTX 4090, and 289 input tokens with three questions. Record the LMDeploy commit, checkpoint/revision, temperature, dtype, hardware, dependency versions, and workload/concurrency so any parity claim is assessable. Backend floating-point differences should be reported as measured deltas.

Validation performed for this review: 453 existing tests plus 27 real-checkpoint preprocessing checks across the three sizes passed. Both protocol adapters produced the same compiled prompts as upstream for all 96 pilot inputs on each size. Four OpenAI SDK tests were skipped because the installed SDK (3.13.0) does not expose Decisions; please document the SDK version used for the claimed SDK validation and ensure that coverage runs in an appropriate CI environment. No full-weight accuracy or latency benchmark was run during this review, so these checks do not establish model-quality or performance parity.

[silencelamb]

[P1] Resolve calibration from the checkpoint actually loaded by the engine

For a normal launch such as lmdeploy serve api_server internlm/Intern-Decision-0.8B --task intern-decision, this call still passes the original model ID. The PyTorch engine resolves/downloads that ID to a local snapshot, but resolve_temperature reads Path(model_path) / "inference.py" relative to the working directory. It therefore warns and falls back to the 4B temperature even when the downloaded checkpoint contains the correct metadata.

The released temperatures are 2.747760550703 for 0.8B, 2.100509348278 for 2B, and 1.99241824 for 4B; see the official checkpoint-specific presets. I reproduced the local-path/model-ID difference with the official metadata. For synthetic candidate logits [0, 4], the 0.8B P(yes) changes from 0.810879 to 0.881594 under the wrong fallback. This affects probability, confidence, score, and calibration metrics even when the selected label is unchanged.

Please read metadata from the resolved checkpoint/revision used for the loaded weights, and use an explicit configuration or error for unknown calibration instead of assigning the 4B temperature to every model. Add a startup regression covering remote IDs and local paths for all three sizes.

[zzwwrr]

fixed. Only cases where the temperature can be determined, such as through explicit configuration, are now supported. Otherwise, an error is raised.

[silencelamb]

Verified on bbe8cf1c: remote checkpoints use the resolved snapshot for metadata, and explicit --decision-temperature is validated and takes precedence. Metadata takes precedence over the documented exact model-path fallback; unknown paths without calibration fail. Tests cover remote/local paths and all three model sizes, including invalid overrides. Real checkpoint metadata still yields the expected 0.8B / 2B / 4B temperatures. The original wrong-4B-fallback finding is addressed.

[silencelamb]

[P2] Document the complete three-model support matrix and expose this page in the docs

Both language versions introduce only 0.8B; 4B appears in a later example, and 2B is absent. Please explicitly list Intern-Decision-0.8B, Intern-Decision-2B, and Intern-Decision-4B, including their calibration temperatures, PyTorch-only requirement, supported protocols, and launch examples or a clearly parameterized command.

The shared Qwen3.5 implementation does not appear to need separate size-specific model classes: the real tokenizer/processor preprocessing tests passed for all three checkpoints. Please distinguish that coverage from full-weight inference and benchmark validation.

Also add this page to both English and Chinese toctrees, and add Intern-Decision to the corresponding English and Chinese model-support documentation with the correct backend restriction. Currently the new pages have no navigation entry and the support tables do not identify this task.

[zzwwrr]

fixed

[silencelamb]

Verified on bbe8cf1c: English and Chinese guides, navigation, and PyTorch support tables cover all three sizes and the supported protocols. I also independently replayed the newly published A100 responses through the pinned official scorer: all 21 suite accuracies and all three Hard Brier/ECE pairs matched. This original coverage finding is addressed. The new review has separate inline follow-ups for the H200 evidence and the broken reference permalink.

[silencelamb]

Withdrawn pending the author's latest push. This follow-up review was based on the previously pushed commit (e312b02f), before the author's latest changes were available. I will re-review after those changes are pushed.

[silencelamb]

Re-reviewed bbe8cf1c and the updated PR description. The three-model documentation, calibration selection, and distinct OpenAI Decisions / Systemone routes address those earlier findings. I found no new model/protocol correctness issue in the incremental changes reviewed.

The historical A100 quality evidence is now independently checkable. I ran the artifact audit and official-scorer replay: all hashes/coverage checks passed, all 21 suite accuracies matched, and all three Jevbench-Hard Brier/ECE pairs matched. Each model has 10,751 requests / 12,351 fields. The recomputed seven-suite means are:

Model	LMDeploy accuracy	Difference from published official mean
0.8B	79.1617%	-0.2183 pp
2B	84.9129%	+0.2329 pp
4B	89.8408%	-0.1792 pp

This supports close historical average accuracy. Calibration metrics still have the explicitly reported differences; these are not identical distributions. The evaluated source remains cda4c3b5, so this is not a full-weight rerun of the current head or a latency/throughput comparison.

Two documentation follow-ups remain inline: publish/link the H200 run provenance and raw evidence, and fix the official-results permalink, which currently uses a blob SHA as a commit revision.

The updated description correctly defers prompt disconnect cancellation. My current-head real-HTTP probe confirms it remains absent both with and without the concurrency limiter. I will keep that in the existing cancellation thread as a separate serving issue; it does not invalidate the completed quality evaluation or imply a regression in normally completed requests' benchmark results.

Validation at this head: 530 regression tests plus 27 real-checkpoint preprocessing cases passed. The latter cover 0.8B, 2B, and 4B. Both protocol adapters also matched upstream compiled prompts on all 96 pilot inputs per size. Four SDK-specific tests remain unverified because the installed SDK lacks Decisions types. No new full-weight inference benchmark was run for this review.

[silencelamb]

[P2] Link the H200 run evidence and identify the evaluated LMDeploy source

The new H200 tables are used to substantiate full-weight text/image support, but this section does not identify the LMDeploy commit or link the archived responses, launchers, and scoring artifacts. It identifies the model and reference/adapter revisions, while explicitly saying that launcher overrides and tokenizer alignment are part of the evaluated setup. Without the LMDeploy source revision and those wrappers, a reviewer cannot determine which implementation produced these results or reproduce them.

Please publish a versioned H200 evidence bundle and link it from both language versions. Include the evaluated LMDeploy revision/patch, launcher changes, dataset revisions and preparation recipe (including the 57 corrected POPE questions and 183 resized images), raw predictions, and scoring commands. The existing A100 bundle contains the A100 text runs; it does not supply the provenance for these H200/image tables. The A100 audit/replay already passes, so this request is specifically about the newly added H200 claims.

[silencelamb]

[P3] Use a commit SHA for the official-results permalink

This link returns 404: GitHub reports No commit found for the ref c7c2535c29830cb69495c8e66a598303ac2cdcf7. That hash is the README's blob object, not a commit.

The exact file is available at this valid pinned README link. I verified that 565350244af860e1c276768cd80bdd9b8be93913:README.md resolves to blob c7c2535c29830cb69495c8e66a598303ac2cdcf7. Please replace the revision in the English page, Chinese page, PR description, and evidence README link so readers can inspect the baseline being compared.

[yidingcheng0206]

[P2] When candidate scores are taken from a tensor that has undergone BF16 log_softmax, small differences may be rounded away. In a minimal CPU reproduction, candidate logits [1.0, 1.0078125] become [-19, -19], and the choice changes from yes to no. Subsequent conversion to FP32 by the parser can no longer recover this. The default raw_logits path is not affected by this issue; it is recommended to retain the original candidate logits, use FP32 normalization, or temporarily prohibit the decision task from using this mode. The author has disclosed the risk, but the code still allows it to be triggered.

[caikun-pjlab]

This change affects more than Intern-Decision: forcing FP32 in the shared Qwen3_5Attention also changes Q/K norm weights for standard Qwen3.5, Qwen3.5 MoE, InternS2 Preview, MetaMoe, InternS2 Mobius, and Qwen3.5 MTP.

Previously, these parameters followed LMDeploy’s resolved model dtype, which defaults to the Hugging Face configuration but can be overridden by the engine settings. Checkpoint tensor dtypes are not preserved automatically.

Could we clarify the intended scope? If this is specific to Intern-Decision, please restrict the FP32 policy accordingly. If it is a general Qwen3.5 precision fix, please check the derived models’ constructors and weight loaders as well: they override those paths, so changing this file alone does not consistently apply FP32 storage and FP32 weight + 1 across the family. GPU numerical and performance validation would also help establish the impact on existing models.

[yidingcheng0206]

create_decision_context()--rejects speculative decoding and non-Hybrid roles only here (decision_parser.py:66–69), but line 401 has already constructed the full engine. These configuration restrictions can be determined before initialization; currently, the code may first download models, create processes, and allocate GPU resources, and only then report a parameter error. Please move the configuration capability checks before model download/engine construction; read calibration after the checkpoint path is resolved and before engine construction; and keep checks that depend on the actual tokenizer/processor after initialization. Please add invalid-configuration tests that assert the engine builder was not called, rather than only that it can close after construction.

[yidingcheng0206]

Here, .encode('ascii') is called directly on the client-provided Content-Type, while only CR/LF has been checked beforehand. An illegal multipart request header containing non-ASCII parameters will trigger a UnicodeEncodeError, and because _SystemoneRoute only catches AssetError and read timeouts, it enters the server 500 path. This has been reproduced through the real route; for example, when the boundary parameter contains byte 0xFF, the last frame of the exception is exactly here. Please validate this at the MIME parsing boundary and convert this expected format error into an AssetError, returning a clear 4xx; add an HTTP regression test to confirm that no uncaught exception is thrown and that the engine is not entered.