Skip to content

fix(sampling): use neutral temperature and accept top_k=-1 - #5081

Merged
silencelamb merged 1 commit into
InternLM:mainfrom
silencelamb:fix/sampling-defaults-compat
Nov 2, 2026
Merged

silencelamb merged 1 commit into
InternLM:mainfrom
silencelamb:fix/sampling-defaults-compat

Conversation

@silencelamb

Copy link
Copy Markdown
Collaborator

Motivation

Requests that omit temperature currently fall back to 0.8, which sharpens the model distribution, and clients using top_k=-1 to disable top-k filtering are rejected. Use a neutral temperature fallback and accept the disabling convention used by vLLM and SGLang.

This complements #5079 and can be reviewed and merged independently.

Modification

  • Change the temperature fallback to 1.0 in GenerationConfig, the PyTorch sampling parameters, and the CLI argument helper.
  • Accept top_k=-1 alongside 0 in Chat Completions, Completions, Responses, Anthropic Messages, and /generate. Normalize -1 to 0 in the shared GenerationConfig before either inference backend receives it; continue rejecting values below -1.
  • Cover request/model precedence, model config loading, API validation and adapter paths, Python configuration, and zero-temperature greedy conversion with regression tests.
  • Document the sampling behavior and migration in English and Chinese.

BC-breaking

Sampled outputs may change when neither the request nor the effective model/server defaults supplies a temperature. Set temperature=0.8 explicitly to retain the previous fallback behavior. Explicit request/model temperatures retain their precedence; greedy decoding is unaffected. Accepting top_k=-1 is an additive compatibility change.

Validation

  • Focused configuration, message, Chat/Completion, Responses, Anthropic Messages, and /generate tests: 211 passed, 4 deselected with Transformers 5.16.0.
  • The same suite with Transformers 4.57.6: 211 passed, 4 deselected. The four deselected cases require unrelated tokenizer downloads.
  • All applicable pre-commit hooks passed on the changed files; git diff --check passed.
  • Rendered both new documentation sections with Sphinx/MyST, treating warnings as errors.
  • Tests use CPU and fake serving engines; no GPU end-to-end model inference was run.

@RunningLeon RunningLeon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@silencelamb
silencelamb merged commit 5396984 into InternLM:main Nov 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants