Skip to content

fix: preserve generated vision token text embeddings - #207

Merged
tastelikefeet merged 1 commit into
modelscope:mainfrom
dingzhiqiang:fix/generated-vision-token-provenance
Sep 25, 2026
Merged

tastelikefeet merged 1 commit into
modelscope:mainfrom
dingzhiqiang:fix/generated-vision-token-provenance

Conversation

@dingzhiqiang

Copy link
Copy Markdown
Contributor

When an autoregressive multimodal sequence contains a generated image/video special token, matching visual embedding positions by token ID alone also selects that generated token. This can misalign visual features or fail during embedding scatter.

Use optional mm_token_type_ids (0=text, 1=image, 2=video) to select real visual input positions while preserving generated special-token IDs, text embeddings, and gradients. Validate shape, modality, placeholder IDs, payload presence, and feature counts. Callers without modality types retain token-ID-based selection, with explicit feature-count validation.

Validation:

  • CPU regression coverage for image/video embedding placement and gradients, legacy callers, invalid metadata/payloads, and image-only configurations.
  • 55 tests passed in the combined bridge, AReaL mRoPE, and packed-input regression suite.
  • Full bridge pre-commit checks passed.
  • A downstream multimodal RL run completed its next optimizer step and saved an optimizer-inclusive recovery checkpoint with this change.

Scope: callers must supply correct input-aligned modality types to distinguish generated special tokens. This change only covers the shared HuggingFaceVit embedding helper; it does not modify generation or model-specific mRoPE handling.

@hjh0119 hjh0119 self-assigned this Sep 23, 2026
@tastelikefeet
tastelikefeet merged commit 1eaf3e4 into modelscope:main Sep 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants