Skip to content

fix: avoid existing chunk duplication in multi-node optimize(mode="ap… - #867

Open
hillhack wants to merge 1 commit into
Lightning-AI:mainfrom
hillhack:fix/multi-node-append-duplication
Open

fix: avoid existing chunk duplication in multi-node optimize(mode="ap…#867
hillhack wants to merge 1 commit into
Lightning-AI:mainfrom
hillhack:fix/multi-node-append-duplication

Conversation

@hillhack

@hillhack hillhack commented Aug 9, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes #865

Fixes duplicate existing chunks when optimize(..., mode="append") is used with multiple nodes.

Previously, each node included the existing dataset chunks in its intermediate index. During the final merge, these chunks were combined multiple times, resulting in duplication.

Changes

  • DataProcessor._done(): Excludes existing_index from intermediate node-level merges when using multiple nodes, while preserving the existing behavior for single-node append.
  • DataProcessor._upload_index(): Adds existing_index only once during the final global merge.
  • Tests: Added test_multinode_done_append_no_duplication to verify that existing chunks appear exactly once in the final index.

Expected result

For an existing dataset [A, B] and new chunks [C, D] produced by two nodes:

Before: [A, B, C, A, B, D]
After:  [A, B, C, D]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Multi-node optimize(mode="append") duplicates existing dataset chunks N times in index.json

1 participant