List group members level by level instead of enumerating all keys - #92
Open
konstibob wants to merge 1 commit into
Open
List group members level by level instead of enumerating all keys#92konstibob wants to merge 1 commit into
konstibob wants to merge 1 commit into
Conversation
Group.list() called storeHandle.list(), which is fully recursive and returns every key below the group, including all chunk keys. Listing a group with 1M chunks on S3 cost ~1000 paginated requests to find a handful of zarr.json files. Walk the hierarchy with listChildren() instead, one level at a time, and never descend into arrays. The implementation moves to core.Group, so v2 and v3 share it, and adds members() for the immediate children only. Also: - S3Store.listChildren() used a single listObjectsV2 call, silently truncating groups with more than 1000 children. Use the paginator. - FilesystemStore.get() threw instead of returning null when a path component is a file rather than a directory. Probing a plain file next to a group's nodes is normal now, and such a key holds no data. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
normanrz
requested changes
Sep 4, 2026
| } | ||
|
|
||
| /** | ||
| * Lists all descendants (arrays and groups) of this group, at any depth. |
Member
There was a problem hiding this comment.
Suggested change
| * Lists all descendants (arrays and groups) of this group, at any depth. | |
| * Recursively lists all descendants (arrays and groups) of this group, at any depth. |
| * Lists the keys directly below {@code prefix} that may hold a child node, relative to this | ||
| * group. | ||
| */ | ||
| private List<String[]> childKeys(String[] prefix) { |
Member
There was a problem hiding this comment.
Suggested change
| private List<String[]> childKeys(String[] prefix) { | |
| private List<String[]> descendantKeys(String[] prefix) { |
| * Opens the node at {@code key}, or returns null if there is no node there. | ||
| */ | ||
| @Nullable | ||
| private Node openChild(String[] key) { |
Member
There was a problem hiding this comment.
Suggested change
| private Node openChild(String[] key) { | |
| private Node openDescendant(String[] key) { |
| /** | ||
| * Keys that hold metadata of the group itself and never point at a child node. | ||
| */ | ||
| private static final Set<String> METADATA_KEYS = Collections.unmodifiableSet( |
Member
There was a problem hiding this comment.
This breaks the v2 vs. v3 abstraction. Could this be delegated to the subclasses?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Group.list() called storeHandle.list(), which is fully recursive and returns every key below the group, including all chunk keys. Listing a group with 1M chunks on S3 cost ~1000 paginated requests to find a handful of zarr.json files.
Walk the hierarchy with listChildren() instead, one level at a time, and never descend into arrays. The implementation moves to core.Group, so v2 and v3 share it, and adds members() for the immediate children only.
Also: