Repository navigation
NWB stored in a Zarr store, not HDF5 (0.11.0) - #11
Merged
Merged
Conversation
hdmf-zarr writes the same NWB model into a Zarr directory. The root group's attributes carry nwb_version, namespace and neurodata_type, and a store written with consolidated metadata keeps the whole tree in one document, so a single bounded read describes the session. Both container versions: Zarr v3 from the root zarr.json, Zarr v2 from the consolidated .zmetadata, which is now a registered sidecar name tried before .zattrs. .zmetadata is claimed only for an NWB store, so every other consolidated document declines and the directory dispatch falls through to the per-node sidecars as before. What this cannot report, and the HDF5 flavour can, is the scalar strings: identifier, session description, start time, institution, lab, subject. In HDF5 those sit in the object header; in Zarr they are chunk data behind a compressor this library does not carry. They are absent rather than guessed, and stringValuesUnread names the paths that hold them. Verified against two real files, both committed whole: the consolidated metadata of a DANDI 000719 asset (Zarr v2, 3170 keys, 139 acquisition series, fetched from the DANDI S3 bucket) and the root of a store written by hdmf-zarr 0.14.0 with pynwb 4.2.0 (Zarr v3).
Zarr writers emit bare NaN, Infinity and -Infinity tokens, which Python's json reads as floats and strict JSON rejects. An NWB TimeSeries writes "resolution": NaN whenever the resolution is unspecified, 278 times over in the committed DANDI store. No reported field carries one today, so the fixture is unchanged, but an attribute this record does carry could, and the Go port turns the same token into null. Both now agree on absent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
hdmf-zarrwrites the same NWB model into a Zarr directory instead of an HDF5 file, and DANDI hosts assets in that form. Until now those decoded as a barezarr-group.Both container versions are read, from one metadata document each:
zarr.json, whoseconsolidated_metadataholds the whole tree as a flat path map.zmetadata, keyed<path>/.zattrs,/.zgroup,/.zarray, folded into the same node mapBoth fixtures are committed whole rather than trimmed. The DANDI one is 1.4 MB of JSON that packs to about 52 KB.
Worth a reviewer's attention:
hdf5.pyreads the identifier, session description and subject straight out of metadata. In Zarr those are chunk data, in the sample behind zstd, and no chunk is ever opened here, the same rule the rest ofzarr.pyfollows. So those keys are absent rather than null, because null already means "not present" in the HDF5 record, andstringValuesUnreadnames the paths a reader would have to open.test_string_values_are_named_not_guessedasserts the absence as deliberately as the other tests assert values..zmetadatais a new registered sidecar name, tried before.zattrsbecausesources._decode_directorywalks the names in registration order. It is claimed only for an NWB store; any other consolidated document declines and the dispatch falls through exactly as before, whichtest_non_nwb_zarr_is_untouchedpins..zattrs, which still names and versions the file.specifications/<namespace>/<version>gives the schema version each namespace was cached at, sospecificationsis{"core": ["2.11.0"]}rather than a bare name list.596 passed, 2 skipped, green on Python 3.9 and 3.12.