[VibhuAg]

Summary

This PR adds a new command-line tool, zarr-to-b2nd, for converting Zarr arrays to Blosc2 .b2nd files. I created a file for utility functions shared between the b2nd-to-zarr and zarr-to-b2nd tools. The new tool works by determining the shape of the input in terms of the chunks and shards (if they exist) used in the Zarr dataset and its compression settings. Then data is copied in batches, sized by --buffer-size. When shards are present in the Zarr dataset, shards become b2nd chunks and inner chunks become b2nd blocks. When shards are not present, Zarr chunks become b2nd chunks and the blocks are chosen by Blosc2.

Closes #721. I plan to write docs in a separate PR for #714.

Changes

• Shared helpers module in conversion_utils.py
• b2nd_to_zarr.py now uses the helper module with no behaviour change
• New zarr_to_b2nd command registered in pyproject.toml
• New tests for zarr_to_b2nd.py
• All options are validated before an existing output is deleted with --force

Compression handling

• Zarr compressed with Blosc -> same codec, level, and shuffle
• Zarr compressed with Zstd -> Level 0 -> clevel 5 and levels above 9 -> 9 since zstd goes up to level 22 and Blosc2 stops at 9
• Zarr compressed with Gzip -> Output gets zlib since Blosc2 has no gzip and zlib is the closest equivalent
• --codec / --clevel / --shuffle override all of the above
• If a Zarr Blosc array uses a codec Blosc2 lacks, the tool falls back to zstd. But if the user types an unknown codec it's an error.

Testing

• Added a new test file at tests/test_zarr_to_b2nd.py. 26 tests total. Converter tests pass, and so does the full suite.

I tested the utility on the 2-metre air temperature dataset from NOAA published as Zarr on AWS. It is a Zarr v2 dataset compressed with Blosc (lz4, level 9, shuffle) and carries attributes such as units: "K" and long_name.

I downloaded the dataset with the following commands:

mkdir -p TMP.zarr
B=https://hrrrzarr.s3.amazonaws.com/sfc/20210601/20210601_00z_anl.zarr/2m_above_ground/TMP/2m_above_ground/TMP
curl -sf -o TMP.zarr/.zarray "$B/.zarray"
curl -sf -o TMP.zarr/.zattrs "$B/.zattrs"
for i in $(seq 0 7); do for j in $(seq 0 11); do curl -sf -o "TMP.zarr/$i.$j" "$B/$i.$j"; done; done

I tested the conversion utility with the following command and got these results:

$ zarr-to-b2nd TMP.zarr --full-verify
Converting zarr -> b2nd... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ? 0:00:00 0:00:00
Performing full verification across all slices...
✓ Full element verification passed!

                       Conversion Summary
 Property            Value
 Source              /Users/vagrawal118/hrrr/TMP.zarr (1.16 MB)
 Destination         /Users/vagrawal118/hrrr/TMP.b2nd (1.15 MB)
 Shape / Dtype       (1059, 1799) / float16
 Chunks              (150, 150)
 Uncompressed Size   3.63 MB
 Source Compression  3.14x
 Blosc2 Compression  3.15x
 Conversion Speed    51.6 MB/s (0.07 s)
 Total Time          0.10 s

Not included

• Remote urls are not supported as input yet.

AI tool usage

I used AI to make edits of the code and tests after I wrote an initial version. I have read through the code and understand how it works.

[FrancescAlted]

I like this PR. It is well-designed, effectively eliminates duplicate code between b2nd_to_zarr.py and zarr_to_b2nd.py, and maps Zarr shards → b2nd chunks and Zarr inner chunks → b2nd blocks cleanly (great!). The test coverage is comprehensive too.

Some requests for improvements:

1. Add an zarr-to-blosc2 alias in pyproject.toml
2. Include Blocks in print_summary() (both in rich and text outputs)
3. Defensive attribute access on Zarr arrays: in Zarr v2 array objects (e.g. if opened with older Zarr versions or custom store wrappers), .shards does not exist and raises AttributeError. Similarly, Zarr v2 uses .compressor (singular) rather than .compressors (plural).
4. Please use type annotation on output_path in zarr_to_b2nd (like output_path: str | Path | None = None).
5. Warn on dropped attributes in verbose mode. Silently suppressing exceptions can lead to unexpected metadata loss (e.g. unstorable types). Suggestion: pass verbose: bool = False to copy_attrs() and emit a notification if an attribute cannot be stored.

Keep the good work!

[VibhuAg]

Thank you for your feedback @FrancescAlted.
I have addressed all comments apart from #3. My understanding is that pyproject.toml requires zarr>=3.0.9 so the path you described would never run. Instead I added a check in require_zarr in conversion_utils.py to ensure the zarr version number is not < 3.

I found three other bugs

1. zarr returns Fortran-ordered data for these arrays, and Blosc2 only accepts C-ordered (row-by-row) data. So now each batch now goes through np.asarray(src[s], order="C") in copy_batches. This only makes a copy when the data isn't already C-ordered, so b2nd-to-zarr is unaffected.
2. The verify code called np.array_equal(..., equal_nan=True), and equal_nan only works for types that can hold NaN. A new helper only compares with equal_nan on only for floats, complex numbers and dates/times. This bug appears to be present in b2nd-to-zarr and should also be fixed by the fix in the shared module.
3. In zarr_to_b2nd.py the converter now raises TypeError: Only fixed-size dtypes can be converted to .b2nd, got StringDType() for variable-length strings. That happens before anything is deleted.

I also added the change for 5 to b2nd-to-zarr's copy_metadata.

[FrancescAlted]

Do a rebase please.

[FrancescAlted]

Thanks for addressing these, and for catching the additional bugs!

You’re right about point 3: I conflated the Zarr package version with the on-disk format version. With zarr>=3.0.9 as our supported dependency, there’s no need to support the Zarr 2 Python API. Reading both v2 and v3 datasets through Zarr 3 is what matters. A clear version check is fine.

The C-contiguous batch conversion, dtype-aware verification, and early rejection of variable-length strings all sound sensible. Thanks also for applying the metadata warnings to both converters.

Please ensure the regression tests cover:

• A Fortran-ordered source.
• Verification of nonnumeric data and NaN/NaT values where supported.
• Unsupported dtypes leaving an existing destination untouched, even with --force.
• Both Zarr v2 and v3 datasets using the supported Zarr 3 package.

After the rebase, I’ll take another look. Thanks for the careful work!