Add scale_offset array->array codec - #87
Open
konstibob wants to merge 4 commits into
Open
Conversation
Implements the zarr-extensions scale_offset codec: encode applies (in - offset) * scale, decode applies (in / scale) + offset, in the input data type's arithmetic. Integer arithmetic is exact with a representability check at each step (out-of-range or non-exact division is an error); floating-point uses native float/double ops. Supports the 10 real-number data types this library models. The data type is unchanged; only the fill value is transformed and propagated downstream. Registered in the v3 CodecRegistry and exposed via CodecBuilder.withScaleOffset(...). Tests in the new dev.zarr.zarrjava.codec test package. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Contributor
Author
|
Here is a link for the specification https://github.com/zarr-developers/zarr-extensions/tree/main/codecs/scale_offset |
- Do not fail the codec pipeline when the fill value is not representable after the encode transformation. The fill value is metadata, not stored data, so an unsigned array with a fill value below the offset (e.g. uint16 with fill_value 0 and offset 1000) previously could not be created or opened at all. It is now kept untransformed in that case. - Reject a scale of 0 when the configuration is parsed. Encoding with a scale of 0 mapped every value to 0 without an error, while decoding either threw or produced NaN, so the data was unrecoverable. - Use long arithmetic with Math.*Exact and explicit bound checks for every integral type except uint64, instead of allocating several BigIntegers per array element, and skip the element-by-element rebuild entirely when neither offset nor scale is configured. Co-Authored-By: Claude Opus 5 <[email protected]>
normanrz
approved these changes
Sep 4, 2026
normanrz
reviewed
Sep 4, 2026
normanrz
left a comment
normanrz
left a comment
Member
There was a problem hiding this comment.
Should also be added to the userguide.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implemented the scale_offset array→array codec. Applies (in - offset) * scale on encode / (in / scale) + offset on decode, keeping dtype and shape unchanged.