Skip to content

Allow for descriptions in test cases #46

Open

Description

@clbarnes

At least some of the test cases could benefit from more of some free text with an explanation of either the feature they're exercising or why they're invalid.

Previously, the parse_attributes cases had a _conformance object which had some obsolete information in. The coordinate transformation tests in https://github.com/clbarnes/ome_zarr_transformations_conformance have a sidecar file containing required parts of the test, and also has a description field. I think for the test cases with a zarr hierarchy, such a sidecar file is better than stashing it in the zarr.json. In the oztc repo above, the sidecar is in TOML, but for the sake of consistency I was leaning towards making it JSON.

So, for the different test kinds:

  • parse_attributes
    • "_conformance" key in the root JSON with an object value
  • validate_zarr
    • _conformance.json file in the root of the Zarr hierarchy (maybe .conformance.json?)
  • transform_coordinates
    • _conformance.json file in the root of the Zarr hierarchy
    • also includes the required information for the test case - source and target coordinate systems, input coordinates, expected output coordinates

These objects could look something like

field requirement type description
description MAY string description of the test, why it's valid or invalid
transform_coordinates MUST for transform_coordinates tests object source and target coordinate systems, tolerances, starting coordinates and expected ending coordinates

We could potentially add a field for added_on: date. I also considered encoding the test kind/ version/ profile/ validity/ path, but that duplicates information with the path of the test and means more things to update when you copy a test for e.g. a new version. Also, we want to consider users who want to use the test cases from this repo but not the oztest tool (using their own language's preferred test harness); putting more required information into this sidecar means those implementations become more dependent on it.

Activity

  1. tlambert03 commented on Oct 31, 2026

    @tlambert03
    Contributor

    👋
    fwiw, I opened ome/ngff#646 about stable codes for spec rules, which seems somewhat related to this (and mentioned by @kevinyamauchi in #50 (review)). I ran your cases through my branch, and 344 of the 462 invalid ones come back with a code, so there's decent overlap already. Going the other way, 99 of my 163 spec rules with examples never fired on any of those invalid cases, so where a case really is missing, I'm happy to turn my example into a proper one.

  2. clbarnes commented on Nov 1, 2026

    @clbarnes
    CollaboratorAuthor

    That's a useful Venn diagram to have.

    I don't think the conformance tests necessarily need to have a separate case for every spec rule. I think that would quickly become quite onerous, producing several cases for "omit this field", "this field but it's a string" for all combinations of fields. I'm more concerned with the valid cases ("you must be able to parse this") than the invalid ones, because the set of invalid data is infinite and allowing some level of permissive reading is better than encouraging all readers to reject as much data as possible.

    If we had test cases for write/ mutation operations, the invalid cases would get a lot more important, because conservative writing is very desirable. But there are other problems with that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions