Repository navigation
Description
hey folks, I've been working on something in yaozarrs that seems like it could be useful more broadly, so I wanted to float it (I mentioned this briefly on the pre-hackathon zoom 2 days ago).
I'd like to propose a shared set of stable codes for the normative statements in the spec and schemas .. that any implementation or test case could use. (I wasn't able to find any prior discussion of this, so apologies if I missed it)
My goal in yaozarrs was to raise a ValidationError for each normative statement or rule in the spec or schema, with a code that links to a URL on my documentation page, which in turn links to the exact line in the spec or schema that mandates it... kinda like a linter would. I think it would immediately help clarify confusion about where/why a given thing is invalid or not.
In [1]: v06.WellDef.model_validate({"images": [{"path": "0"}, {"path": "0", "acquisition": 1}]})
ValidationError: 1 validation error for WellDef
Field of view paths must be unique. Duplicates: ['0']
See https://imaging-formats.github.io/yaozarrs/rules/NGFF-WL-002/ [type=NGFF-WL-002, ...](those docs pages aren't published yet, it's still on a branch)
It seems like other projects are running into similar issues. German-BioImaging/oztest#46 is asking for descriptions of why a test case is invalid, and on German-BioImaging/oztest#50 it wasn't clear which rule a couple of cases were meant to test. ngff-zarr has something similar too... @thewtex has a table of 15 structural rules with stable identifiers like axis-count, shared between the python and typescript ports. So we're already starting to grow separate vocabularies for the same statements, and it seems like a good moment to see if we can share one.
I have a working prototype on this branch with 170 rules (164 from the spec and schemas for v0.4 to v0.6, plus 6 that are yaozarrs-specific), and I'm happy with it. A rule looks like this:
[rules.NGFF-WL-002]
slug = 'field-path-unique'
level = 'MUST'
summary = 'Field of view `path`s must be unique within a well.'
introduced = '0.4'
[[rules.NGFF-WL-002.sources]]
quote = 'MUST NOT be a duplicate of any other `path` in the `images` list.'
# where that quote appears in each spec version: [first line, last line]
lines = { "0.4" = [605, 606], "0.5" = [540, 541], "0.6" = [1680, 1680] }A code stays the same across spec versions for as long as the requirement does. If the requirement changes, the old code gets retired and a new one gets added (e.g. v0.6 relaxed the allowed characters in a field of view path, so NGFF-WL-001 became NGFF-WL-010). Spec files are pinned to a commit and vendored, and a test checks that each quote is still at those lines.
I think the rules themselves could live in a structured file (toml, json, whatever) somewhere outside of yaozarrs... possibly here, or in ngff-spec, or in oztest (German-BioImaging/oztest#28 makes me think those may end up close together anyway). Implementations could then share the codes, and oztest cases could say which rule they test. There would be clear and simple guides about how a rule gets added, when a code has to change, permanence, etc. (what I have so far is a Claude skill, and I could imagine agentic skills like that helping with this too... e.g. proposing rule updates when a new spec version comes out)
Each rule on my branch also has a minimal failing example, but we could leave those out of a shared file and have each rule point at the oztest cases that exercise it instead.
My rules file only covers what yaozarrs checks so far. For a shared file, I think we'd want every normative statement that can be violated, even if nothing checks it yet (with stuff about reader/writer behavior marked as such). A test could flag any line with a keyword that isn't cited by a rule or explicitly skipped. None of this has merged, so the names and numbering are all up for grabs.
Thoughts? Where do you think this sort of thing would/should live? (here or related seems most natural) The upcoming Zurich hackathon seems like a good place to work on this together, and I'm happy to put together a PR for one small area beforehand if that makes it easier to look at.
cc @clbarnes @joshmoore @jo-mueller @kevinyamauchi @DragaDoncila
Activity
clbarnes commentedon Nov 1, 2026
This is a great effort, thanks for kicking it off! It's definitely something we've talked about and would make the validation story more consistent across implementations.
Not a blocker, but something to consider is that keeping in sync the normative text, at least one set of schemas (currently JSONSchema, but there's interest in LinkML too), this rule list, and the oztest cases if they become part of the main project. Some inertia around a stable format is great but any update requiring changes to 4+ different formalisms could be a bit much.
Is the rule manifest format something custom or are there other examples out there? When extensions come in we may like some mechanism for people to define their own rules, like linter plugins (with all the namespacing headache that brings).
Ideally we'd have anchors for the rules in the text rather than needing to rely on line references which are obviously susceptible to formatting changes and so on - not that this is as much a problem for published specs, but a headache in development.
d-v-b commentedon Nov 1, 2026
also worth reacting to the structure of these failures by changing the spec to make them unrepresentable, where possible. If the motivating examples (uniqueness checks for fields that are used as keys) are common, then consider using JSON objects to handle key uniqueness at the JSON level. You might get new kinds of errors when fields declared in 2 places don't match, but that can be scoped as broken links between well-formed objects.
tlambert03 commentedon Nov 1, 2026
totally agree. I guess my thinking was
yeah, i agree. but that's why the spec files are pinned to a commit.
Even if the ngff ground is shifting underneath, the provenance of a given rule is very clear. And once assigned, those IDs are permanent and never reused, and there's a way to link antiquated rules with the ones that supercede them.
it doesn't strictly adhere to one existing standard... but i did explore multiple options including VeraPDF, OGC ModSpec, ruff's own patterns, and some others. For this one i wanted:
What I landed on is ultimately a lot like OASIS TAM, (which I only found after the fact). Each assertion has an ID, a normative source (a quote or reference) and a level like mandatory/preferred/permitted. Duvet also does something similar in practice, with TOML extracted from RFCs. So if we wanted to strictly adhere to a pattern being used elsewhere, those standards could be explored (i don't personally feel that's necessary, and i think it can sometimes cause a bit of square-peg-round-hole)
clbarnes commentedon Nov 1, 2026
Thanks, that all makes sense!
There's also a bit of overlap here with a prototype I had for a "capability manifest" - a TOML file listing the features defined by a specification where implementations could check off which features they did and didn't support. https://github.com/clbarnes/capability-manifest . The features are more general than strict assertions, but there's also a utility for importing other manifests so that tool A can say "my support comes from library B, so I'm inheriting their manifest".
I don't know if I'm sounding like "you'll never learn to code properly if you use this newfangled 'C' without reading the assembly it generates!" but I feel quite wary about putting too much weight on processes which are only feasible with LLMs doing most of the work. It's not like it's going away, but it doesn't feel like a reliable part of the supply chain (although in open source, what is).
tlambert03 commentedon Nov 2, 2026
right, I remember the conversation at German-BioImaging/ome-zarr-ideas#71
I see the similarities... but i think they're minor/complementary rather than conflicting.
😂 Believe me... I fully sympathize. there are all kinds of places in which i fear (while also begrudgingly admire) what LLMs are doing to how we work. I think however, that this particular use case is right in it's wheelhouse.
I don't see this as "only feasible" with LLMs doing most of the work. In fact, I started by sitting down myself and simply reading the spec (like a regular human 😂) and writing down each normative statement... before thinking "this is silly... this is exactly what an LLM is good at". Then had it continue so that I could check it's work rather than start from scratch. In any case, whether one uses an LLM or not, i do not intend to be suggesting a feature that fundamentally depends on LLMs. This is all fully doable by humans: the association of a code with a normative statement in some other document can be and has been done by humans in other projects for decades. It is both doable and verifiable by humans. So I don't think this is a risky dependence
lubianat commentedon Nov 2, 2026
looks great to me. This would also close #631
I wonder if this is RFC-sized, perhaps together with oztest? Or even together with general schema management/LinkML and what to do when two sources (e.g. text vs schema) differ
RFC XX Rules, Conformance and Schemas
Maybe too much?
tlambert03 commentedon Nov 2, 2026
yeah, it occurred to me. But yes, since it was slightly ancillary to the spec itself, it felt different enough from the other RFCs that I figured a conversation was a better starting point. However: more than happy to draft up an RFC in collaboration with any other interested parties if requested.
lubianat commentedon Nov 2, 2026
There are a few ancillary RFCs coming up now
RFC 10 is the Editorial Board
RFC 11 is for organizing the core conceps
RFC 13 is about versioning of the spec
So it is aligned with the zeitgeist. I think the RFC can come up after the work, though, just to make things official and have a record
lubianat commentedon Nov 4, 2026
btw I heard the need for better decision records in OME-Zarr during the hackathon, I think this is a step in the right direction
Edit: for that, there could be an optional field in the rules for "background" or similar, whenever it makes sense
refactor: use yaozarrs' own namespace for rule codes
clbarnes commentedon Nov 8, 2026
Something I've been thinking (briefly) about - Common Expression Language is designed for compilable, very fast, formally-provable validation with implementations in many languages. I haven't tried it out yet but if we could provide a cel statement with each rule, that's basically everyone's validation story sorted. In combination with a linkml schema for codegen, validation levels 1-3 could be done in a language-agnostic way.
imagesc-bot commentedon Nov 9, 2026
This issue has been mentioned on Image.sc Forum. There might be relevant details there:
https://forum.image.sc/t/2026-ome-zarr-hackathon-stable-error-codes-for-spec-violations/123127/1
AlexSurtees commentedon Nov 9, 2026
This is a great idea that I'd love to see implemented!
feat(py,ts): check the transformation rules from one shared table
lubianat commentedon Nov 13, 2026
the Portoloan spec in the Geo world has some interesting usage of stable codes btw
it is also a collection for a JSON-spec related to cloud-ready research data
https://spec.portolan-sdi.org/v0.2.0/downloads/portolan-spec-v0.2.0.pdf
e.g.
https://github.com/portolan-sdi/portolan-spec/blob/9083822f03ffab5a5f8a2fd43d51f0c5895e71b7/specs/portolan/requirements.yaml#L793

PORTO-CORE-070
I am not sure how this relates to schemas though
EDIT: ping @nlebovits in case you have some extra pointers on how you handle codes for spec rules 🙏