Skip to content

proposal: stable codes for spec rules, in a shared registry? #646

Open

Description

@tlambert03

hey folks, I've been working on something in yaozarrs that seems like it could be useful more broadly, so I wanted to float it (I mentioned this briefly on the pre-hackathon zoom 2 days ago).

I'd like to propose a shared set of stable codes for the normative statements in the spec and schemas .. that any implementation or test case could use. (I wasn't able to find any prior discussion of this, so apologies if I missed it)

My goal in yaozarrs was to raise a ValidationError for each normative statement or rule in the spec or schema, with a code that links to a URL on my documentation page, which in turn links to the exact line in the spec or schema that mandates it... kinda like a linter would. I think it would immediately help clarify confusion about where/why a given thing is invalid or not.

In [1]: v06.WellDef.model_validate({"images": [{"path": "0"}, {"path": "0", "acquisition": 1}]})
ValidationError: 1 validation error for WellDef
  Field of view paths must be unique. Duplicates: ['0']
  See https://imaging-formats.github.io/yaozarrs/rules/NGFF-WL-002/ [type=NGFF-WL-002, ...]

(those docs pages aren't published yet, it's still on a branch)

It seems like other projects are running into similar issues. German-BioImaging/oztest#46 is asking for descriptions of why a test case is invalid, and on German-BioImaging/oztest#50 it wasn't clear which rule a couple of cases were meant to test. ngff-zarr has something similar too... @thewtex has a table of 15 structural rules with stable identifiers like axis-count, shared between the python and typescript ports. So we're already starting to grow separate vocabularies for the same statements, and it seems like a good moment to see if we can share one.

I have a working prototype on this branch with 170 rules (164 from the spec and schemas for v0.4 to v0.6, plus 6 that are yaozarrs-specific), and I'm happy with it. A rule looks like this:

[rules.NGFF-WL-002]
slug = 'field-path-unique'
level = 'MUST'
summary = 'Field of view `path`s must be unique within a well.'
introduced = '0.4'

[[rules.NGFF-WL-002.sources]]
quote = 'MUST NOT be a duplicate of any other `path` in the `images` list.'
# where that quote appears in each spec version: [first line, last line]
lines = { "0.4" = [605, 606], "0.5" = [540, 541], "0.6" = [1680, 1680] }

A code stays the same across spec versions for as long as the requirement does. If the requirement changes, the old code gets retired and a new one gets added (e.g. v0.6 relaxed the allowed characters in a field of view path, so NGFF-WL-001 became NGFF-WL-010). Spec files are pinned to a commit and vendored, and a test checks that each quote is still at those lines.

I think the rules themselves could live in a structured file (toml, json, whatever) somewhere outside of yaozarrs... possibly here, or in ngff-spec, or in oztest (German-BioImaging/oztest#28 makes me think those may end up close together anyway). Implementations could then share the codes, and oztest cases could say which rule they test. There would be clear and simple guides about how a rule gets added, when a code has to change, permanence, etc. (what I have so far is a Claude skill, and I could imagine agentic skills like that helping with this too... e.g. proposing rule updates when a new spec version comes out)

Each rule on my branch also has a minimal failing example, but we could leave those out of a shared file and have each rule point at the oztest cases that exercise it instead.

My rules file only covers what yaozarrs checks so far. For a shared file, I think we'd want every normative statement that can be violated, even if nothing checks it yet (with stuff about reader/writer behavior marked as such). A test could flag any line with a keyword that isn't cited by a rule or explicitly skipped. None of this has merged, so the names and numbering are all up for grabs.

Thoughts? Where do you think this sort of thing would/should live? (here or related seems most natural) The upcoming Zurich hackathon seems like a good place to work on this together, and I'm happy to put together a PR for one small area beforehand if that makes it easier to look at.

cc @clbarnes @joshmoore @jo-mueller @kevinyamauchi @DragaDoncila

Activity

  1. clbarnes commented on Nov 1, 2026

    @clbarnes
    Contributor

    This is a great effort, thanks for kicking it off! It's definitely something we've talked about and would make the validation story more consistent across implementations.

    Not a blocker, but something to consider is that keeping in sync the normative text, at least one set of schemas (currently JSONSchema, but there's interest in LinkML too), this rule list, and the oztest cases if they become part of the main project. Some inertia around a stable format is great but any update requiring changes to 4+ different formalisms could be a bit much.

    Is the rule manifest format something custom or are there other examples out there? When extensions come in we may like some mechanism for people to define their own rules, like linter plugins (with all the namespacing headache that brings).

    Ideally we'd have anchors for the rules in the text rather than needing to rely on line references which are obviously susceptible to formatting changes and so on - not that this is as much a problem for published specs, but a headache in development.

  2. d-v-b commented on Nov 1, 2026

    @d-v-b
    Contributor

    also worth reacting to the structure of these failures by changing the spec to make them unrepresentable, where possible. If the motivating examples (uniqueness checks for fields that are used as keys) are common, then consider using JSON objects to handle key uniqueness at the JSON level. You might get new kinds of errors when fields declared in 2 places don't match, but that can be scoped as broken links between well-formed objects.

  3. tlambert03 commented on Nov 1, 2026

    @tlambert03
    ContributorAuthor

    Some inertia around a stable format is great but any update requiring changes to 4+ different formalisms could be a bit much.

    totally agree. I guess my thinking was

    • the prose, schema, possible LinkML, and now oztest cases were already 4 different things that had no particular autosync going on between them as it is. Yes, this is now a 4th/5th (depending on LinkML), but it's actually sort of the only one that gives a linkable id to the concepts being described across them all. In other words: that was kind of the problem statement that this proposal attempts to make somewhat easier. (maybe not in terms of keeping them in sync, but at least in having a vocabulary to talk about them).
    • I imagined (correct me if I'm wrong), that many of these things (prose, schema) are relatively stable within a version. So I didn't picture most of these rules changing after publication of a spec (I know that's not strictly true for whatever the current version is), I picture this as a periodic effort around each release.
    • Even then, this matching of a normative statement to a line in some permanent sha-based document somewhere is definitely something that LLMs are very good at. So at the very least, a skill that gets two different git shas and is asked to check whether any statements have been added/removed/changed is very useful, and when I've done that and then manually followed up on each one (as in my current branch), it's extremely accurate. So, a biweekly/monthly job here that says "looks like a rule changed!" feels very feasible.

    Ideally we'd have anchors for the rules in the text rather than needing to rely on line references which are obviously susceptible to formatting changes and so on

    yeah, i agree. but that's why the spec files are pinned to a commit.
    Even if the ngff ground is shifting underneath, the provenance of a given rule is very clear. And once assigned, those IDs are permanent and never reused, and there's a way to link antiquated rules with the ones that supercede them.

    Is the rule manifest format something custom or are there other examples out there?

    it doesn't strictly adhere to one existing standard... but i did explore multiple options including VeraPDF, OGC ModSpec, ruff's own patterns, and some others. For this one i wanted:

    • does not ship with the source code (unlike ruff)
    • assumes the spec is prose and schema is structured json or other
    • points to lines in the spec/schema somehow
    • has a 3-level rule (must, should, may) for each normative statement

    What I landed on is ultimately a lot like OASIS TAM, (which I only found after the fact). Each assertion has an ID, a normative source (a quote or reference) and a level like mandatory/preferred/permitted. Duvet also does something similar in practice, with TOML extracted from RFCs. So if we wanted to strictly adhere to a pattern being used elsewhere, those standards could be explored (i don't personally feel that's necessary, and i think it can sometimes cause a bit of square-peg-round-hole)

  4. clbarnes commented on Nov 1, 2026

    @clbarnes
    Contributor

    Thanks, that all makes sense!

    There's also a bit of overlap here with a prototype I had for a "capability manifest" - a TOML file listing the features defined by a specification where implementations could check off which features they did and didn't support. https://github.com/clbarnes/capability-manifest . The features are more general than strict assertions, but there's also a utility for importing other manifests so that tool A can say "my support comes from library B, so I'm inheriting their manifest".

    I don't know if I'm sounding like "you'll never learn to code properly if you use this newfangled 'C' without reading the assembly it generates!" but I feel quite wary about putting too much weight on processes which are only feasible with LLMs doing most of the work. It's not like it's going away, but it doesn't feel like a reliable part of the supply chain (although in open source, what is).

  5. tlambert03 commented on Nov 2, 2026

    @tlambert03
    ContributorAuthor

    overlap here with a prototype I had for a "capability manifest"

    right, I remember the conversation at German-BioImaging/ome-zarr-ideas#71
    I see the similarities... but i think they're minor/complementary rather than conflicting.

    I feel quite wary about putting too much weight on processes which are only feasible with LLMs doing most of the work.

    😂 Believe me... I fully sympathize. there are all kinds of places in which i fear (while also begrudgingly admire) what LLMs are doing to how we work. I think however, that this particular use case is right in it's wheelhouse.

    I don't see this as "only feasible" with LLMs doing most of the work. In fact, I started by sitting down myself and simply reading the spec (like a regular human 😂) and writing down each normative statement... before thinking "this is silly... this is exactly what an LLM is good at". Then had it continue so that I could check it's work rather than start from scratch. In any case, whether one uses an LLM or not, i do not intend to be suggesting a feature that fundamentally depends on LLMs. This is all fully doable by humans: the association of a code with a normative statement in some other document can be and has been done by humans in other projects for decades. It is both doable and verifiable by humans. So I don't think this is a risky dependence

  6. lubianat commented on Nov 2, 2026

    @lubianat
    Contributor

    looks great to me. This would also close #631

    I wonder if this is RFC-sized, perhaps together with oztest? Or even together with general schema management/LinkML and what to do when two sources (e.g. text vs schema) differ

    RFC XX Rules, Conformance and Schemas

    Maybe too much?

  7. tlambert03 commented on Nov 2, 2026

    @tlambert03
    ContributorAuthor

    I wonder if this is RFC-sized

    yeah, it occurred to me. But yes, since it was slightly ancillary to the spec itself, it felt different enough from the other RFCs that I figured a conversation was a better starting point. However: more than happy to draft up an RFC in collaboration with any other interested parties if requested.

  8. lubianat commented on Nov 2, 2026

    @lubianat
    Contributor

    There are a few ancillary RFCs coming up now

    RFC 10 is the Editorial Board

    RFC 11 is for organizing the core conceps

    RFC 13 is about versioning of the spec

    So it is aligned with the zeitgeist. I think the RFC can come up after the work, though, just to make things official and have a record

  9. lubianat commented on Nov 4, 2026

    @lubianat
    Contributor

    btw I heard the need for better decision records in OME-Zarr during the hackathon, I think this is a step in the right direction

    Edit: for that, there could be an optional field in the rules for "background" or similar, whenever it makes sense

  10. clbarnes commented on Nov 8, 2026

    @clbarnes
    Contributor

    Something I've been thinking (briefly) about - Common Expression Language is designed for compilable, very fast, formally-provable validation with implementations in many languages. I haven't tried it out yet but if we could provide a cel statement with each rule, that's basically everyone's validation story sorted. In combination with a linkml schema for codegen, validation levels 1-3 could be done in a language-agnostic way.

  11. imagesc-bot commented on Nov 9, 2026

    @imagesc-bot

    This issue has been mentioned on Image.sc Forum. There might be relevant details there:

    https://forum.image.sc/t/2026-ome-zarr-hackathon-stable-error-codes-for-spec-violations/123127/1

  12. AlexSurtees commented on Nov 9, 2026

    @AlexSurtees

    This is a great idea that I'd love to see implemented!

  13. lubianat commented on Nov 13, 2026

    @lubianat
    Contributor

    the Portoloan spec in the Geo world has some interesting usage of stable codes btw

    it is also a collection for a JSON-spec related to cloud-ready research data
    https://spec.portolan-sdi.org/v0.2.0/downloads/portolan-spec-v0.2.0.pdf

    e.g.

    https://github.com/portolan-sdi/portolan-spec/blob/9083822f03ffab5a5f8a2fd43d51f0c5895e71b7/specs/portolan/requirements.yaml#L793
    image

    PORTO-CORE-070

      - id: PORTO-CORE-070
        severity: MUST
        enforcement: validator
        file: specs/portolan/core.md
        quote: >-
          when a collection provides more than one style, exactly one style asset
          MUST carry both `style` and `default` in its `roles`.
    

    I am not sure how this relates to schemas though

    EDIT: ping @nlebovits in case you have some extra pointers on how you handle codes for spec rules 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions